A RetroSearch Logo

Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Search Query:

Showing content from https://github.com/ispc/ispc/wiki/ISPC-Development-Guide below:

ISPC Development Guide · ispc/ispc Wiki · GitHub

You may follow the steps below or use docker files (available for Linux only) as a reference on how to build LLVM and ISPC.

Most of development day-to-day tasks can be performed using pre-built Clang/LLVM that can be downloaded from

ispc.dependencies

. Download and unpack the archive. Make sure that unpacked clang works that is usually fine with Windows but it can be tricky with Linux.

Additionally, on Windows you need:

Additionally, on Linux you need:

These section explains how to build ISPC using prebuilt Clang/LLVM.

Ubuntu 22.04/24.04:

apt-get install git curl cmake xz-utils m4 flex bison python3 python3-dev libtbb-dev g++-multilib

Arch Linux:

pacman -S git cmake make gcc-libs glibc lib32-glibc gcc m4 flex bison python onetbb
choco install winflexbison3
choco install gnuwin32-m4

Build ISPC with the following commands:

git clone --recursive https://github.com/ispc/ispc
./ispc/scripts/quick-start-build.py

Build ISPC with the following commands:

git clone --recursive https://github.com/ispc/ispc
python ispc\scripts\quick-start-build.py

On Windows, the quick-start-build.py script requires py7zr to unpack downloaded archives.

pip install py7zr

Note: Incremental builds can be performed by simply re-running the quick-start-build.py script.

There is a script named quick-start-build.py that automatically downloads Clang/LLVM and builds ISPC.

You can provide the required LLVM version as a positional argument. By default, this script downloads the current version LLVM used for development.

./ispc/scripts/quick-start-build.py 18

You can also specify the directory where LLVM/Clang will be unpacked usign the LLVM_HOME environment variable.

bash:

LLVM_HOME=/path/to/llvm python quick-start-build.py

cmd (Windows):

set LLVM_HOME=C:\path\to\llvm && python quick-start-build.py

PowerShell:

$env:LLVM_HOME = "C:\path\to\llvm"; python quick-start-build.py

Additionally, there is an environment variable ISPC_HOME to specify where this script should create the build directory for the ISPC build. Under this directory, this script creates a build-<version_number> directory, where you can find the built ISPC executable, rerun the build, or inspect it. It also safe to rerun the entire quick-start-build.py script to rebuild ISPC after making changes to the codebase.

Supported LLVM & glibc versions

When running quick-start-build.py, the script downloads LLVM built on Ubuntu 22.04, which has glibc 2.35. So for quick-start-build.py to work correctly, your system must have glibc 2.35 or later. Below are the default glibc versions for different Ubuntu releases:

Ubuntu Version Default glibc Version Ubuntu 18.04 glibc 2.27 Ubuntu 20.04 glibc 2.31 Ubuntu 22.04 glibc 2.35

Here is a sample output from running quick-start-build.py with a Ubuntu 20.04 as operating system:

  It fails with the following output:

    Change Dir: /home/user/workspace/ispc/build-18/CMakeFiles/CMakeTmp

    Run Build Command(s):/usr/bin/make cmTC_8dbd9/fast && /usr/bin/make -f CMakeFiles/cmTC_8dbd9.dir/build.make CMakeFiles/cmTC_8dbd9.dir/build
    make[1]: Entering directory '/home/user/workspace/ispc/build-18/CMakeFiles/CMakeTmp'
    Building C object CMakeFiles/cmTC_8dbd9.dir/testCCompiler.c.o
    /home/user/workspace/llvm-18/bin/clang    -o CMakeFiles/cmTC_8dbd9.dir/testCCompiler.c.o   -c /home/user/workspace/ispc/build-18/CMakeFiles/CMakeTmp/testCCompiler.c
    /home/user/workspace/llvm-18/bin/clang: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /home/user/workspace/llvm-18/bin/clang)
    /home/user/workspace/llvm-18/bin/clang: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /home/user/workspace/llvm-18/bin/clang)
    /home/user/workspace/llvm-18/bin/clang: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /home/user/workspace/llvm-18/bin/clang)
    /home/user/workspace/llvm-18/bin/clang: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /home/user/workspace/llvm-18/bin/clang)
    /home/user/workspace/llvm-18/bin/clang: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /home/user/workspace/llvm-18/bin/clang)
    /home/user/workspace/llvm-18/bin/clang: /lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.13' not found (required by /home/user/workspace/llvm-18/bin/clang)
    make[1]: *** [CMakeFiles/cmTC_8dbd9.dir/build.make:66: CMakeFiles/cmTC_8dbd9.dir/testCCompiler.c.o] Error 1
    make[1]: Leaving directory '/home/user/workspace/ispc/build-18/CMakeFiles/CMakeTmp'
    make: *** [Makefile:121: cmTC_8dbd9/fast] Error 2

What to do if your Ubuntu system has an older LLVM or glibc version? If your system has an older version that does not meet the minimum requirement, you have two options:

apt install llvm clang libclang-dev
Then configure ISPC with: 
PATH=/usr/lib/llvm-18/bin:$PATH cmake <ispc-source-dir>
If you need help upgrading or building LLVM, refer to the official LLVM documentation.

For other Linux and BSD OSes, the best way to determine how to build ISPC with the system LLVM is to consult the ISPC package build instructions. The following links point to some of them:

  1. Arch Linux
  2. Alpine Linux
  3. Fedora Linux
  4. FreeBSD

First, build the LLVM headers and libraries and the clang compiler on your system. For successful ISPC build LLVM must be built with proper flags. The recommended way to build LLVM is to use superbuild CMake configuration. It passes all required flags depending on LLVM version and applies required patches.

Note: all examples below are using LLVM 18.1 version but you can use any supported LLVM version. They also suppose that ISPC_HOME environment variable points to the path of ISPC repository and HOME_DIR points to the directory where an user starts to run command listed below.

Building LLVM using superbuild 
git clone --recursive https://github.com/ispc/ispc
cmake -B build-llvm ispc/superbuild --preset os -DLLVM_VERSION=18.1 -DCMAKE_INSTALL_PREFIX=llvm-18.1 -DBUILD_STAGE2_TOOLCHAIN_ONLY=ON
cmake --build build-llvm

You may want to speed up your build with -jN switch, where N is a number of parallel jobs to run. Once cmake build is completed, the llvm-18.1 directory contains clang and LLVM with required libraries and headers ready to be used to build ISPC.

Make sure to add bin folder under this directory to the system PATH environment variable.

For Linux/Mac OS:

export PATH=$HOME_DIR/llvm-18.1/bin:$PATH

For Windows PowerShell:

$env:PATH = "$env:HOME_DIR\llvm-18.1\bin;" + $env:PATH

For Windows cmd:

set "PATH=%HOME_DIR\llvm-18.1\bin;%PATH%"
2.1 Building ISPC for CPU development

You're now ready to build ISPC. Run CMake with the following options:

cmake -B build-ispc ispc
cmake --build build-ispc

ISPC executable is placed under bin directory in the build directory. You may use it or provide CMAKE_INSTALL_PRERIX and run install target to install ISPC in the desired location.

ISPC-specific variables

You can specify path to required tools using the following variables:

2.2 Building ISPC with cross-OS compilation support

You can build ISPC with cross-OS compilation support by passing -DISPC_CROSS=ON on Linux/macOS and -DISPC_CROSS=ON -DISPC_GNUWIN32_PATH=/absolute/path/to/GnuWin32/bin on Windows to CMake command described in 2.1. Depending on your host system ISPC may be built for Windows, Linux, FreeBSD, macOS, Android, iOS, and PS targets. You can disable any target system by passing -DISPC_[WINDOWS|LINUX|FREEBSD|MACOS|ANDROID|IOS|PS]_TARGET=OFF to CMake. By default the following combinations are supported:

Depending on which target OSes are selected you will be asked to provide additional paths:

2.3 Building ISPC with Xe support

Xe-enabled build is supported on Windows and Linux, and it has three additional dependencies:

Please use exact commits SHA as used in

osPresets

and the same LLVM as you plan to build ISPC. CMake commands used in

Dockerfile

are applicable for both Windows and Linux. Install build artifacts from SPIRV-LLVM-Translator and vc-intrinsics to one folder (

XE_DEPS

) and artifacts from level-zero build to another folder (

L0_ROOT_DIR

). Now you're ready to build ISPC with Xe support, just provide extra flags to CMake commands from

2.1

:

-DXE_ENABLED=ON -DXE_DEPS_DIR=$XE_DEPS -DLEVEL_ZERO_ROOT=$L0_ROOT_DIR 2.4 Building ISPC package with cross-compilation and Xe support

If you want to build a complete ISPC package with cross-compilation and Xe support, the easiest way to sue superbuild CMake using prebuilt LLVM from the stage 1. Run the following CMake commands:

On Linux:

On Windows cmd:

The system consists of three main scripts:

Also system has some additional files:

Tests are in three directories:

If you want to validate stability

If you want to check stability of ISPC compiler after your changes you should use run_tests.py or alloy.py -r --only=stability. Note that you must set LLVM_HOME and ISPC_HOME for alloy.py

Run_tests.py will execute all tests from “tests” and “tests_errors” directories for a given target, arch and opt_level. Then it will report which failed tests (runfail or compfail). Also run_tests.py will check fail_db.txt file with known fails and will report if the fails are new or known.

If you want to have more general view you should use alloy.py -r --only=stability. This will run run_tests.py for all/selected supported targets, archs, opt_levels and LLVM versions (Note that generic targets will run only with x86-64 arch, this is by design). Then you will have full report about fails, new_fails, passes and new_passes. If you don’t have appropriate LLVM version alloy will silently download and build it. You can select combinations for your test runs by using option --only=”” and select targets by using --only-targets=””. Note that alloy will automatically detect targets of your system and SDE (if you set SDE_HOME environment variable).

If you want to change fail_db.txt file by adding or deleting fails of current run you should use option --update-errors=F (update fails) or --update-errors=FP (update fails and passes) both in alloy.py and run_tests.py.

Each test in tests folder is in a self-contained ispc source file checking particular functionality; it must define two functions:

The test script compiles each test function to object file for CPU targets or SPIR-V or ZE Binary for Xe targets and then runs the functions above, passing the test function particular values of particular types based on its signature. It then checks to make sure that the values returned from the call to

result()

match the values returned from the call to the test function; if they don't the differing values are printed along with an error message.

To make this concrete, here is a example of a test (a cleaned-up version of tests/bool-float-typeconv.ispc). It does a quick sanity check of bool to float type conversion.

    #include "../test_static.isph"
    task void f_f(uniform float RET[], uniform float aUniform[]) {
        float a = aUniform[programIndex];
        RET[programIndex] = a < 3.;
    }
    
    task void result(uniform float RET[]) { 
        RET[programIndex] = 0; RET[0] = RET[1] = 1; 
    }

The test starts from #include "../test_static.isph". It contains CPU wrappers for task functions used in tests. It is motivated by different entry points of ISPC program on CPU and Xe - on CPU it is an export function, on Xe it is a task. To have unified set of tests for CPU and Xe we use task modifier in test functions and wrap them into export functions with a launch task inside for CPU using test_static.isph. For example, the wrapper for the f_f task will be:

task void f_f(uniform float res[], uniform float vfloat[]);
export void f_f_cpu_entry_point(uniform float res[], uniform float vfloat[]) { launch[1] f_f(res, vfloat); }

Now let's look into ISPC functions themselves. First, notice that the test function defined here is f_f. In addition to the array in which to store the result values computed by the function, the RET parameter, functions with the name f_f are also passed an array of floats, with values {1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}. The test function converts this to a varying value

a

by directly indexing into this array with

programIndex

, giving

a

the value one in the first program instance and so forth. By inspection, we can see that the boolean test in the last line of f_f should evaluate to true for the first two program instances running, and false for all of the rest, and that the conversion of those bools to floats should put 1 in the first two program instances result values and zero in the rest. These, in turn, are the values that result() reports are expected.

Here are the types and values passed as parameters by the test driver for functions with the various signatures listed above:

    task void f_v(uniform float RET[]);  // i.e. no parameters passed other than the output array

    // a[] = { 1, 2, 3, ... };
    task void f_f(uniform float RET[], uniform float a[]);

    // a[] = { 1, 2, 3, ... };
    // b = 5;
    task void f_fu(uniform float RET[], uniform float a[], float b);

    // a[] = { 1, 2, 3, ... };
    // b[] = { 2, 4, 6, ... };
    task void f_fi(uniform float RET[], uniform float a[], int b[]);

    // a[] = { 1, 2, 3, ... };
    // b[] = { 5, 6, 7, ... };
    task void f_di(uniform float RET[], uniform double a[], int b[]);

    // a[] = { 1, 2, 3, ... };
    // b = 5;
    task void f_du(uniform float RET[], uniform double a[], double b);

    // a[] = { 1, 2, 3, ... };
    // b = 5;
    task void f_duf(uniform float RET[], uniform double a[], float b);

There is a slightly different group of tests related to print (e.g. print_uf, print_f, print_fuf). Both testing function and result prints output to stdout and run_tests.py checks that that output is correct.

New functionality should have targeted tests that exercise the set of features that the functionality introduces. If the functionality is in any way dependent on the mask, it's important to exercise a few cases like 'mask all on', 'mask all off', and 'mixed mask'.

Main usage of run_tests.py to understand stability

To run the tests, run the run_tests.py python script in the top-level ispc source directory.

If successful, the test script will print output like:

% ./scripts/run_tests.py
Executed 1517 / 1543 (26 skipped)

PASSRATE (1517/1517) = 100%

1517 / 1543 tests PASSED
0 / 1543 tests FAILED compilation
0 / 1543 tests FAILED execution
26 / 1543 tests SKIPPED

<List of skipped tests>

No new fails                            
%

If some tests fail, the test system will generate an additional output indicating which test failed and how it failed. The exit code is equal to the number of tests that failed. Thus, if all pass, it generates a regular exit code of 0.

Useful commands:

Main usage of alloy.py to validate stability Tips for running tests on Windows

When running on Windows, failing tests may cause a dialog window suggesting to find a solution on the web or debug a problem. This may be annoying. To turn it off, you need to do two things:

 https://docs.microsoft.com/en-us/windows/win32/wer/wer-settings

In addition to functional tests described in previous sections, ISPC has a set of lit tests in tests/lit-tests based on LLVM test infrastructure. They are mainly used to check compiler code generation or to check compiler diagnostics. To write ISPC lit tests, follow the same rules as for LLVM FileCheck. ISPC lit tests can be run using make check-all on Linux or Tests/check-all target in Visual Studio on Windows. There is a set of features that you can use in lit tests if you need to set up execution rules:

The whole list of supported features, you can find in

tests/lit-tests/lit.cfg

Example of ISPC lit test:

// The test checks that cpu definitions (including all synonyms) are successfully consumed by compiler.

// RUN: %{ispc} %s -o %t.o --nostdlib --target=sse2-i32x4 --cpu=znver3
// RUN: %{ispc} %s -o %t.o --nostdlib --target=sse2-i32x4 --cpu=alderlake
// RUN: %{ispc} %s -o %t.o --nostdlib --target=sse2-i32x4 --cpu=adl
// RUN: %{ispc} %s -o %t.o --nostdlib --target=sse2-i32x4 --cpu=sapphirerapids
// RUN: %{ispc} %s -o %t.o --nostdlib --target=sse2-i32x4 --cpu=spr

// REQUIRES: X86_ENABLED
// REQUIRES: LLVM_12_0+

uniform int i;

void foo() {}

All compiler changes should be covered by lit tests.

If you want to validate performance (applied to CPU only)

If you want to validate how your changes influence ISPC performance you should use perf.py or alloy -r --only=performance. Note that you must set LLVM_HOME and ISPC_HOME for alloy.py

If you want to measure performance of your changes use perf.py. Perf.py will build and run all tests listed in perf.ini from “examples/cpu” directory and report performance numbers. If you want to compare performance of two ISPC compilers you should use perf.py --ref=reference_compiler. This will generate a comparison report between two compiler versions.

If you want to compare two branches of ISPC (For example branch with your changes and master) you should use alloy -r --only=performance. This will build the newest LLVM if needed (Note that LLVM will be built silently. If you want selfbuild or source from tar you should use alloy -b first), build your ISPC compiler, switched to “master” branch, build reference compiler and then execute perf.py. Logs will be in alloy_results[date] directory. Option --compare-with=name_of_chekout_or_branch will change reference branch.

If you get suspicious results of runs you can increase the number of runs using -nX ( the switch is available in both alloy.py and perf.py).

Main usage of perf.py to measure performance Main usage of alloy.py to compare performance Adding New Target to ISPC

With the introduction of generic targets in ISPC, adding a new target has become straightforward. Follow the steps below to integrate a new target seamlessly. You can use https://github.com/ispc/ispc/pull/3188 as a reference:

1. Define the New Target in ISPCTarget Class 2. Assign to a Target Group 3. Set Default Properties 4. Modify the Build System 5. Add the target to hierarchical set of ISPC targets 
  ispc --support-matrix

  scripts/run_tests.py --target=<your-target>

  scripts/run_tests.py --target=neon-i16x16
7. Enable Hardware Features (Optional) 8. Test the Target Thoroughly 
  scripts/run_tests.py --target=<your-target> -o [O0|O1|O2]
10. Integrate into Regular Testing (CI)

By following these steps, you can successfully add and validate a new target in ISPC.

HOWTO: Bisecting ISPC and LLVM

Bisecting is often necessary to find the faulty change or the change that fixes an issue. This can be accomplished by dissecting ISPC and/or LLVM history with git bisect.

Important Note: Bisect works well when something changes monotonously in history (e.g., good → good → good → bad → bad → bad). However, if there are multiple points where the behavior swaps back and forth, bisect may find the wrong change.

Before starting bisect, you need:

  1. A reproducible test case that you can compile and run (or just compile)
  2. Two known revisions:
Terminology Note

: If your test case was previously failing and accidentally started passing, and you need to find the change that fixed it, you can use

old

/

new

aliases instead of

bad

/

good

. You can also override these names with user-provided ones (consult the git manual for details).

1. Minimize the Test Case

It's highly recommended to minimize your test case before bisecting:

Sometimes the wrong behavior can be isolated to LLVM only. In this case:

  1. Start bisect: git bisect start
  2. Rebuild the project
  3. Test your case and mark the revision as good/bad (or old/new) accordingly
  4. Repeat until the guilty commit is found

If the faulty behavior can be checked automatically, create a script and run:

git bisect run <your-test-script>
Linear History Considerations

Bisecting is only possible on linear history. Use git merge-base to find start and end points of linear history when the project has forked release branches.

Handling Problematic States 
git bisect skip
ISPC/LLVM Bisection Workflow

This example demonstrates bisecting LLVM history when an ISPC test starts failing due to an LLVM update.

# Known failing state
PATH=/path-to-llvm-K+1/bin:$PATH ./scripts/run_tests.py ./tests/func-tests/bug.ispc  # → FAIL

# Known working state  
PATH=/path-to-llvm-K/bin:$PATH ./scripts/run_tests.py ./tests/func-tests/bug.ispc    # → OK
1. Configure and Build LLVM (Known Good State) 
# Checkout known good LLVM version
[/llvm-project] $ git checkout K

# Configure LLVM (build only what you need for faster iteration)
[/build-llvm] $ cmake /llvm-project/llvm -G Ninja \
    -DLLVM_TARGETS_TO_BUILD="X86" \
    -DLLVM_ENABLE_PROJECTS="clang" \
    -DCMAKE_INSTALL_PREFIX=/dir-llvm-bin \
    -DCMAKE_BUILD_TYPE=Release \
    -DLLVM_ENABLE_ZLIB=OFF \
    -DLLVM_ENABLE_ZSTD=OFF \
    -DLLVM_INSTALL_UTILS=ON

# Build and install
[/build-llvm] $ ninja && ninja install
2. Configure and Build ISPC 
# Checkout ISPC version
[/ispc] $ git checkout 1.N

# Configure ISPC
[/build-ispc] $ PATH=/dir-llvm-bin/bin:$PATH cmake /ispc -G Ninja \
    -DISPC_SLIM_BINARY=ON \
    -DARM_ENABLED=OFF

# Build ISPC
[/build-ispc] $ ninja
3. Verify Known Good State 
[/ispc] $ PATH=/build-ispc/bin:$PATH ./scripts/run_tests.py ./tests/func-tests/bug.ispc
# Should return: OK
# Initialize bisect
[/llvm-project] $ git bisect start

# Mark current (known good) commit as good
[/llvm-project] $ git bisect good

# Checkout known bad commit
[/llvm-project] $ git checkout K+1

# Rebuild and install LLVM
[/build-llvm] $ ninja && ninja install

# Rebuild ISPC
[/build-ispc] $ ninja clean && ninja

# Test the behavior (should fail)
[/ispc] $ PATH=/build-ispc/bin:$PATH ./scripts/run_tests.py ./tests/func-tests/bug.ispc
# Should return: FAIL

# Mark as bad
[/llvm-project] $ git bisect bad

At this point, git will change HEAD to an intermediate commit. Repeat the following process:

  1. Rebuild LLVM: ninja && ninja install
  2. Rebuild ISPC: ninja clean && ninja
  3. Test: PATH=/build-ispc/bin:$PATH ./scripts/run_tests.py ./tests/func-tests/bug.ispc
  4. Mark result: git bisect good or git bisect bad

Continue until git identifies the guilty commit.

# When done, clean up bisect state
[/llvm-project] $ git bisect reset

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4