Unpack LLVM Distribution, and setup lit to run tests for specific test suite by bob80905 · Pull Request #1149 · llvm/offload-test-suite

bob80905 · 2026-05-04T23:54:34Z

We are still encountering errors when configuring cmake with ninja, as ninja cannot be found:
https://github.com/llvm/offload-test-suite/actions/runs/25338698984/job/74291993540#step:13:155
Though the path is correct on the build machine's environment, it no longer holds in the test machine's (a different machine's) environment.
@Icohedron recommended a completely different approach, which involves installing an llvm distribution to the test machine. This avoids needing to replace exe paths in CMakeCache.txt files, and avoids dealing with CMake entirely.
This PR sets up the infrastructure for installing the LLVM offload distribution (with all relevant files) without needing ninja or cmake, and just running lit tests on the appropriate suite.

Assisted by Github Copilot.
Fixes #1150

… timestamps

Icohedron · 2026-05-08T18:59:08Z

As discussed offline, I do not think this is the correct approach. The CMakeCache file is very specific to the machine and is not portable to other machines. Simply patching the CMakeCache file is not a robust solution.

Instead, I think the better approach is to define an LLVM distribution that builds and installs the offloader, clang, and other dependencies into a self-contained and portable package. This package can then be distributed and ran on other machines without any additional building required.

Similarly for the offload tests, an install target can be defined that copies over all the lit tests into its own install prefix (which may or may not be the same as the install prefix for the LLVM distribution).

The main issue to solve would be getting lit to find all the dependencies required to run the tests. A solution for this would be to include a Python script that scans for the required dependencies and uses the information to generate a lit.site.cfg.py to satisfy the requirements for running lit tests.

bob80905 · 2026-05-08T22:26:26Z

+            cmake -G Ninja ${{ inputs.LLVM-ExtraCMakeArgs }} -DCMAKE_BUILD_TYPE=${{ inputs.BuildType }} -DLLVM_ENABLE_ASSERTIONS=On -C ${{ github.workspace }}/llvm-project/clang/cmake/caches/HLSL.cmake -C ${{ github.workspace }}/OffloadTest/cmake/caches/sccache.cmake -DDXC_DIR=${{ github.workspace }}/DXC/build/bin -DLLVM_EXTERNAL_OFFLOADTEST_SOURCE_DIR=${{ github.workspace }}/OffloadTest -DLLVM_EXTERNAL_PROJECTS="OffloadTest" -DLLVM_LIT_ARGS="--xunit-xml-output=testresults.xunit.xml -v" -DOFFLOADTEST_TEST_CLANG=${{steps.Test-Clang.outputs.TEST_CLANG || 'Off' }} -DGOLDENIMAGE_DIR=${{ github.workspace }}/golden-images -DHLSL_ENABLE_OFFLOAD_DISTRIBUTION=${{ inputs.SplitBuild == true && 'On' || 'Off' }} -DCMAKE_INSTALL_PREFIX=${{ github.workspace }}/install ${{ github.workspace }}/llvm-project/llvm/
            ninja hlsl-test-depends
-      - name: Dump GPU Info
+      - name: Dump GPU Info (build runner)


Note, though it doesn't seem to make sense long term to have a dump gpu info step on the build runner machine, it does make sense to have it short term, since PRs will continue to run with splitbuild disabled while I test and validate that everything is working correctly.

bob80905 · 2026-05-08T22:30:37Z

+    "d3d12": (True, False, False, False, False, "DXC-compiled shaders on DirectX 12"),
+    "vk": (False, True, False, False, False, "DXC-compiled shaders on Vulkan"),
+    "mtl": (False, False, True, False, False, "DXC-compiled shaders on Metal"),
+    "clang-d3d12": (


This looks different because darker formatted it like this.

Icohedron · 2026-05-08T23:06:33Z

            mkdir build
            cd build
-            cmake -G Ninja ${{ inputs.LLVM-ExtraCMakeArgs }} -DCMAKE_BUILD_TYPE=${{ inputs.BuildType }} -DLLVM_ENABLE_ASSERTIONS=On -C ${{ github.workspace }}/llvm-project/clang/cmake/caches/HLSL.cmake -C ${{ github.workspace }}/OffloadTest/cmake/caches/sccache.cmake -DDXC_DIR=${{ github.workspace }}/DXC/build/bin -DLLVM_EXTERNAL_OFFLOADTEST_SOURCE_DIR=${{ github.workspace }}/OffloadTest -DLLVM_EXTERNAL_PROJECTS="OffloadTest" -DLLVM_LIT_ARGS="--xunit-xml-output=testresults.xunit.xml -v" -DOFFLOADTEST_TEST_CLANG=${{steps.Test-Clang.outputs.TEST_CLANG || 'Off' }} -DGOLDENIMAGE_DIR=${{ github.workspace }}/golden-images ${{ github.workspace }}/llvm-project/llvm/
+            cmake -G Ninja ${{ inputs.LLVM-ExtraCMakeArgs }} -DCMAKE_BUILD_TYPE=${{ inputs.BuildType }} -DLLVM_ENABLE_ASSERTIONS=On -C ${{ github.workspace }}/llvm-project/clang/cmake/caches/HLSL.cmake -C ${{ github.workspace }}/OffloadTest/cmake/caches/sccache.cmake -DDXC_DIR=${{ github.workspace }}/DXC/build/bin -DLLVM_EXTERNAL_OFFLOADTEST_SOURCE_DIR=${{ github.workspace }}/OffloadTest -DLLVM_EXTERNAL_PROJECTS="OffloadTest" -DLLVM_LIT_ARGS="--xunit-xml-output=testresults.xunit.xml -v" -DOFFLOADTEST_TEST_CLANG=${{steps.Test-Clang.outputs.TEST_CLANG || 'Off' }} -DGOLDENIMAGE_DIR=${{ github.workspace }}/golden-images -DHLSL_ENABLE_OFFLOAD_DISTRIBUTION=${{ inputs.SplitBuild == true && 'On' || 'Off' }} -DCMAKE_INSTALL_PREFIX=${{ github.workspace }}/install ${{ github.workspace }}/llvm-project/llvm/


Note to reviewers: HLSL_ENABLE_OFFLOAD_DISTRIBUTION is introduced in PR llvm/llvm-project#196640

…parate artifact for DXC, addressing Deric

…t path bugs

@Icohedron

This PR adds a new distribution to the HLSL cmake file. The purpose is so that we can portably distribute binaries to other machines so that offload testing can be optimized. Assisted by: Github Copilot This PR was inspired by @Icohedron 's comment here: llvm/offload-test-suite#1149 (comment) This addresses llvm/offload-test-suite#1150

@Icohedron

This PR adds a new distribution to the HLSL cmake file. The purpose is so that we can portably distribute binaries to other machines so that offload testing can be optimized. Assisted by: Github Copilot This PR was inspired by @Icohedron 's comment here: llvm/offload-test-suite#1149 (comment) This addresses llvm/offload-test-suite#1150

@Icohedron

This PR adds a new distribution to the HLSL cmake file. The purpose is so that we can portably distribute binaries to other machines so that offload testing can be optimized. Assisted by: Github Copilot This PR was inspired by @Icohedron 's comment here: llvm/offload-test-suite#1149 (comment) This addresses llvm/offload-test-suite#1150

@Icohedron

This PR adds a new distribution to the HLSL cmake file. The purpose is so that we can portably distribute binaries to other machines so that offload testing can be optimized. Assisted by: Github Copilot This PR was inspired by @Icohedron 's comment here: llvm/offload-test-suite#1149 (comment) This addresses llvm/offload-test-suite#1150

@Icohedron

This PR adds a new distribution to the HLSL cmake file. The purpose is so that we can portably distribute binaries to other machines so that offload testing can be optimized. Assisted by: Github Copilot This PR was inspired by @Icohedron 's comment here: llvm/offload-test-suite#1149 (comment) This addresses llvm/offload-test-suite#1150

bob80905 added 2 commits May 4, 2026 16:51

patch ninja exe path to whatever is in test runner env, and foolproof…

2295213

… timestamps

another self review

2951641

bob80905 added 2 commits May 8, 2026 13:28

address Deric

9fccc9f

formatting

36c4d4a

bob80905 changed the title ~~Patch ninja exe path to whatever is in test runner env, and foolproof cmake timestamps~~ Unpack LLVM Distribution, and setup lit to run tests for specific test suite May 8, 2026

bob80905 commented May 8, 2026

View reviewed changes

bob80905 mentioned this pull request May 8, 2026

[HLSL] Add Offload distribution llvm/llvm-project#196640

Merged

Icohedron reviewed May 8, 2026

View reviewed changes

Comment thread .github/workflows/build-and-test-callable.yaml Outdated

Icohedron reviewed May 8, 2026

View reviewed changes

make separate doc explaining distribution infrastructure, and make se…

eb7ca2d

…parate artifact for DXC, addressing Deric

Icohedron reviewed May 8, 2026

View reviewed changes

Comment thread test/CMakeLists.txt Outdated

Comment thread .github/workflows/build-and-test-callable.yaml Outdated

Icohedron reviewed May 8, 2026

View reviewed changes

Comment thread utils/configure-test-suite.py

bob80905 added 7 commits May 8, 2026 16:31

address Deric, make golden images checkout unconditional

1698781

change /U windows backslash, and fix up other golden image problems

417d14a

iterate

4e8b91b

iterate

df58adc

iterate

7f07999

iterate

16dc95b

remove redundance comment, make it cmakelists specific

91ad0d6

Icohedron reviewed May 12, 2026

View reviewed changes

Comment thread utils/configure-test-suite.py Outdated

Comment thread utils/configure-test-suite.py Outdated

Comment thread utils/configure-test-suite.py Outdated

Comment thread utils/configure-test-suite.py Outdated

Comment thread .github/workflows/build-and-test-callable.yaml Outdated

address Deric

55ae0f1

Icohedron reviewed May 12, 2026

View reviewed changes

Comment thread .github/workflows/build-and-test-callable.yaml Outdated

bob80905 added 3 commits May 12, 2026 16:04

separate bin from lib

8746896

iterate

13e9f9a

iterate, on extraction the dlls and exes should be together to preven…

0ed120e

…t path bugs

Icohedron reviewed May 13, 2026

View reviewed changes

Comment thread .github/workflows/build-and-test-callable.yaml Outdated

Comment thread .github/workflows/build-and-test-callable.yaml Outdated

address Deric, simplify / skip install for DXC

853ec69

Icohedron reviewed May 13, 2026

View reviewed changes

Comment thread .github/workflows/build-and-test-callable.yaml

package pdbs too

be3c890

Icohedron reviewed May 13, 2026

View reviewed changes

Comment thread .github/workflows/build-and-test-callable.yaml Outdated

account for Release configs lacking pdbs and erroring the copy step

0a47a3c

Icohedron approved these changes May 13, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unpack LLVM Distribution, and setup lit to run tests for specific test suite#1149

Unpack LLVM Distribution, and setup lit to run tests for specific test suite#1149
bob80905 wants to merge 19 commits into
llvm:mainfrom
bob80905:fix-test-job-regen

bob80905 commented May 4, 2026 •

edited

Loading

Uh oh!

Icohedron commented May 8, 2026 •

edited

Loading

Uh oh!

bob80905 May 8, 2026

Uh oh!

bob80905 May 8, 2026

Uh oh!

Uh oh!

Icohedron May 8, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

bob80905 commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Icohedron commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bob80905 May 8, 2026

Choose a reason for hiding this comment

Uh oh!

bob80905 May 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Icohedron May 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bob80905 commented May 4, 2026 •

edited

Loading

Icohedron commented May 8, 2026 •

edited

Loading