Previously, in #1328, we enabled the split build feature.
This feature allows a specific build and test / vendor tuple to be built by a non-matching vendor, as long as the platform architecture matches.
In this way, if AMD is idle and we need to build and test something on NVIDIA, AMD can build the binaries and distribute it to the NVIDIA machine to download the binaries in the artifact and test with them, saving some time and improving parallelization / performance.
This did make a marked improvement, but after adding the lavapipe configuration to the test matrix, the testing load demands that we carry the original idea through:
We need a dedicated build machine that frequently builds the relevant binaries, and distributes them to all machines. We don't need to duplicate builds of llvm / dxc, we only need the offload test suite to be the latest build.
So, this issue depends on #1376 getting merged in first, so that the offload test suite can be built independently.
The pipeline / yaml should be changed so that one distinguished label gets assigned the responsibility of frequently rebuilding dxc / clang, and uploading these binaries as artifacts.
Other machines will pull the appropriate artifacts (based on an expected commit hash and other config stuff like architecture), and build latest offload test suite binaries, and run the tests.
This cuts the need for redundant builds from N (where N is the number of different test configurations for a specific architecture (e.g., clang vs dxc, Warp vs lavapipe vs intel vs nvidia vs AMD, d3d12 vs vulkan, ~2 * 5 * 2, or about 20), to 1.
This should also improve the qualcomm performance, since qualcomm has not taken advantage of the split build feature.
It is the only vendor with its own unique (arm64) architecture, and doesn't benefit from another machine building the relevant binaries.
If the dedicated builder can build the binaries targeting arm64, then qualcomm can dramatically reduce the time spent building for tests.
This issue tracks setting up a dedicated frequent builder that builds dxc/clang targeting x64 and arm64, and distributes these builds to all test runners via an artifact store. The test runners ought to be capable of pulling down the right artifacts, and test jobs depend on the Build job; a failed build automatically skips dependent test jobs.
Previously, in #1328, we enabled the split build feature.
This feature allows a specific build and test / vendor tuple to be built by a non-matching vendor, as long as the platform architecture matches.
In this way, if AMD is idle and we need to build and test something on NVIDIA, AMD can build the binaries and distribute it to the NVIDIA machine to download the binaries in the artifact and test with them, saving some time and improving parallelization / performance.
This did make a marked improvement, but after adding the lavapipe configuration to the test matrix, the testing load demands that we carry the original idea through:
We need a dedicated build machine that frequently builds the relevant binaries, and distributes them to all machines. We don't need to duplicate builds of llvm / dxc, we only need the offload test suite to be the latest build.
So, this issue depends on #1376 getting merged in first, so that the offload test suite can be built independently.
The pipeline / yaml should be changed so that one distinguished label gets assigned the responsibility of frequently rebuilding dxc / clang, and uploading these binaries as artifacts.
Other machines will pull the appropriate artifacts (based on an expected commit hash and other config stuff like architecture), and build latest offload test suite binaries, and run the tests.
This cuts the need for redundant builds from N (where N is the number of different test configurations for a specific architecture (e.g., clang vs dxc, Warp vs lavapipe vs intel vs nvidia vs AMD, d3d12 vs vulkan, ~2 * 5 * 2, or about 20), to 1.
This should also improve the qualcomm performance, since qualcomm has not taken advantage of the split build feature.
It is the only vendor with its own unique (arm64) architecture, and doesn't benefit from another machine building the relevant binaries.
If the dedicated builder can build the binaries targeting arm64, then qualcomm can dramatically reduce the time spent building for tests.
This issue tracks setting up a dedicated frequent builder that builds dxc/clang targeting x64 and arm64, and distributes these builds to all test runners via an artifact store. The test runners ought to be capable of pulling down the right artifacts, and test jobs depend on the Build job; a failed build automatically skips dependent test jobs.