[CI] Shard Async Engine/Inputs/Utils/Worker/Config (CPU) into 4 timing-balanced groups - #52348
Draft
khluu wants to merge 3 commits into
Draft
[CI] Shard Async Engine/Inputs/Utils/Worker/Config (CPU) into 4 timing-balanced groups#52348khluu wants to merge 3 commits into
khluu wants to merge 3 commits into
Conversation
…g-balanced groups Split the 15 sequential commands of async-engine-inputs-utils-worker-config-cpu (~53m wall in build 83851) across parallelism: 3 using per-shard command guards. Groups are balanced on measured per-command runtime, not command count: shard 0 ~17.1m (tokenizers_ 16.5m + fast env tests), shard 1 ~17.1m (multimodal, renderers, reasoning, pooling, ray), shard 2 ~16.8m (tool_parsers, config, parser, transformers_utils, inputs/outputs/sampling). Expected wall per shard ~19m incl. ~1.9m container setup. Timeout 65 -> 25. Command set and flags are unchanged; every command runs on exactly one shard. Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
…-way Build 83920 measured walls 20.48/19.54/19.17m: the tokenizers_ shard missed the <20m gate by 0.48m. Reconciliation shows total command time ~53m + ~1.9m per-shard setup, so even a perfectly balanced 3-way split floors at ~19.6m - three-way cannot produce real margin while tokenizers_ (17.4m) stays whole. tokenizers_ is 1369 nodes with the longest at 0.20m (no single-test floor), so it is pytest-sharded 2-way across shards 0-1 (simulated sha256 bucket split: 8.5m/9.0m) and the remaining 14 commands are statically grouped by measured runtime: shard 0 ~12.5m, shard 1 ~13.4m, shard 2 ~13.2m, shard 3 ~13.4m of command time, ~15.3m walls incl. setup. parallelism 3 -> 4 (cpu-small, no GPU); every non-tokenizers_ command still runs exactly once; tokenizers_ union is exact 2-way pytest-shard by construction. Timeout stays 25. Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
An index outside 0-3 would skip every per-shard guard and let the job pass empty; validate BUILDKITE_PARALLEL_JOB explicitly before any test command and exit 1 with a diagnostic otherwise. Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
khluu
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Phase 2 of the CI job-splitting effort (45–60m jobs → shards <20m).
async-engine-inputs-utils-worker-config-cpuran 53.0m in build 83851.Validation (terminal, build 83933)
Targeted build 83933 on this exact head: 4/4 passed, walls 14.33 / 14.68 / 15.17 / 15.52m — max 15.52m with real margin (3.4× vs baseline).
Running 450, shard 1Running 476, 450 + 476 = 926 = collected, exact and disjoint by construction.History
First head (3-way) validated green in build 83920 but was rejected on the strict gate (walls 20.48/19.54/19.17m): total command time ≈53m + ~1.9m setup means even a perfectly balanced 3-way split floors at ≈19.6m.
tokenizers_(longest node 0.20m — no single-test floor) was therefore split 2-way and the job moved toparallelism: 4.Notes
gh pr listfor the step key and file).