ci: speed up CI pipeline and parallelize tests#3833
Open
joheredi wants to merge 1 commit intoAzure:mainfrom
Open
ci: speed up CI pipeline and parallelize tests#3833joheredi wants to merge 1 commit intoAzure:mainfrom
joheredi wants to merge 1 commit intoAzure:mainfrom
Conversation
Three independent improvements to CI and test performance:
1. CI pipeline: Add pnpm store caching and targeted build templates
- New install.yml with Cache@2 for pnpm store and PUPPETEER_SKIP_DOWNLOAD
- New build-autorest.yml and build-typespec-emitter.yml for scoped builds
- Downstream CI jobs now build only the packages they need instead of
full monorepo builds
2. Modular scenario tests: Split into parallel shards
- Replace monolithic scenarios.spec.ts with 34 directory-level wrapper specs
- Enable mocha --parallel --jobs 4 for scenario test execution
- Add auto-generation script and verification guard to prevent drift
- New scripts: regen-scenario-specs, check:scenario-specs
3. Integration tests: Parallelize RLC and Modular suites
- Run RLC (port 3000) and Modular (port 3002) in parallel within
test:standard and test:azure since they use different ports and
output directories
- Make generation worker count configurable via GEN_WORKERS env var
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
joheredi
requested review from
JialinHuang803,
MaryGao and
qiaozha
as code owners
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three independent improvements to CI and test performance:
1. CI Pipeline: Targeted builds with pnpm caching
install.ymltemplate withCache@2for pnpm store andPUPPETEER_SKIP_DOWNLOAD=truebuild-autorest.ymlandbuild-typespec-emitter.yml— scoped build templates that only compile the packages each downstream job actually needsci.ymlso downstream jobs use targeted builds instead of full monorepopnpm buildnightly-prepare.ymlwith caching and targeted buildImpact: Every downstream CI job avoids rebuilding the entire monorepo. Combined with pnpm store caching, this reduces install+build time significantly per job.
2. Modular Scenario Tests: Parallel execution via file-level sharding
scenarios.spec.ts(which funneled all 282 scenario tests through one Mocha file) with 34 directory-level wrapper specs undertest/modularUnit/scenario-specs/scenarioHarness.tsmocha --parallel --jobs 4for scenario wrapper specsregen-scenario-specs) and verification guard (check:scenario-specs) to prevent new scenario directories from being silently missedImpact: Scenario tests now run across 4 parallel Mocha workers instead of a single serial file. Validated: 476 scenario tests passing, 69 core tests passing.
3. Integration Tests: Parallel RLC + Modular suites
test:standardandtest:azurenow run their RLC (port 3000) and Modular (port 3002) integration suites in parallel vianpm-run-all --parallelGEN_WORKERSenv var (defaults to 4)Safety verification:
tsp-spector server stop -p <port>is port-specific ✅temp/specs/is read-only during generation ✅Impact: ~2x speedup for the integration test phase within each CI job.
Files changed
install.yml,build-autorest.yml,build-typespec-emitter.yml,build.yml,ci.yml,smoke-test.yml,nightly-prepare.ymlscenarioHarness.ts,scenario-specs/*.spec.ts(34 files),modular-scenario-specs.ts,generate-modular-scenario-specs.ts,check-modular-scenario-specs.tspackage.json,gen-cadl-ranch.jsTesting
pnpm build✅pnpm format✅npm run unit-test:modular— 69 core + 476 scenario tests passing ✅npm run check:scenario-specs— 34 files up to date ✅pnpm --filter) validated for all 3 packages ✅