Skip to content

test: free RAM between multi-package describes to fix CI OOM - #161

Closed
ryukzak wants to merge 12 commits into
mainfrom
ci/free-ram-multi-package-tests
Closed

test: free RAM between multi-package describes to fix CI OOM#161
ryukzak wants to merge 12 commits into
mainfrom
ci/free-ram-multi-package-tests

Conversation

@ryukzak

@ryukzak ryukzak commented May 20, 2026

Copy link
Copy Markdown
Collaborator

Why

The Canary & Release job has been OOM-killed (make: *** [Makefile:34: test] Killed, exit 143) on ubuntu-latest during test/api/write-generator/multi-package/sql-on-fhir.test.ts. The kernel SIGKILLs bun test once resident memory exceeds the runner's ~7 GiB. Re-runs are deterministic-fail.

Root cause

All three multi-package test files use top-level await inside describe:

describe("TypeScript Generation", async () => {
    const result = await new APIBuilder(...).generate();
    it("...", () => { expect(result...)... });
});

Two compounding problems:

  • Top-level awaits inside describe resolve during discovery, so every section's result is built upfront — three (or four) results live concurrently per file.
  • result is captured by every it() in the section, so GC can't reclaim it for the file's lifetime.

Each result.filesGenerated is a Record<string, string> of full file contents (CDA TS: 124 files, SQL-on-FHIR + R5 deps: 45+ R5 files plus generators × 3). With bunfig.toml setting parallel = true, multiple files' worth of residency pile up at the same time.

Fix

Move generation into beforeAll and null the result in afterAll (+ Bun.gc(true)) so each section's payload can be reclaimed before the next section starts:

describe("TypeScript Generation", () => {
    let result: GenerationReport;
    beforeAll(async () => { result = await new APIBuilder(...).generate(); });
    afterAll(() => { result = undefined as unknown as GenerationReport; Bun.gc(true); });
    it("...", () => { expect(result...)... });
});

Applied to:

  • test/api/write-generator/multi-package/cda.test.ts
  • test/api/write-generator/multi-package/local-package.test.ts
  • test/api/write-generator/multi-package/sql-on-fhir.test.ts

Verification

  • bun run typecheck — clean
  • bun run lint — only pre-existing warnings in src/typeschema/utils.ts
  • bun test — 257 pass, 0 fail, 75 snapshots intact
  • Local peak RSS (macOS) dropped to ~6.08 GiB; the OOM only reproduces on ubuntu-latest, so CI on this PR is the real check.

ryukzak added 12 commits May 20, 2026 12:54
Top-level `await` inside `describe` built every result during discovery
and pinned it via closure for the file's lifetime, causing OOM during
sql-on-fhir.test.ts on ubuntu-latest. Move generation into `beforeAll`
and null out the result in `afterAll` (+ Bun.gc) so each section's
~tens-of-MB of in-memory generated files can be reclaimed between
sections.
The previous commit moved generation from describe-discovery (no
timeout) into `beforeAll`, which inherits bunfig's `timeout = 30000`.
SQL-on-FHIR fetches `https://build.fhir.org/.../package.tgz` on cold
cache from GH runners and routinely exceeds 30s; the hook timed out,
got retried, and OOM-killed the suite.
`ccdaManager` was a module-level `await mkCCDARegister()` that loaded
the CDA package into memory on test process start and pinned it for the
whole run, even though only one describe in typescript.test.ts uses it.
Inline the call so CDA is constructed (and reclaimable) inside that
describe's scope instead.
`make test` was OOM-killed on ubuntu-latest during SQL-on-FHIR's build:
the test fetches the remote tarball and runs the TS + Python + C#
generators against R5 + SQL-on-FHIR, and its peak alone — on top of
whatever the rest of the suite already had resident — exceeded the
runner's ~7 GiB.

Split it into a separate `bun test` invocation so its heap starts
fresh, independent of the other test files' module-level state.
Even in its own bun process, sql-on-fhir.test.ts was OOM-killed: its
three describes (TypeScript / Python / C#) each built a full generator
output against R5 + the SQL-on-FHIR IG, and the three results held by
closure simultaneously exceeded the ubuntu-latest runner's ~7 GiB.

Split into three per-language files (matching the layout of the
existing typescript.test.ts / python.test.ts / csharp.test.ts unit
tests) and run each as a separate bun invocation in the Makefile, so
peak memory is bounded to one generator's output at a time.

Snapshots split alongside; describe paths are flat
("SQL-on-FHIR <lang> Generation") so snap keys are unchanged.
`oven-sh/setup-bun@v2` defaults to latest stable. Bun 1.3.14 introduced
a memory regression that OOM-kills `make test` on `ubuntu-latest`; the
same suite passed in 62s on 1.3.13. Pin via `.bun-version` (read by
setup-bun automatically) so CI and local dev share the known-good
version.

Also document the SQL-on-FHIR per-language test split in CLAUDE.md so
the structure is not "cleaned up" back into one file.
The bare `oven-sh/setup-bun@v2` step does NOT auto-detect .bun-version;
the previous push still ran Bun 1.3.14. Add explicit
`bun-version-file: .bun-version` to every setup-bun invocation so the
file actually pins the version.
@ryukzak ryukzak closed this May 20, 2026
@ryukzak
ryukzak deleted the ci/free-ram-multi-package-tests branch May 20, 2026 13:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant