QA: mutation to 6 modules (≥80), continuous deep-fuzz + load/SLO harnesses (closes #14)#16
Merged
Merged
Conversation
…zz + load/SLO harnesses Closes the remaining Issue #14 items 1-3. Item 1 — mutation beyond the validator: - Stryker scope 1 -> 6 modules, all now >= the 80 break threshold. - mock/engine.ts 77.4% -> 82.2% (mock-engine-mutation.test.ts) and spec/drift.ts 75.6% -> 92.3% (drift-mutation.test.ts) via targeted killing tests; the other four (validate-response, resolve, scaffold, capture) were already >=80. - vitest.mutation.config.ts excludes the ~11s fuzz smoke from the mutation run (it was in the covering set and re-ran per mutant, so engine never finished). Item 2 — continuous fuzz to the 1M-exec budget: - qa/fuzz/deep-fuzz.mjs: feedback-driven (corpus grows on new output signatures), 7 targets, 1,000,000 execs OR --minutes each, crash/hang findings + persisted corpus. Scheduled in fuzz-deep.yml. Smoke: 140k execs, 0 findings. Fixed two harness bugs found along the way (global findings counter; jsonpath's correct throw on a malformed author path was wrongly flagged). Item 3 — throughput/latency/leak SLO: - qa/load/load-test.mjs + load.yml. Hard SLOs: zero errors and bounded post-GC heapUsed growth (the real leak signal — RSS only tracks the high-water mark). Latency gated on p95 (stable), not p99 (single GC spikes). mock ~2400 rps / p95 ~40ms, web ~740 rps static / p95 ~68ms, both leak-free. 326 tests green, typecheck 7/7.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the remaining #14 items — 1 (expand mutation), 2 (continuous fuzz), 3 (throughput-SLO load). With this, Issue #14 is fully addressed.
1. Mutation beyond the validator
Expanded the Stryker scope 1 → 6 modules, all now ≥ the 80 break threshold:
engine and drift were under 80 and lifted by targeted killing tests (
mock-engine-mutation.test.ts,drift-mutation.test.ts) asserting exact content-types, the validate-mode 400 body, route specificity, regex semantics,normalizeKeywhitespace handling, and sorted diff output.vitest.mutation.config.tsexcludes the ~11s fuzz smoke from mutation runs (it was in the covering set and re-ran per mutant, so engine never finished inline).2. Continuous fuzz to the 1M-exec budget
qa/fuzz/deep-fuzz.mjs— feedback-driven (keeps inputs that hit a new output signature, AFL-style corpus growth), 7 targets, up to 1,000,000 execs OR--minutesper target, crash/hang findings →qa/fuzz/findings.jsonl, corpus persisted toqa/corpus/. Scheduled weekly infuzz-deep.yml. Smoke: 140k execs, 0 findings. (Found+fixed two bugs in the fuzzer itself: a global-not-per-target findings counter, and treating jsonpath's correct throw on a malformed author expression as a crash.)3. Throughput / latency / leak SLO
qa/load/load-test.mjs+load.yml. Hard SLOs: zero errors and bounded post-GCheapUsedgrowth — the real leak signal. (I first gated on RSS, saw it climb 58MB, then realised RSS only tracks V8's high-water mark; post-GC heap is flat, +0.6MB over 47k requests → no leak.) Latency gated on p95 (stable), not p99 (a single GC spike blew p99 to 1469ms). Results: mock ~2400 rps / p95 ~40ms, web ~740 rps static / p95 ~68ms, both leak-free.Verification
326 vitest tests green, typecheck 7/7, all 6 mutation modules ≥80, deep-fuzz 0 findings, load SLOs met repeatably.