Skip to content

conformance.md

Latest commit

 

History

History
67 lines (43 loc) · 5.54 KB

conformance.md

File metadata and controls

67 lines (43 loc) · 5.54 KB
Raw
title test262 Conformance
description How zig-js is measured against the real test262 corpus.

test262 Conformance

zig-js is scored against the real tc39/test262 corpus from the pinned test262/ submodule through the configured runner. Progress is a number, not a vibe, but the number only covers tests the configured runner actually scores.

Sources. Numbers come from docs/.data/test262.json; runner scope (subtrees, skips, exclusions, timeouts) from conformance/test262.zig. Regenerate both with bun run docs:data.

How a test is run

The harness (conformance/test262.zig) walks the corpus and, for each .js file:

  1. Reads the frontmatter — the /*--- … ---*/ YAML block declaring flags (raw, onlyStrict, async, module), includes: harness files, and negative: expectations.
  2. Assembles the source — prepends "use strict" when onlyStrict, loads the real upstream harness/sta.js + assert.js plus any includes:, then appends the test body.
  3. Runs it in an isolated subprocess worker under a 30-second backstop timeout (120 s for a few known-slow sweeps), where a timeout is charged as a host failure. Each worker streams one <index>:<outcome> line per test. If a worker crashes or times out, the parent respawns it just past the failure — so a single segfault or hang can never zero out a run. The parent exits non-zero if any aggregate parse, runtime, host, negative, or skip count is non-zero.
  4. Scores the outcome on one of two axes.

Two axes

Valid and negative tests measure different things, so they are scored separately:

  • VALID — "can we run it?" pass + parse-fail + runtime-fail + host-fail. This is the headline metric.
  • NEGATIVE — "do we reject bad input?" pass-negative + fail-negative.

Skipped tests are excluded from both denominators. The configured runner currently has zero unsupported metadata skips; docs/.data/test262-skips.tsv is still generated as the audit file.

The configured runner also has zero excluded files; docs/.data/test262-excluded.tsv is still generated as the audit file. Two non-normative SpiderMonkey staging files (sm/regress/regress-602621.js and sm/lexical-environment/block-scoped-functions-annex-b-arguments.js) are removed from the configured corpus definition before scoring because their pending Annex B block-scoped-function expectations contradict the normative Annex B tests in test/annexB. They are not skip-list or exclusion-list entries. Proper-tail-call coverage, dynamic-import catch-target behavior, import-defer async-module behavior, plain modules, module+async/top-level-await graph-ordering tests, plain async tests, and CanBlockIsFalse tests are part of the runner where supported.

The denominator is the configured subtree list in conformance/test262.zig; directories not enumerated there are outside the measurement entirely — distinct from skipped (unsupported metadata) and excluded/removed (corpus-definition) files. So 100% means 100% of the configured corpus, not of every path under test262/test.

Current numbers

Axis Passing Total Rate
VALID (can we run it) {{ data.test262.valid.passing }} {{ data.test262.valid.total }} {{ data.test262.valid.percentage }}%
NEGATIVE (strictness) {{ data.test262.negative.passing }} {{ data.test262.negative.total }} {{ data.test262.negative.percentage }}%

Failure shape on the VALID axis: {{ data.test262.valid.parseFail }} parse-fail · {{ data.test262.valid.runtimeFail }} runtime-fail · {{ data.test262.valid.hostFail }} host-fail. Skipped: {{ data.test262.skipped }}.

Per-suite breakdown

The table below is populated only when docs/.data/test262.json was regenerated from a saved run that includes subtree lines. If the suite list is empty, do not invent rows; rerun or parse a saved transcript with bun run docs:data -- --from run.txt.

@foreach (data.test262.suites as suite) @endforeach
SuitePassingTotalRate
test/{{ suite.name }}{{ suite.passing }}{{ suite.total }}{{ suite.percentage }}%

Some rows are individual files or worker shards (e.g. …generated/.#0:20) rather than directories, reflecting how the runner sub-divides large subtrees.

These figures are read from docs/.data/test262.json, regenerated by bun run docs:data. Run the suite, regenerate, rebuild — the whole site reflects the new run.

Remaining High-Impact Work

The configured runner has zero skips, zero excluded files, and no VALID failure tail in the checked-in run. Keep that invariant from regressing; separately, audit anything outside the configured denominator:

  • When using a saved transcript, keep docs/.data/test262-run-*.txt and docs/.data/test262.json paired. Otherwise regenerate docs/.data/test262.json directly with bun run docs:data.
  • Keep docs/.data/test262-skips.tsv in sync with zig-out/bin/test262 --list-skips.
  • Keep docs/.data/test262-excluded.tsv in sync with zig-out/bin/test262 --list-excluded.
  • Treat removed corpus inputs as a corpus-definition audit: keep the list tiny, source-specific, and justified against a stronger normative test.
  • Keep README/docs claims tied to either docs/.data/test262.json, conformance/test262.zig, or a committed run transcript.