Skip to content

Frontend: decode escaped identifiers in one span-based pass #538

Description

@chrisbbreuer

Parent: #496
Related: #493, #460, #461, #536, #537

Problem

Lexer.lexIdentName scans every escaped identifier once to validate Unicode escapes and IdentifierStart/IdentifierPart, then decodeIdent rescans the entire raw identifier and appends every non-escaped byte individually into arena storage. Generated code using escaped bindings/properties, obfuscator output, conformance workloads, and adversarial wide parameter lists therefore pay two full scans plus avoidable per-byte growth/copy work.

Scope

  • Add parser-only 1K/2K/4K escaped-identifier growth rows with frozen source construction, decoded-name structural checks, warmup, and timed parse boundary.
  • Preserve the existing allocation-free source slice for unescaped ASCII/Unicode identifiers.
  • At the first escape, materialize decoded storage once, copy raw spans in batches, decode each Unicode escape during the validating scan, and remove the redundant second scan.
  • Preserve escaped-keyword distinction, IdentifierStart/IdentifierPart validation, Unicode scalar bounds, \\uXXXX/\\u{X} behavior, non-ASCII raw identifiers, diagnostics/source positions, private names, and source lifetime.

Security and correctness

  • Exact decoded bytes, not hashes, determine identifier identity.
  • Invalid escapes and invalid decoded start/continue characters must retain exact SyntaxError classification/location.
  • Decoding must remain OOM-propagating and failure-atomic; no global cache or attacker-predictable intern table may be introduced.
  • Do not special-case benchmark names, loosen Unicode validation, change escaped keyword semantics, or defer early errors.

Acceptance

  • Benchmark source and decoded-name structural checks land before lexer changes.
  • Pointer/allocation witnesses prove unescaped identifiers remain borrowed and escaped identifiers own exact decoded bytes.
  • 1K/2K/4K escaped rows show bounded near-linear work and materially lower instruction/allocation cost versus the exact parent; an ordinary control remains unchanged.
  • Lexer/parser/frontend suites, escaped keyword/private/property/binding cases, malformed Unicode/OOM diagnostics, forced execution paths, full units, and affected test262 identifier subtrees pass with no regressions.

Metadata

Metadata

Assignees

No one assigned

    Labels

    performanceMeasured runtime, memory, scaling, or build performance

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions