Fix decoder performance and encoding edge cases

[thesyncim]

Summary

This PR fixes fifty-two independently reproduced correctness, compatibility, streaming, ownership, allocation, diagnostics, and performance defects. The fixes cover stable portable builds and the pinned Go 1.27 portable/SIMD lanes. Option-only behavior is compile-time specialized so default plans keep their dense dispatch, compact plan layout, cache domains, and allocation profile.

Bug ledger

1. Public escaped-string decoding performed a redundant full validation pass. AppendDecodedJSONString scanned escaped input once to validate it and again to decode it, making the defensive public API unnecessarily expensive. It now decodes and validates in one checked sparse/dense pass, rolls malformed input back losslessly, and has separately tuned portable and Go 1.27 SIMD implementations. Parser-backed trusted decoding remains unchanged.

2. Pointer-receiver marshalers were suppressed below addressability-restoring boundaries. A non-addressable map/interface struct value incorrectly kept slice elements and pointer pointees non-addressable. The cold fallback now preserves non-addressability through structs and arrays while restoring ordinary method dispatch below slices and pointers, matching encoding/json without adding a hot-path branch.

3. Escaped object-probe fallback could allocate repeatedly. When caller storage covered the hash table but not escaped-key side slots, repeated append growth violated the one-allocation fallback contract. The side storage now grows once for the maximum remaining escaped members.

4. A named inline tag was treated as a catch-all. json:"named,inline" incorrectly flattened the map. Only the exact empty-name spelling json:",inline" now enables the extension.

5. An invalid explicit tag name was mistaken for the empty-name inline opt-in. Tag-name validation normalized an invalid name to empty before inline classification. Classification now uses the raw spelling, while ordinary field-name fallback still uses the normalized name.

6. Dynamic encoding lost InlineFields. Encoder[any] and non-empty interface plans compiled runtime concrete values without the option, so catch-all members were emitted under the map field instead of flattened. Inline dynamic interfaces now use dedicated compile-time opcodes and an option-partitioned cache.

7. Dynamic decoding lost InlineFields. Decoding through any or a non-empty interface into an existing concrete pointer compiled that concrete type without the option, so unknown members were skipped or rejected instead of entering the catch-all. Decode plans now use the corresponding cold inline specialization.

8. null in a reused byte-array decode cleared an existing element. The array form of []byte decoded through a temporary zero byte, so [1,null] over [7,8,9] became [1,0] instead of the encoding/json merge result [1,8]. Elements now decode directly into reused backing slots.

9. The stream writer accepted impossible write counts. An io.Writer returning n > len(p), nil was treated as success. Any count other than the requested length now becomes sticky io.ErrShortWrite.

10. Encoder compilation accepted decode-only map keys. A key implementing only encoding.TextUnmarshaler passed the encode capability check and failed too late. Encoder compilation now requires the value-side encoding.TextMarshaler contract.

11. Decoder compilation accepted encode-only map keys. A key implementing only encoding.TextMarshaler passed the decode capability check and failed too late. Decoder compilation now requires pointer-side encoding.TextUnmarshaler.

12. Decode-only byte element methods changed encoding shape. A []byte-kind slice whose element only implemented unmarshal methods was emitted as a numeric array instead of base64. Byte-slice method detection is now directional.

13. Encode-only byte element methods changed decoding shape. A []byte-kind slice whose element only implemented marshal methods rejected the standard base64 string form. Decode classification now ignores encode-only methods.

14. Decode-method byte slices could not support both standard JSON forms. For an element with unmarshal methods, base64 strings must bypass the element method while numeric arrays must invoke it. The decoder now represents this as a specialized byte plan: string/null stays on the base64 path, and only array input enters the element-method loop.

15. Replace passed stale state into json.Unmarshaler. A present custom JSON field observed the reused destination rather than the zero value promised by fresh-destination semantics. Replace now zeros the complete receiver before dispatch.

16. Replace passed stale state into encoding.TextUnmarshaler. Text receivers had the same stale-state leak. They now receive a complete zero value, including the non-pointer null path.

17. Replace passed stale state into native decode hooks. UnmarshalVibeJSON receivers bypassed the Replace lifecycle. Native hooks now receive zero state before cursor dispatch in both portable and SIMD builds.

18. Absent custom-decoder fields retained hidden state. Structural reset only visited exported JSON fields, so unexported/internal state owned by a custom decoder survived when the field was absent. Reset plans now zero the custom value as one complete unit.

19. null retained a reused time.Time. The time fast path ran before Replace zeroing, and time.Time.UnmarshalJSON(null) is a no-op. The fast path now starts from the true zero value.

20. Quoted "null" retained reused ,string scalars. Pointer fields were cleared, but scalar numbers kept their prior value. Replace now zeros the scalar while default merge semantics remain unchanged.

21. Replace merged an existing pointer held by an empty interface. An any destination containing *T decoded into that pointee instead of selecting the dynamic shape a fresh nil interface would produce. Replace now ignores the old dynamic value and rebuilds from JSON.

22. Replace merged an existing pointer held by a non-empty interface. A reused implementation pointer made otherwise undecodable JSON succeed. Replace now clears the interface first and reports the same error as a fresh nil non-empty interface.

23. Absent flattened embedded pointers remained non-nil. Replace cleared their exported children but retained &zero instead of restoring a nil embedded pointer. Replace plans now use the existing whole-record reset route for embedded pointer hops.

24. Aliased pointer fields overwrote one another. Two fields sharing one pointee reused it sequentially, so the later field changed the earlier result. Replace reuses a unique pointee allocation-free and detaches only a duplicate alias.

25. Overlapping slice fields overwrote one another. Two slice windows into the same backing array allowed the later decode to corrupt the earlier field. Operation-local range tracking preserves unique capacity and detaches only overlapping storage.

26. Shared map fields destroyed one another. Clearing and decoding the later field mutated the same map already returned through the earlier field. Replace now detects map identity and allocates only for the duplicate owner.

27. DecodeArray allowed aliases to cross reused elements. Shared nested maps/slices/pointers in old element storage let a later element overwrite an earlier decoded result. One operation-local tracker now spans the repeated element graph without bulk-clearing the outer capacity.

28. Native encode hooks on byte element types were silently bypassed. A pointer MarshalerSimd on a uint8-kind element still selected base64 output. Directional byte classification now selects the element array route when a native encode hook exists.

29. Native decode hooks on byte element types were silently bypassed. A pointer UnmarshalerSimd on a uint8-kind element rejected/ignored numeric-array hook dispatch. Array input now invokes the hook while base64 input deliberately retains the byte-slice form.

30. String-tagged pointer fields bypassed Replace alias tracking. Quoted numbers took a direct strconv path and quoted bool/string values decoded through a synthetic cursor that dropped operation state. Aliased ,string pointers therefore overwrote one another. Both paths now share the parent operation tracker and detach only duplicate owners.

31. Repeated JSON members falsely looked like cross-field aliases. Decoding the same reference field twice registered its reused storage twice, so the second occurrence detached from itself and forced two allocations per warm decode. Tracker entries now carry destination ownership: repeated members update their own record, while storage shared by different fields or elements still detaches.

32. Malformed JSON inside a ,string field reported coordinates in a temporary buffer. A nested string syntax failure could claim line 1 and a byte offset unrelated to the original document. The error path now maps decoded offsets back through simple escapes, Unicode escapes, and surrogate pairs, then rebuilds exact byte/line/column coordinates against the outer input.

33. Replace retained stale claims after storage was released or relocated. Reference ownership was recorded before decoding but not refreshed after null, slice growth, quoted "null", nested element relocation, or a composite reset. Later fields could detach from storage that was no longer shared, and repeated nested owners could allocate on every warm decode. Replace now gives each logical owner a hierarchical scope, clears that scope and its descendants before a repeated decode, removes released claims, and refreshes live storage identities only after successful decoding.

34. The stream reader trusted impossible read counts. A broken io.Reader returning a negative count or more bytes than the supplied buffer could trigger a slice-bounds panic or advance the cursor beyond valid storage. Reader.fill now validates the count with one unsigned bounds check before mutating state and reports a sticky error for both contract violations.

35. Missing-field reset retained stale Replace reference claims. When a duplicate object member first decoded nested reference storage and a later occurrence omitted that field, the reset cleared the value but left its ownership claim live. A later sibling sharing the old storage detached unnecessarily and warm decoding allocated. Missing-field reset now releases each reference scope before zeroing it, restoring zero-allocation reuse.

36. DecodeNext hid a deferred source error behind an incomplete-value syntax error. If a reader returned a partial JSON value together with a non-EOF error, typed streaming reported parser EOF/syntax diagnostics instead of the underlying source failure. It now gives the source error precedence only when the buffered value is genuinely incomplete, matching encoding/json.

37. Replace ignored aliases between pointers and slice backing storage. Reference tracking compared only equal storage kinds, so a pointer into a sibling slice could overwrite the already-decoded slice result. Pointer pointees now carry their writable span and pointer/slice storage uses range overlap checks.

38. A lone reused pointer could alias ordinary storage inside its destination. Single-pointer plans deliberately skipped the cross-reference tracker, but safe Go values such as dst.Value = &dst.Scalar could still make pointer decoding overwrite a sibling field. Decoder compilation now proves whether pointee and destination layouts can physically overlap and enables a compact destination-range guard only for those plans; Decode, DecodePrefix, and DecodeArray are covered.

39. Pointers with different base addresses could still overlap. Exact pointer identity missed interior aliases such as a *[2]int and a *int pointing at its second element. Pointer ownership now uses overflow-safe memory ranges, preserving a unique outer pointee while detaching only the overlapping interior owner.

40. A reused pointer could target an object containing the active destination. In recursive safe-Go layouts, decoding an inner record could reuse a pointer to the outer object that contains it, resetting or overwriting the destination currently being decoded. The destination guard now checks overlap in both directions.

41. Quoted non-numeric pointer decoding dropped destination-alias context. The synthetic cursor used for ,string bool and string scalars inherited operation state but not the active destination range, so an aliased pointer could overwrite a sibling field. Quoted subcursors now carry the same compact destination guard as their parent.

42. Reader source-error precedence hid syntax errors already present in the buffer. Giving a deferred read error unconditional precedence made malformed JSON bytes look like transport failures. The error path now distinguishes incomplete literal, number, string, escape, Unicode, and surrogate prefixes from definite malformed input: source errors win only for truncation, while known syntax errors remain precise.

43. A reused slice could alias ordinary storage inside its destination. Destination-overlap analysis covered pointer pointees but not slice backing arrays, so safe Go layouts such as Values = Fixed[:] let decoding the slice overwrite a sibling fixed array that had already been decoded. Compile-time layout analysis now identifies slice/byte backing that can originate inside the destination, and the existing range guard detaches only that alias.

44. A truncated UTF-8 rune hid the underlying Reader error. Partial two-, three-, and four-byte rune prefixes inside a JSON string were classified as definite syntax errors even though another read could complete them. Incomplete-value diagnosis now uses utf8.FullRune, so non-EOF source errors win only for genuinely truncated rune prefixes while complete malformed UTF-8 remains a syntax error.

45. Replace retained fields that JSON can never select. Unexported fields, json:"-" fields, and fields removed by dominance conflicts survived destination reuse, diverging from a fresh decode. Replace compilation now builds a selective ignored-field reset program, including nested embedded layouts, while default decoders retain their original plan and dispatch.

46. Any embedded pointer forced whole-record reset and discarded reusable storage. The former fallback nilled every embedded pointer before decoding, which also threw away present nested slices, maps, and pointees. Replace plans now compile a presence mask for every embedded-pointer prefix: present prefixes reuse unique storage, absent prefixes return to nil, aliases still detach, and warm present-field decoding remains allocation-free.

47. DecodeNext enforced MaxValueBytes after mutating the destination. A framed oversized value was decoded first and rejected afterward, leaving caller state changed despite failure. The limit is now checked against the known frame extent before typed decoding begins.

48. Replace discarded inline catch-all map buckets and could preserve cross-owner aliases. Catch-all maps were eagerly reset, losing reusable buckets, while shared catch-all/sibling maps and reused DecodeArray elements could still overwrite one another. The first unknown member now lazily clears and reuses a unique map, detaches only duplicate owners, refreshes ownership after growth, and restores an absent catch-all to nil; warm unique reuse is allocation-free.

49. Terminal scalar/source-error precedence diverged across Go stream contracts. A scalar ending in the same read as a non-EOF source error could be committed when the active encoding/json version still required a confirming boundary, or rejected after the standard library had begun committing complete strings and literals. Build-tagged compatibility logic now follows Go through 1.26 versus Go 1.27 exactly; numbers remain boundary-ambiguous, closed containers still commit, and typed streaming rejects before destination mutation.

50. Replace had a correctness and reuse cliff above 64 JSON fields. Wide records could not represent presence in uint64, so the fallback reset the entire record and discarded present late-field pointers, slices, and nested storage. A dedicated Replace executor now uses retained operation-local bitsets for every field and embedded-pointer prefix. Recursive wide records receive independent scratch slots, warm decoding allocates zero bytes, and ordinary narrow records retain their original one-word loop.

51. Inline-map key ownership was sampled after decoding its value. A key could alias caller input when parsed, then the value could trigger a one-time private source copy; checking ownership afterward falsely treated the earlier key as moved, so later caller mutation corrupted the retained map key. Ownership is now captured at key creation, source-backed keys are remapped by offset into the private copy without allocation, and escaped arena keys are cloned when required.

52. DecodeNext retained and copied the entire buffered suffix for each owned value. Although framing had already found one value, typed decoding received buf[start:end], so a string in an early NDJSON record could copy every later buffered record, producing quadratic work and oversized retention. Typed decoding now receives the exact framed extent. The stream gate improves by 2.87% and the regression test bounds a tiny value before a 1 MiB suffix to under 4 KiB allocated per operation.

Performance design

• Default encode/decode dispatch and cache domains remain unchanged; Replace-only reference opcodes live outside the dense generated operation range.

• typedNode remains 272 bytes and decoderCursor remains exactly one 64-byte cache line; each Replace reference record remains 32 bytes.

• Unique pointer, slice, byte-slice, and map storage is reused. Duplicate or overlapping aliases alone detach.

• Destination-overlap checks are enabled only when compile-time Go layout analysis proves a pointer pointee can overlap its root; ordinary one-pointer Replace and all default decoders avoid the tracker.

• Alias tracking uses sixteen inline GC-visible reference slots. Wider graphs grow retained overflow storage once and return to zero allocations after warm-up; no Go pointer is hidden in uintptr.

• Tracker ownership distinguishes repeated writes to one destination from true cross-destination aliases without adding work to default plans.

• Hierarchical Replace scopes clear stale descendants when owners repeat or move; the scope ID occupies existing padding, so each reference record remains exactly 32 bytes on 64-bit targets.

• Slice/map/byte-slice scope maintenance is behind one cold Replace helper, keeping the shared inline-kind dispatcher compact.

• DecodeArray shares the operation tracker across elements instead of clearing reusable capacity.

• Native byte-element hook checks happen at compile time and are direction-specific, so the opposite direction keeps standard base64 behavior.

• Custom-receiver zeroing is confined to Replace plans; default method dispatch does not gain an option branch.

• Quoted syntax-coordinate reconstruction executes only after a nested parse failure; successful decoding performs no offset scan and allocates nothing for diagnostics.

• Wide or ignored-field Replace records use a dedicated retained presence executor. Nested records lease independent bitsets, while ordinary records keep the original unconditional one-word seen update and compact 272-byte plan node.

• Uncommon Replace flags occupy unused high bits of existing shape metadata, avoiding any plan-layout growth. Cold compilation helpers live in the Replace compilation unit so the generated hot record decoder keeps its cache-line placement.

• Inline catch-all maps clear lazily on the first unknown member and reuse unique buckets; absent maps return to nil without adding a default-plan branch.

• Streamed typed decoding consumes the exact known frame, eliminating quadratic suffix copies without an additional scan or allocation.

• Go-version source-boundary behavior is selected at build time; neither portable nor SIMD stream loops pay a runtime version check.

Validation

• full stable, pinned Go 1.27 portable, and pinned Go 1.27 SIMD test suites

• focused incomplete-versus-malformed stream error differential coverage against encoding/json, including literals, numbers, escapes, Unicode, and surrogate pairs

• focused permanent regressions for every ledger item, including empty/non-empty interfaces, cross-field/cross-element aliases, repeated members, quoted pointer aliases, released/grown/relocated Replace storage, nested composite resets, impossible Reader counts, native hook directionality, hidden receiver state, and escaped inner-error coordinates

• zero-allocation warm reuse contracts for unique pointers, repeated reference members, and reference graphs wider than the sixteen-slot inline tracker

• all 15 fuzz targets at 1000 executions per target under SIMD hook-integrity mode; extended campaigns include 678,959 decoder differential executions plus millions of encoder and containment executions

• full race and strict SIMD checkptr=2 suites

• vet, SA* staticcheck, generated-file reproduction, module/test-contract, and unsafe-inventory checks

• Linux amd64, arm64, 386, and s390x cross-builds

• GC/lifetime stress matrix

• exact-source interleaved portable gate against main (8 rounds, 250 ms): DecodeSmall, reused-map decode, zero-allocation DecodeArray, and large-index build were all statistically unchanged; geomean was -0.43%, with identical B/op and allocs/op

• final post-lifecycle gates: reused-map decode was statistically unchanged over 12 rounds at 500 ms (p=0.219), and Reader/Cursor was unchanged over 10 rounds at 400 ms (p=0.393), with identical B/op and allocs/op

• post-range-alias gates against the pre-change checkpoint over 12 rounds at 500 ms: one-pointer Replace (p=0.054), DecodeSmall (p=0.326), reused-map decode (p=0.921), Reader/Cursor (p=0.178), and streamed NDJSON (p=0.178) were statistically unchanged with identical B/op and allocs/op

• strict checkptr=2, race, 14-target fuzz smoke, and extended 10-second encoder/stream differential fuzz campaigns after the destination-range changes

• public string decoder gates remain at 0 B/op and 0 allocs/op, with 19.7% portable and 20.2% SIMD geomean speedups over the pre-fix implementation

• ten new permanent regression groups cover destination-backed slices, UTF-8 truncation, ignored fields, selective embedded-pointer prefixes, pre-mutation size limits, inline-map reuse/aliasing, versioned scalar boundaries, wide records, key ownership, and exact-frame retention

• final stable Go, pinned Go 1.27 portable, and pinned Go 1.27 SIMD full-package suites after the complete checkpoint

• final focused race suites on stable and SIMD plus strict SIMD checkptr=2 coverage for every new ownership/range/stream contract

• final fuzz smoke: all 14 portable targets and all 15 SIMD targets at 500 executions per target, including stream chunk equivalence, structural parity, decode trust, encoder parity, lifecycle operations, and SIMD scanner parity

• final vet, SA* staticcheck, generated-file reproduction, test-contract, unsafe-inventory, and Linux 386/s390x/SIMD-amd64 cross-build checks

• final isolated pinned-SIMD gate against commit 5354ca58 (10 alternating rounds, 400 ms): pointer reuse, inline decode, streamed Reader, plan compilation, DecodeSmall, and reused-map decode are within the 2% gate with identical B/op and allocs/op; StreamDecodeNextNDJSON is 2.87% faster (p=0.001) and overall geomean is +0.29%