encoding: streaming-first Go deserialization redesign

[trippwill]

Summary

Complete redesign of the Go deserialization surface based on the deserialization design exploration. Replaces the old Unmarshal / Decoder.UnmarshalNext / Decoder.More API with a streaming-first, iterator-based architecture.

Target: redesign/deserialization (integration branch — .NET redesign will follow separately)

---

New API

Tier 1 — UnitReader (primary streaming interface):

ur := encoding.NewUnitReader(r)
defer ur.Close()
for prop := range ur.Properties() {
    switch prop.Name {
    case "config":
        cfg, err := encoding.ReadValue[Config](ur)
    case "events":
        for event := range encoding.PackItems[LogEvent](ur) {
            process(event)
        }
    default:
        ur.Skip()
    }
}
if err := ur.Err(); err != nil { ... }

Tier 2 — Materialization (sugar):

cfg, err := encoding.UnmarshalNew[Config](data)

Tier 3 — Custom converters:

ur := encoding.NewUnitReader(r, encoding.RegisterConverter[Instant](myConverter))

Policies:

encoding.UnmarshalNew[Config](data,
    encoding.UnknownFields(encoding.ErrorUnknown),
    encoding.Duplicates(encoding.ErrorDupes),
    encoding.MissingFields(encoding.ErrorMissing),
)

Performance

Event.Value changed from string to []byte with borrow semantics (valid until next Decode()). Scalar readers write directly to a reusable valBuf via byteAppender interface. unsafe.String for zero-copy parsing of non-string scalars.

Benchmark (Fin 10K)	Before	After	Improvement
Decode ns/op	—	18.1ms	Beats JSON (20.5ms)
Decode B/op	—	842KB	vs JSON 14.6MB
Decode allocs	—	97K	vs JSON 380K

Removed

• Unmarshal(data, &v) → UnmarshalNew[T](data)
• Decoder.UnmarshalNext / More → UnitReader + iterators
• Decoder.SetSpec / Spec / ParseSpec — deferred to future
• unmarshal_visitor.go, reader_reflect.go, spec.go — replaced by event-based path
• CLI --spec flag — deferred
• 9 dead string-returning scalar readers (382 lines)

Spec change

• Error codes compacted: removed reserved code 2, renumbered 3→2, 4→3, 5→4 (Go + .NET + spec)

Naming

• StatementReader → UnitReader, Statement → Property, Statements() → Properties()
• StructFields (tag introspection) → ReflectStructFields; StructFields now names the navigation helper

Quality

• Test coverage: 77.6% (+6.3pp)
• 5 fuzz tests: FuzzDecode, FuzzUnmarshalNew, FuzzReadString, FuzzParseIntLiteral, FuzzParseType
• Weekly fuzz CI workflow (.github/workflows/fuzz.yml)
• Lint rules tightened: +gosec, +nilerr, +exhaustive; fixed real G115 overflow in marshal.go
• Financial benchmark dataset (1K/10K trades with map pack, embedded composites, non-string heavy)
• Site + README updated with new API examples
• Design doc: design/deserialization-design.md

All tests pass with -race. golangci-lint run clean.

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	pakt	9ec9157	Commit Preview URL Branch Preview URL	Apr 13 2026, 03:02 AM

[codecov]

Codecov Report

❌ Patch coverage is 67.39130% with 360 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.80%. Comparing base (f81685b) to head (9ec9157).
⚠️ Report is 1 commits behind head on redesign/deserialization.

Files with missing lines	Patch %	Lines
encoding/read_value.go	65.26%	68 Missing and 31 partials ⚠️
encoding/reader_scalar_buf.go	62.05%	45 Missing and 40 partials ⚠️
encoding/navigation.go	59.82%	32 Missing and 13 partials ⚠️
encoding/unmarshal_new.go	70.79%	21 Missing and 12 partials ⚠️
encoding/pack_iter.go	52.00%	16 Missing and 8 partials ⚠️
encoding/converter.go	64.40%	12 Missing and 9 partials ⚠️
encoding/unit_reader.go	79.20%	16 Missing and 5 partials ⚠️
encoding/reader.go	73.52%	9 Missing ⚠️
encoding/reader_value_helpers.go	62.50%	5 Missing and 4 partials ⚠️
encoding/event.go	83.72%	3 Missing and 4 partials ⚠️
... and 3 more

Additional details and impacted files

@@                     Coverage Diff                      @@
##           redesign/deserialization       #5      +/-   ##
============================================================
+ Coverage                     70.01%   72.80%   +2.78%     
============================================================
  Files                            40       45       +5     
  Lines                          7074     6835     -239     
  Branches                        463      463              
============================================================
+ Hits                           4953     4976      +23     
+ Misses                         1586     1372     -214     
+ Partials                        535      487      -48     

Flag	Coverage Δ
dotnet	72.84% <ø> (ø)
go	72.76% <67.39%> (+5.55%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
dotnet/src/Pakt/PaktException.cs	100.00% <ø> (ø)
encoding/bytesource.go	86.66% <ø> (+10.47%)	⬆️
encoding/decoder.go	92.30% <ø> (+20.78%)	⬆️
encoding/options.go	100.00% <100.00%> (ø)
encoding/tags.go	85.21% <100.00%> (ø)
encoding/unmarshal.go	68.64% <100.00%> (+11.34%)	⬆️
encoding/marshal.go	67.79% <60.00%> (-0.90%)	⬇️
encoding/reader_state.go	77.36% <80.00%> (-0.04%)	⬇️
encoding/errors.go	91.66% <82.35%> (+49.56%)	⬆️
encoding/event.go	86.48% <83.72%> (+3.15%)	⬆️
... and 9 more

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

This PR redesigns the Go deserialization surface in encoding/ around a streaming-first UnitReader + iter.Seq iterators, replacing the prior Unmarshal / Decoder.UnmarshalNext / More APIs and aligning related docs/spec/test infrastructure with the new approach.

Changes:

• Introduces UnitReader, ReadValue[T], PackItems[T], UnmarshalNew[T], options/policies, and a converter hook surface.
• Switches Event.Value to borrowed []byte (with helper accessors) and updates tests/helpers accordingly.
• Removes spec projection APIs and updates CLI, spec error codes, documentation, CI/lint/fuzz workflows.

Reviewed changes

Copilot reviewed 56 out of 56 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
spec/pakt-v0.md	Renumbers/updates normative error category codes.
site/layouts/index.html	Updates homepage examples/messaging to highlight UnitReader + pack streaming.
site/content/docs/install.md	Rewrites install/usage docs around UnitReader, PackItems, UnmarshalNew.
README.md	Updates README examples to new deserialization APIs.
encoding/unmarshal.go	Refactors reflect setters; removes old Unmarshal surface and unsafe string retention issues.
encoding/unmarshal_visitor.go	Removes old visitor-based unmarshal implementation.
encoding/unmarshal_next_test.go	Removes tests for removed UnmarshalNext / More API.
encoding/unmarshal_new.go	Adds UnmarshalNew* materialization API over UnitReader.
encoding/unmarshal_new_test.go	Adds coverage for UnmarshalNew behavior and policies.
encoding/unit_reader.go	Adds primary streaming UnitReader property iterator + skipping.
encoding/unit_reader_test.go	Adds tests for UnitReader navigation behavior.
encoding/tags.go	Renames StructFields to ReflectStructFields.
encoding/tags_test.go	Updates tests for ReflectStructFields rename.
encoding/spec.go	Removes spec projection parsing/skip machinery.
encoding/reader_value_test.go	Updates value tests for borrowed []byte event values.
encoding/reader_value_helpers.go	Switches scalar direct-reading to borrowed []byte buffer path.
encoding/reader_test.go	Updates scalar parsing tests to use new readScalarDirect semantics.
encoding/reader_state.go	Threads full statement type to statement-start events; adjusts map key value encoding to []byte.
encoding/reader_scalar_buf.go	Adds reusable scalar buffer writers (read*To).
encoding/reader_reflect.go	Removes old reflect-based scalar readers.
encoding/read_value.go	Adds event-driven reflective value materialization + converter dispatch.
encoding/pack_test.go	Updates pack tests to new APIs and ValueString() usage.
encoding/pack_iter.go	Adds PackItems / PackItemsInto iterator helpers.
encoding/pack_iter_test.go	Adds tests for pack iterators incl. early-break draining.
encoding/options.go	Adds deserialization options/policies and converter registry.
encoding/navigation.go	Adds navigation iterators for composites (struct/list/map/tuple).
encoding/navigation_test.go	Adds tests for navigation iterators and draining behavior.
encoding/marshal.go	Fixes uint→int64 overflow and updates struct-field reflection helper name.
encoding/integration_test.go	Updates integration tests for borrowed event values + removed spec projection.
encoding/fuzz_test.go	Adds fuzz targets for decoder, unmarshal, string/int/type parsing.
encoding/event.go	Changes Event.Value to []byte, adds ValueString()/nil detection, updates JSON marshal/unmarshal.
encoding/event_test.go	Updates tests for new Event.Value representation.
encoding/errors.go	Renumbers ErrorCodes and adds DeserializeError.
encoding/errors_test.go	Adds tests for DeserializeError formatting/unwrapping and error codes.
encoding/encoder_test.go	Updates encoder round-trip tests for borrowed event values.
encoding/bytesource.go	Removes bytesSource used by old Unmarshal fast path.
encoding/bench_test.go	Updates benchmarks to new APIs and adds financial dataset benchmarks.
encoding/doc.go	Updates package docs for new deserialization surface and removes spec projection docs.
encoding/decoder.go	Removes spec projection and incremental unmarshal APIs from Decoder.
encoding/decoder_test.go	Updates decoder tests for borrowed event values.
encoding/converter.go	Adds converter interfaces and registration options.
encoding/converter_test.go	Adds tests for converter surface and ValueReader helpers.
dotnet/tests/Pakt.Tests/CoreTypeTests.cs	Updates .NET tests to match renumbered error codes.
dotnet/src/Pakt/PaktException.cs	Updates .NET error code enum values (removes reserved code).
codecov.yml	Lowers patch coverage target threshold.
cli.go	Removes --spec flag usage and spec loading from CLI commands.
cli_test.go	Removes CLI spec projection test.
.mise.toml	Pins golangci-lint tool version.
.golangci.yml	Enables additional linters and configures gosec/exhaustive settings.
.github/workflows/fuzz.yml	Adds scheduled fuzzing workflow for Go fuzz targets.
.github/workflows/ci.yml	Switches CI to mise-based Go + golangci-lint install; disables fail-fast.

[Copilot]

Pull request overview

Copilot reviewed 56 out of 56 changed files in this pull request and generated 8 comments.

[Copilot]

Pull request overview

Copilot reviewed 56 out of 56 changed files in this pull request and generated 7 comments.