Standalone ClickHouse data-type parser (C++ + TypeScript port) - #885
Merged
Conversation
A faithful TypeScript port of the standalone C++ ClickHouse data-type parser in type-parser/mini-parser-extracted. The module layout tracks the C++ sources one-to-one (ast/lexer/parser/json), preserving the original control flow, branch ordering, and pos save/restore points. Output is verified byte-identical to the C++ chdt-parse across the full corpus (50 supported + 5 unsupported + edge/error cases) and matches the real server's EXPLAIN AST json=1 data_type subtree (oracle: 50/50). Includes a tsx-based CLI, a dependency-free node:test unit suite, and TS ports of the oracle/unsupported test harnesses. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a standalone ClickHouse data-type string parser under type-parser/, with both a self-contained C++ implementation (extracted/minimized from the server parser) and a faithful TypeScript port that emits a JSON AST matching the server’s EXPLAIN AST json = 1 data-type subtree (format v2).
Changes:
- Introduces a dependency-free C++20
mini-parser-extractedlibrary +chdt-parseCLI with CMake build and oracle/unsupported test harnesses. - Adds a TypeScript
mini-parser-tsport with matching lexer/parser/JSON serialization, plus CLI and Node test/oracle tooling. - Adds corpus files (
cases.txt,cases_unsupported.txt) and helper scripts for validation against a ClickHouse binary.
Reviewed changes
Copilot reviewed 31 out of 32 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| type-parser/mini-parser-ts/tsconfig.json | TypeScript build configuration for the standalone TS port. |
| type-parser/mini-parser-ts/tool/main.ts | TS CLI entrypoint (chdt-parse) reading args/stdin and printing JSON AST. |
| type-parser/mini-parser-ts/test/parser.test.ts | Dependency-free unit tests for representative parses + deliberate rejections. |
| type-parser/mini-parser-ts/test/oracle_compare.ts | Oracle comparison script vs clickhouse local EXPLAIN AST ... output. |
| type-parser/mini-parser-ts/test/check_unsupported.ts | Script asserting unsupported types are rejected. |
| type-parser/mini-parser-ts/test/cases.txt | Supported type corpus for oracle comparison. |
| type-parser/mini-parser-ts/test/cases.ts | Shared helpers (read cases, canonicalize JSON, deep compare). |
| type-parser/mini-parser-ts/test/cases_unsupported.txt | Deliberately unsupported type corpus. |
| type-parser/mini-parser-ts/src/parser.ts | Core TS parser port (control-flow mirroring server/C++). |
| type-parser/mini-parser-ts/src/lexer.ts | Purpose-built TS tokenizer for type strings. |
| type-parser/mini-parser-ts/src/json.ts | Byte-faithful TS JSON serializer matching server/C++ output. |
| type-parser/mini-parser-ts/src/index.ts | Public TS barrel exports. |
| type-parser/mini-parser-ts/src/ast.ts | Minimal TS AST node model + constructor helper. |
| type-parser/mini-parser-ts/README.md | Usage/docs for the TS parser + CLI and test commands. |
| type-parser/mini-parser-ts/package.json | TS package metadata, scripts, bin entry. |
| type-parser/mini-parser-ts/package-lock.json | Locked dev dependencies for the TS package (tsx/esbuild/typescript). |
| type-parser/mini-parser-ts/.gitignore | Ignores node_modules/ and dist/ for TS package. |
| type-parser/mini-parser-extracted/tool/main.cpp | C++ CLI chdt-parse reading args/stdin and printing JSON AST. |
| type-parser/mini-parser-extracted/test/oracle_compare.py | Python oracle comparison vs ClickHouse server AST output. |
| type-parser/mini-parser-extracted/test/CMakeLists.txt | CTest registration for oracle + unsupported suites. |
| type-parser/mini-parser-extracted/test/check_unsupported.py | Python script asserting unsupported types are rejected. |
| type-parser/mini-parser-extracted/test/cases.txt | Supported type corpus for C++ oracle suite. |
| type-parser/mini-parser-extracted/test/cases_unsupported.txt | Unsupported type corpus for C++ suite. |
| type-parser/mini-parser-extracted/src/parser.cpp | Core C++ parser implementation (server-extracted/minimized). |
| type-parser/mini-parser-extracted/src/lexer.h | Token definitions + tokenizer interface for the C++ parser. |
| type-parser/mini-parser-extracted/src/lexer.cpp | Purpose-built C++ tokenizer implementation. |
| type-parser/mini-parser-extracted/src/json.cpp | JSON serializer for the C++ AST, matching server shape. |
| type-parser/mini-parser-extracted/README.md | Usage/docs for the C++ library, CLI, and tests. |
| type-parser/mini-parser-extracted/include/chdt/parser.h | Public C++ API for parse + error surface. |
| type-parser/mini-parser-extracted/include/chdt/ast.h | Public C++ AST model + JSON serialization API. |
| type-parser/mini-parser-extracted/CMakeLists.txt | Top-level CMake build for library + CLI + tests. |
| type-parser/mini-parser-extracted/.gitignore | Ignores build/ directory for C++ project. |
Files not reviewed (1)
- type-parser/mini-parser-ts/package-lock.json: Generated file
Grow test/cases.txt to 356 ClickHouse data types (scalars, parametric, deeply nested, enums, tuples, SQL/MySQL aliases, geo, interval, ...) and capture the server's data_type AST for each as a static snapshot under test/snapshots/ (one <sha1>.json per query). The snapshot test then runs with no clickhouse binary, so `npm test` is fully self-contained. A snapshot is only written when the server accepts the type AND the parser matches it, so "parser == snapshot" means "parser == server". All 356 pass. Tooling: - update_snapshots.ts (npm run snapshot:update -- --clickhouse <path>): validates cases.txt + candidates.txt against the server, keeps only matching types, appends new keepers to cases.txt, writes/prunes snapshots, and reports rejected/divergent types. - oracle.ts: shared server-query + parser helpers, refactored out of oracle_compare.ts. - snapshot.test.ts: the CLI-free comparison against the static snapshots. - candidates.txt: the generated seed list of candidate types. Also seeds validate_types_live.ts, a live type-instantiation sanity check (refined in the follow-up commit). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
EXPLAIN SYNTAX/AST only checks parse syntax (it accepts Bogus(UInt8)), so
it can't prove the corpus uses real types. Add validate_types_live.ts
(npm run validate:live), which instantiates each cases.txt type via a
session-scoped CREATE TEMPORARY TABLE over the HTTP interface — with the
relevant experimental settings enabled — so an unknown family fails with
UNKNOWN_TYPE.
Against a stock v26.1 server: 347/356 instantiate, 0 unexpected. The 9
that don't are parser-valid but factory-invalid (partial tuple naming,
Nullable in Variant, Nullable(Tuple), Dynamic max_types=255, BINARY alias
without size, legacy Object('json')); they are allowlisted in
non_instantiable.txt with reasons. This library mirrors ParserDataType,
not the type factory, so these stay as intentional parser test inputs.
The check exits non-zero only on an unexpected failure or a stale
allowlist entry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
peter-leonov-ch
force-pushed
the
rowbinary-type-parser
branch
from
June 24, 2026 17:25
d24dd27 to
dbb6560
Compare
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
…tatype-parser
The package could not be published as-is: main/types pointed at
dist/index.js (the real entry is dist/src/index.js), there was no files
whitelist (npm would have shipped src/, the whole test harness, and 356
snapshot JSONs), and the test files were compiled into dist/.
- Fix main/types and add an exports map (dist/src/index.*).
- Add a files whitelist (dist, README.md, LICENSE).
- Add tsconfig.build.json (src + tool only); `build` uses it so dist ships
just the library and CLI. `typecheck` still covers tests.
- Add publish metadata: scoped name, Apache-2.0 license, repository with
directory, engines (node >=18), keywords, sideEffects:false, and
publishConfig.access=public for the scoped package.
- prepack runs the build so the tarball is always fresh.
- Bundle LICENSE, point the README import at the scoped name, ignore *.tgz.
Verified: npm pack ships 15 files (16.7 kB); installing the tarball into a
fresh project resolves both `import { parseDataType, toJSON }` and the
`chdt-parse` bin.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Apply the repo's prettier config across the package sources, tests, and docs. No behavior change — typecheck clean, 369/369 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- parser.ts: correct the matchWords doc comment (it takes an array and returns a boolean while advancing pos, not a count / joined string). - update_snapshots.ts: drop the unused readFileSync import. - mini-parser-extracted/CMakeLists.txt: guard -Wall -Wextra behind non-MSVC; use /W4 on MSVC so Windows generators build. The other two review notes (package main/types -> dist/src/*, and Prettier formatting) were already handled in earlier commits. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a small, self-contained ClickHouse data-type string parser under
type-parser/— the kind of string sent in the types row ofRowBinaryWithNamesAndTypes(e.g.Array(Nullable(UInt64)),Tuple(a UInt8, b String),Enum8('a' = 1),Decimal(10, 2)) — parsed into a JSON AST that mirrors the server'sEXPLAIN AST json = 1data-type subtree (format v2).Two implementations:
mini-parser-extracted/— the C++ library, extracted from the server'sParserDataType(src/Parsers/ParserDataType.cpp) but with no dependency on the ClickHouse source tree (only the C++20 stdlib). A purpose-built tokenizer replaces the fullLexer; plain structs replaceIAST/Field.mini-parser-ts/— a faithful TypeScript port, module-for-module (ast/lexer/parser/json), preserving the original control flow, branch ordering, andpossave/restore points. No runtime dependencies.Coverage
Scalars, parametric types with literal args (
Decimal,FixedString,DateTime64, …), nested type args (Array,Map,Nullable,LowCardinality,Variant, …), enums (explicit →EnumDataType; auto-assigned → genericDataType), named/unnamed/mixed tuples,Nested,Dynamic(max_types = N), legacyObject('json'), and the SQL-standard multi-word aliases (DOUBLE PRECISION,CHAR VARYING,INT SIGNED, …).Deliberately deferred (parser returns a clear error):
AggregateFunction/SimpleAggregateFunction, and the newJSON(...)object-argument syntax.Verification
ctest— oracle (50/50) + unsupported (2/2) pass.tscclean;node:testunit suite (12/12); unsupported (5/5); oracle vs. the real server (50/50).chdt-parseacross the full corpus (50 supported + 5 unsupported + edge/error cases) — identical stdout, stderr messages, byte offsets, and exit codes.The oracle suites compare against a
clickhousebinary built from peter-leonov-ch/ClickHouse#1 — the AST-format changes this parser mirrors live in that PR, so a stock server build will not match.🤖 Generated with Claude Code