Refactor to match ClickHouse EXPLAIN AST json=1 format

.gitignore

@@ -1,7 +1,9 @@
 node_modules/
 dist/
 coverage/
+tmp/
 *.tsbuildinfo
 .DS_Store
 .claude/settings.local.json
-create_docs/
+create_docs/
+clickhouse

CLAUDE.md

@@ -1,9 +1,22 @@
 # Claude Development Guide
 
+## AST Shape
+
+- The AST is intended to be a superset of ClickHouse's native AST, as produced by `EXPLAIN AST json=1 ...` from the `./clickhouse` binary.
+- The AST includes additional properties for tracking source locations of each node, comments, and a few non-semantic properties required for formatExplain(). 
+    - Deviations from the reference AST are permitted ONLY (a) to support library-only features such as query parameter parsing and comment round-tripping or (b) to represent semantic information not inferrable from the reference AST.
+    - Semantics-preserving canonicalizations of formatted SQL are permitted and preferred over non-reference additions to the AST.
+- The AST types and should not use `any` or `unknown` - they should be fully-typed, and discriminated unions should be used wherever possible to make type narrowing via type guards possible and make casting unnecessary.
+
 ## Best Practices
 
-- Formatting functions (like `format()` or `formatExplain()` should not include parsing code. The parser should do all parsing, and store structured data in the AST. The formatter should then format the AST data, without further parsing being required).
+- Formatting functions (like `format()` or `formatExplain()` should not include parsing code. The parser should do all parsing, and store structured data in the AST. The formatter should then format SQL based solely on the AST data, without further parsing being required).
 - In the grammar, use PEG grammar rules for parsing, not JavaScript code.
-- All AST Node kinds should be included in `ASTNodeKindMap` (and therefore `ASTNode`).
-- All AST Node types should include comment, location, and parent metadata.
-- No syntax should be stored in the AST as raw strings when it is a syntactical structure. Literals may be stored as unstructured, but otherwise the AST should have enough information that a user could fully understand the input statement without further parsing of any raw strings.
+- All AST node `type`s should be included in `ASTNodeTypeMap` (and therefore `ASTNode`).
+- All AST node types should include comment, location, and parent metadata.
+- No syntax should be stored in the AST as raw strings when it is a syntactical structure. The AST should have enough information that a user could fully understand the input statement without further parsing of any raw strings.
+
+## Non-Semantic Fields for Explain Formatting
+
+- Some underscore-prefixed fields (e.g. `_with_trailing`, `_agg_repeat`, `_settings_before_format`, `_settings_after_order_by`, `_no_parens`) are non-semantic and exist ONLY to let `formatExplain()` reproduce ClickHouse's `EXPLAIN AST` text dump, which exposes internal child ordering/duplication that the `json=1` reference AST discards. They are documented under "Non-Semantic Fields for Explain Formatting" in the README.
+- These fields MUST be used only by `formatExplain()`. NEVER read them in `format()`; `format()` must emit a single canonical SQL form and treat the source variation these fields record as a semantics-preserving canonicalization.

README.md

@@ -27,33 +27,91 @@ The AST for the query above:
 ```json
 [
   {
-    "kind": "select",
-    "select": [
-      { "kind": "columnRef", "parts": ["id"] },
-      { "kind": "columnRef", "parts": ["name"] }
-    ],
-    "from": {
-      "kind": "tableRef",
-      "table": "users"
-    },
-    "where": {
-      "kind": "binaryExpr",
-      "op": "=",
-      "left": { "kind": "columnRef", "parts": ["active"] },
-      "right": { "kind": "literal", "type": "UInt64", "value": "1" }
-    },
-    "orderBy": [
+    "type": "SelectWithUnionQuery",
+    "selects": [
       {
-        "kind": "orderByItem",
-        "expr": { "kind": "columnRef", "parts": ["name"] },
-        "direction": "ASC"
+        "type": "SelectQuery",
+        "select": [
+          { "type": "Identifier", "name": "id" },
+          { "type": "Identifier", "name": "name" }
+        ],
+        "from": {
+          "type": "TablesInSelectQuery",
+          "children": [
+            {
+              "type": "TablesInSelectQueryElement",
+              "table_expression": {
+                "type": "TableExpression",
+                "database_and_table_name": {
+                  "type": "TableIdentifier",
+                  "name": "users"
+                }
+              }
+            }
+          ]
+        },
+        "where": {
+          "type": "Function",
+          "name": "equals",
+          "arguments": [
+            { "type": "Identifier", "name": "active" },
+            { "type": "Literal", "value_type": "UInt64", "value": "1" }
+          ],
+          "is_operator": true
+        },
+        "order_by": [
+          {
+            "type": "OrderByElement",
+            "expression": { "type": "Identifier", "name": "name" },
+            "direction": "ASC"
+          }
+        ]
       }
     ]
   }
 ]
 ```
 
-Each node has a `kind` discriminator field. See [ast.ts](src/ast.ts) for all node types and their Zod schemas.
+Each node has a `type` discriminator field that mirrors ClickHouse's native AST.
+See [ast.ts](src/ast.ts) for all node types and their Zod schemas.
+
+#### Parse options
+
+`parse()` accepts an optional second argument:
+
+```typescript
+parse(sql, {
+  // Attach a `location` (line/column/offset source range) to every node.
+  // Set to false for a leaner, fully JSON-serializable AST. Default: true.
+  locations: true,
+
+  // Set a `parent` reference on every node. This introduces circular
+  // references, which break `JSON.stringify`. Default: false.
+  setParents: false,
+});
+```
+
+With locations enabled (the default) **every** AST node carries a `location`, so
+it is a **required** property — no `| undefined`, no `!`:
+
+```typescript
+const ast = parse(sql); // Statement[]
+ast[0].location.start.offset; // ✓ SourceLocation (always present)
+```
+
+When `locations: false` is passed as a literal, `parse()` returns
+`WithoutLocations<Statement>[]` — a type with `location` recursively removed, so
+reading `.location` is a compile-time error:
+
+```typescript
+const ast = parse(sql, { locations: false });
+ast[0].location; // ✗ type error: property does not exist
+```
+
+The AST-consuming functions — `format`, `formatExplain`, `formatNode`, and
+`transformNodes` — accept `WithoutLocations<…>` inputs. Because a located AST is
+assignable to its location-free counterpart, they transparently accept ASTs both
+with and without locations.
 
 ### Formatting AST back to SQL
 
@@ -118,31 +176,90 @@ try {
   parse('SELECT ???');
 } catch (e) {
   if (e instanceof ParseError) {
-    e.message;  // Human-readable error message
+    e.message; // Human-readable error message
     e.location; // { start: { line, column, offset }, end: { line, column, offset } }
     e.expected; // What the parser expected (e.g. literals, token classes)
-    e.found;    // What was found instead (string | null)
+    e.found; // What was found instead (string | null)
   }
 }
 ```
 
 ## Supported SQL
 
-### Statements
-
-- **SELECT** — full support, including ClickHouse-specific clauses and syntax
-- **UNION ALL / UNION DISTINCT** — with correct precedence
-- **INTERSECT / EXCEPT** — with higher precedence than UNION
-- **EXPLAIN** — with type, settings, and inner query
-- **SET** — session variable assignment
-- **USE** — database selection
-- **SYSTEM** — system commands (parsed as raw text)
-- **CREATE** - Partial DDL support (for all CREATE statements)
+The parser targets full coverage of the ClickHouse SQL surface and is verified
+against a corpus of ~7000 representative reference query files drawn from the
+official ClickHouse repository. For every reference, the AST, formatted SQL,
+and `EXPLAIN AST` projection match ClickHouse's own output exactly.
 
 ### Limitations
 
 - **ClickHouse-only** — this is not a general SQL parser. Syntax from other dialects that ClickHouse doesn't support will not parse.
 - **KQL** — Kusto Query Language syntax is not supported.
+- **Inserted Values** are not parsed - the AST does not include literal values being inserted into a table.
+
+### Divergence from ClickHouse Reference AST
+
+The AST parsed by this library is a superset of the JSON AST produced by ClickHouse introduced [here](https://github.com/peter-leonov-ch/ClickHouse/pull/1). Additional properties are described below.
+
+#### Location Metadata
+
+Every node carries a `location: { start, end }` recording its source
+range in the original SQL, where `start`/`end` are `{ offset, line, column }`
+(compatible with peggy's `LocationRange`). Locations are included by default;
+pass `{ locations: false }` to `parse()` to omit them (see [Parse options](#parse-options)).
+
+#### Parent Metadata
+
+Optionally, every node can be returned with a `parent` reference to its enclosing node. Pass the
+`options: { setParents: true }` argument to `parse()` to enable this feature. This enables upward
+traversal of the tree. It creates circular references, so exclude it when serializing or comparing
+the AST.
+
+#### Comment Data
+
+Comments are attached to the nearest node as arrays of their full source text
+(including `--`, `#`, or `/* */` delimiters):
+
+- `leadingComments` — comments appearing before the node.
+- `trailingComments` — inline comments on the same line as the end of the node.
+
+#### Semantic Fields not in Reference AST
+
+These fields carry semantic information that the reference JSON AST discards but
+that `format()` needs to reproduce the source faithfully.
+
+- `_nonfinite` — Found on `Literal` and `LiteralElement` nodes, this is a discriminator
+for `Float64` values that the native `value` float cannot represent in serialized JSON.
+Non-finite values (`inf`, `-inf`, `nan`, `-nan`) that each collapse to `null` can be
+recovered using this flag.
+- `QueryParameter` nodes - represent query parameters (`{name:type}` syntax) in the source,
+which are not represented by the reference ClickHouse JSON AST.
+
+#### Non-Semantic Fields for Explain Formatting
+
+These underscore-prefixed fields carry no semantic meaning and are **only** read
+by `formatExplain()`. They exist because ClickHouse's `EXPLAIN AST`exposes internal
+child-vector ordering and duplication that the structured JSON AST discards.
+
+- `_with_trailing` — marks a `WITH` clause that ClickHouse appends *after* the
+  select body rather than before it (a `WITH` written before an enclosing
+  `INSERT`, or propagated into a non-leftmost `UNION`/`INTERSECT` member). The
+  reference AST stores the same `with` field in both positions, so the flag is
+  the only record of the trailing placement.
+- `_agg_repeat` — marks the synthetic `SelectQuery` produced when lowering
+  `expr op ANY/ALL (subquery)`. ClickHouse's text dump emits this node's
+  projection and tables twice (`SelectQuery (children 4)`); the reference AST
+  keeps a single copy, indistinguishable from a user-written
+  `(SELECT agg(*) FROM (sub))`.
+- `_settings_before_format` — records that `SETTINGS` preceded `FORMAT` in the
+  source. `format()` canonicalizes to `FORMAT ... SETTINGS ...`; the flag lets
+  the explain projection reproduce the original child order.
+- `_settings_after_order_by` — records that a storage `SETTINGS` clause appeared
+  before the last clause in the source. `format()` canonicalizes it to the
+  required final position; the flag preserves the original `Set` child order.
+- `_no_parens` — records that a codec/engine function was written without
+  parentheses (e.g. `Delta` vs `Delta()`). `format()` canonicalizes to the
+  empty-parens form; the flag reproduces ClickHouse's byte-exact AST.
 
 ## Development
 

eslint.config.mjs

@@ -16,6 +16,7 @@ export default defineConfig(
       'src/parser.js',
       'src/parser.d.ts',
       'scripts/**',
+      'tmp/**',
     ],
   },
 );

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@clickhouse/parser",
-  "version": "0.2.1",
+  "version": "0.3.0",
   "description": "A TypeScript parser for ClickHouse SQL",
   "private": false,
   "repository": {

scripts/diff-ast.ts

@@ -7,8 +7,10 @@
  * Run via: npm run diff:ast -- <reference> [options]   (see --help)
  */
 
-import { computeAst, parseCli, run } from './diff-lib.js';
+import { computeAst, normalizeAstJson, parseCli, run } from './diff-lib.js';
 
 const opts = parseCli(process.argv, 'diff:ast', 'parsed AST vs. expected AST');
 
-run(opts, '.expected.ast.json', (sql) => computeAst(sql));
+// Normalize both sides by sorting object keys so reorderings of properties within an
+// AST object aren't reported as diffs.
+run(opts, '.expected.ast.json', (sql, expected) => computeAst(sql, expected), normalizeAstJson);

scripts/diff-explain.ts

@@ -15,6 +15,10 @@
 
 import { computeExplain, parseCli, run } from './diff-lib.js';
 
-const opts = parseCli(process.argv, 'diff:explain', 'formatExplain() output vs. expected explain output');
+const opts = parseCli(
+  process.argv,
+  'diff:explain',
+  'formatExplain() output vs. expected explain output',
+);
 
 run(opts, '.expected.explain.txt', computeExplain);