feat: add formatJsonExplain, rename library-only fields

CLAUDE.md

@@ -18,5 +18,6 @@
 
 ## Non-Semantic Fields for Explain Formatting
 
-- Some underscore-prefixed fields (e.g. `_with_trailing`, `_agg_repeat`, `_settings_before_format`, `_settings_after_order_by`, `_no_parens`) are non-semantic and exist ONLY to let `formatExplain()` reproduce ClickHouse's `EXPLAIN AST` text dump, which exposes internal child ordering/duplication that the `json=1` reference AST discards. They are documented under "Non-Semantic Fields for Explain Formatting" in the README.
-- These fields MUST be used only by `formatExplain()`. NEVER read them in `format()`; `format()` must emit a single canonical SQL form and treat the source variation these fields record as a semantics-preserving canonicalization.
+- Some library-only fields (e.g. `with_trailing`, `agg_repeat`, `settings_before_format`, `settings_after_order_by`, `no_parens`) are non-semantic and exist ONLY to let `formatExplain()` reproduce ClickHouse's `EXPLAIN AST` text dump, which exposes internal child ordering/duplication that the `json=1` reference AST discards. They are absent from the reference AST, so `formatExplainJson()` (used by the reference ast test) strips them along with node metadata. They are documented under "Non-Semantic Fields for Explain Formatting" in the README.
+- These fields MUST be used only by `formatExplain()`. NEVER read them in `format()`; `format()` must emit a single canonical SQL form and treat the source variation these fields record as a semantics-preserving canonicalization.
+- To add or change the set of library-only fields that the reference AST omits, update the `LIBRARY_ONLY_FIELDS` map in `src/json-explain.ts`, listing every node `type` each field is allowed on (so a same-named reference field on another node type is never dropped). Generally, strive to avoid doing this unless there is no way to capture or infer all semantic information from the reference fields alone. Semantics-preserving canonicalization of formatted SQL is preferable to introducing new library-only fields.

README.md

@@ -230,7 +230,7 @@ Comments are attached to the nearest node as arrays of their full source text
 These fields carry semantic information that the reference JSON AST discards but
 that `format()` needs to reproduce the source faithfully.
 
-- `_nonfinite` — Found on `Literal` and `LiteralElement` nodes, this is a discriminator
+- `nonfinite` — Found on `Literal` and `LiteralElement` nodes, this is a discriminator
 for `Float64` values that the native `value` float cannot represent in serialized JSON.
 Non-finite values (`inf`, `-inf`, `nan`, `-nan`) that each collapse to `null` can be
 recovered using this flag.
@@ -239,30 +239,47 @@ which are not represented by the reference ClickHouse JSON AST.
 
 #### Non-Semantic Fields for Explain Formatting
 
-These underscore-prefixed fields carry no semantic meaning and are **only** read
-by `formatExplain()`. They exist because ClickHouse's `EXPLAIN AST`exposes internal
-child-vector ordering and duplication that the structured JSON AST discards.
+These fields carry no semantic meaning and are **only** read by `formatExplain()`.
+They exist because ClickHouse's `EXPLAIN AST` exposes internal child-vector ordering
+and duplication that the structured JSON AST discards. Like the semantic fields
+above, they are absent from the reference AST, so `formatExplainJson()` strips them.
 
-- `_with_trailing` — marks a `WITH` clause that ClickHouse appends *after* the
+- `with_trailing` — marks a `WITH` clause that ClickHouse appends *after* the
   select body rather than before it (a `WITH` written before an enclosing
   `INSERT`, or propagated into a non-leftmost `UNION`/`INTERSECT` member). The
   reference AST stores the same `with` field in both positions, so the flag is
   the only record of the trailing placement.
-- `_agg_repeat` — marks the synthetic `SelectQuery` produced when lowering
+- `agg_repeat` — marks the synthetic `SelectQuery` produced when lowering
   `expr op ANY/ALL (subquery)`. ClickHouse's text dump emits this node's
   projection and tables twice (`SelectQuery (children 4)`); the reference AST
   keeps a single copy, indistinguishable from a user-written
   `(SELECT agg(*) FROM (sub))`.
-- `_settings_before_format` — records that `SETTINGS` preceded `FORMAT` in the
+- `settings_before_format` — records that `SETTINGS` preceded `FORMAT` in the
   source. `format()` canonicalizes to `FORMAT ... SETTINGS ...`; the flag lets
   the explain projection reproduce the original child order.
-- `_settings_after_order_by` — records that a storage `SETTINGS` clause appeared
+- `settings_after_order_by` — records that a storage `SETTINGS` clause appeared
   before the last clause in the source. `format()` canonicalizes it to the
   required final position; the flag preserves the original `Set` child order.
-- `_no_parens` — records that a codec/engine function was written without
+- `no_parens` — records that a codec/engine function was written without
   parentheses (e.g. `Delta` vs `Delta()`). `format()` canonicalizes to the
   empty-parens form; the flag reproduces ClickHouse's byte-exact AST.
 
+#### Filtered Storage Settings
+
+When ClickHouse emits `EXPLAIN AST json=1`, it filters a
+`CREATE TABLE`/`CREATE DATABASE ... SETTINGS` clause down to only the settings
+that belong to the target engine's own registry (a version- and
+engine-specific set). Session- or query-level settings that were written into
+the clause — e.g. `log_queries`, `allow_suspicious_low_cardinality_types`, or
+`use_hive_partitioning` on a table, or `distributed_ddl_task_timeout` on a
+database — are dropped from the `Storage > Set` `changes` map, and if nothing
+remains the entire `Storage`/`settings` node is dropped.
+
+This library does **not** replicate that engine-specific registry: it keeps
+**every** setting in the clause so `format()` can re-emit the original SQL
+faithfully. The library AST may therefore carry extra `Storage` settings that
+the reference AST omits.
+
 ## Development
 
 ```bash

eslint.config.mjs

@@ -17,6 +17,7 @@ export default defineConfig(
       'src/parser.d.ts',
       'scripts/**',
       'tmp/**',
+      'playground/**',
     ],
   },
 );

playground/src/App.tsx

@@ -1,11 +1,12 @@
 import { useCallback, useMemo, useRef, useState } from 'react';
-import { parse, format, formatExplain } from '@clickhouse/parser';
+import { parse, format, formatExplain, formatExplainJson } from '@clickhouse/parser';
 import type { AstNode, SourceLocation } from './ast-utils';
 import { SqlInput } from './SqlInput';
 import { AstJson } from './AstJson';
 import { AstViz } from './AstViz';
 import { NodeDetail } from './NodeDetail';
 import { TextOutput } from './TextOutput';
+import { JsonExplain } from './JsonExplain';
 
 const DEFAULT_SQL = `SELECT
     user_id,
@@ -19,13 +20,14 @@ HAVING events > {min_events:UInt32}
 ORDER BY events DESC
 LIMIT 100;`;
 
-type Tab = 'json' | 'viz' | 'formatted' | 'explain';
+type Tab = 'json' | 'viz' | 'formatted' | 'jsonExplain' | 'explain';
 
 const TABS: { id: Tab; label: string }[] = [
   { id: 'json', label: 'AST (JSON)' },
   { id: 'viz', label: 'AST (Visual)' },
   { id: 'formatted', label: 'Formatted SQL' },
-  { id: 'explain', label: 'Explain AST Output' },
+  { id: 'explain', label: 'Explain AST' },
+  { id: 'jsonExplain', label: 'Explain AST JSON (v2)' },
 ];
 
 type ParseResult = { ok: true; statements: AstNode[] } | { ok: false; error: string };
@@ -95,6 +97,13 @@ export function App() {
     () => (parsed.ok ? tryRun(() => formatExplain(parsed.statements as never)) : null),
     [parsed],
   );
+  const jsonExplained = useMemo(
+    () =>
+      parsed.ok
+        ? tryRun(() => JSON.stringify(formatExplainJson(parsed.statements as never, 2), null, 2))
+        : null,
+    [parsed],
+  );
 
   return (
     <div className="app">
@@ -160,6 +169,8 @@ export function App() {
                 />
               ) : tab === 'formatted' ? (
                 <TextOutput result={formatted} />
+              ) : tab === 'jsonExplain' ? (
+                <JsonExplain result={jsonExplained} />
               ) : (
                 <TextOutput result={explained} />
               )}

playground/src/JsonExplain.tsx

@@ -0,0 +1,22 @@
+type Result = { ok: true; value: string } | { ok: false; error: string } | null;
+
+type Props = {
+  result: Result;
+};
+
+/**
+ * Renders the `formatExplainJson` output (the ClickHouse-native `EXPLAIN AST
+ * json = 2` view, `{ version, ast }` per statement) as formatted JSON, or its
+ * error.
+ */
+export function JsonExplain({ result }: Props) {
+  if (!result) return <div className="viz-empty">No output.</div>;
+  if (!result.ok) {
+    return <pre className="error">{`Error:\n${result.error}`}</pre>;
+  }
+  return (
+    <div className="json-view">
+      <pre className="json-pre">{result.value}</pre>
+    </div>
+  );
+}

scripts/diff-lib.ts

@@ -13,8 +13,8 @@ import * as path from 'path';
 import { createTwoFilesPatch } from 'diff';
 import { minimatch } from 'minimatch';
 import pc from 'picocolors';
-import { format, formatExplain, parse } from '../src/index.js';
-import { stripAstMeta, stripVolatile } from '../src/meta.js';
+import { format, formatExplain, formatExplainJson, parse } from '../src/index.js';
+import { stripVolatile } from '../src/meta.js';
 import { substituteQueryParameters } from '../src/query-parameters.js';
 
 /**
@@ -43,7 +43,7 @@ const AST_ERROR = '<AST Error>';
  * don't surface as spurious diffs (mirroring the ast reference test).
  */
 export function computeAst(sql: string, expected: string | null = null): string {
-  const actual = stripAstMeta(parse(sql)) as unknown[];
+  const actual = formatExplainJson(parse(sql), 2) as unknown[];
   if (expected !== null) {
     try {
       const expectedEntries = JSON.parse(expected) as unknown[];

src/ast.ts

@@ -60,10 +60,11 @@ const ExprMetadataFields = {
 
 // ── ClickHouse-native expression nodes ────────────────────────────────────────
 // These node types mirror ClickHouse's own AST: the `type` discriminator and all
-// non-underscore fields must match `EXPLAIN AST json = 1` output exactly (the
-// reference ast suite compares them via `stripAstMeta`). Underscore-prefixed
-// fields carry library-only data the native JSON loses but that format() and
-// formatExplain() need. See CLAUDE.md and src/meta.ts.
+// reference AST fields must match `EXPLAIN AST json = 1` output exactly (the
+// reference ast suite compares them via `formatExplainJson`, which drops the
+// library-only fields). A few fields carry library-only data the native JSON
+// loses but that format() and formatExplain() need; these are enumerated in
+// src/json-explain.ts and documented in the README.
 
 /**
  * A query parameter placeholder: `{name:Type}`. Library-extension node type —
@@ -105,14 +106,14 @@ export const IdentifierPartSchema: z.ZodType<WithoutLocations<IdentifierPart>> =
  * {@link LiteralNode} without the `type`/`alias`/metadata: just the element's
  * `value_type`, its decoded `value` (recursively a list for nested
  * collections), and—for non-finite/`-0` `Float64` elements—the same
- * `_nonfinite` discriminator a top-level Float64 literal carries (the native
+ * `nonfinite` discriminator a top-level Float64 literal carries (the native
  * JSON collapses those to `null`/`0`, so it is needed to re-spell `inf`/`-0`
  * in `format()`/`formatExplain()`).
  */
 export type LiteralElement = {
   value_type: LiteralNode['value_type'];
   value: number | string | boolean | null | LiteralElement[];
-  _nonfinite?: LiteralNode['_nonfinite'];
+  nonfinite?: LiteralNode['nonfinite'];
 };
 
 /** Zod schema for {@link LiteralElement}. */
@@ -136,7 +137,7 @@ export const LiteralElementSchema: z.ZodType<WithoutLocations<LiteralElement>> =
       z.null(),
       z.array(LiteralElementSchema),
     ]),
-    _nonfinite: z
+    nonfinite: z
       .union([
         z.literal('inf'),
         z.literal('-inf'),
@@ -186,7 +187,7 @@ export type LiteralNode = {
    * Hex/large-integer source spelling is intentionally not preserved
    * (`0xFF` formats as `255`).
    */
-  _nonfinite?: 'inf' | '-inf' | 'nan' | '-nan' | '-0';
+  nonfinite?: 'inf' | '-inf' | 'nan' | '-nan' | '-0';
 } & NodeMetadata;
 
 /** Zod schema for {@link LiteralNode}. */
@@ -214,7 +215,7 @@ export const LiteralNodeSchema: z.ZodType<WithoutLocations<LiteralNode>> = z.laz
       z.array(LiteralElementSchema),
     ]),
     alias: z.string().optional(),
-    _nonfinite: z
+    nonfinite: z
       .union([
         z.literal('inf'),
         z.literal('-inf'),
@@ -293,7 +294,7 @@ export type FunctionNode = {
    * `arguments: []` for both the no-parens and empty-parens forms, but its
    * EXPLAIN/SHOW CREATE output distinguishes them.
    */
-  _no_parens?: boolean;
+  no_parens?: boolean;
 } & NodeMetadata;
 
 /** Zod schema for {@link FunctionNode}. */
@@ -321,7 +322,7 @@ export const FunctionNodeSchema: z.ZodType<WithoutLocations<FunctionNode>> = z.l
     window_name: z.string().optional(),
     nulls_action: z.union([z.literal('RESPECT NULLS'), z.literal('IGNORE NULLS')]).optional(),
     alias: z.string().optional(),
-    _no_parens: z.boolean().optional(),
+    no_parens: z.boolean().optional(),
     ...ExprMetadataFields,
   }),
 );
@@ -1028,13 +1029,13 @@ export type SelectQueryNode = {
    * WITH ExpressionList after the select body in EXPLAIN AST; this flag lets
    * the explain projection reproduce that ordering.
    */
-  _with_trailing?: boolean;
+  with_trailing?: boolean;
   /**
    * Library-only: marks the synthetic SelectQuery produced when lowering
    * `expr op ANY/ALL (subquery)`. ClickHouse's EXPLAIN AST text dumps the
    * projection + tables twice for this node; the flag drives that doubling.
    */
-  _agg_repeat?: boolean;
+  agg_repeat?: boolean;
   distinct?: boolean;
   select: Expression[];
   from?: TablesInSelectQueryNode;
@@ -1083,8 +1084,8 @@ export const SelectQuerySchema: z.ZodType<WithoutLocations<SelectQueryNode>> = z
     type: z.literal('SelectQuery'),
     with: z.array(WithItemSchema).optional(),
     recursive_with: z.boolean().optional(),
-    _with_trailing: z.boolean().optional(),
-    _agg_repeat: z.boolean().optional(),
+    with_trailing: z.boolean().optional(),
+    agg_repeat: z.boolean().optional(),
     distinct: z.boolean().optional(),
     select: z.array(ExpressionSchema),
     from: TablesInSelectQuerySchema.optional(),
@@ -1144,15 +1145,15 @@ export type QueryTrailingFields = {
    * after FORMAT) — so the flag may flip across a reformat and is treated as
    * volatile in round-trip comparisons.
    */
-  _settings_before_format?: boolean;
+  settings_before_format?: boolean;
 };
 
 const QueryTrailingSchemaFields = {
   out_file: LiteralNodeSchema.optional(),
   outfile_truncate: z.boolean().optional(),
   format: z.string().optional(),
   settings: SettingsNodeSchema.optional(),
-  _settings_before_format: z.boolean().optional(),
+  settings_before_format: z.boolean().optional(),
 };
 
 /**
@@ -1472,6 +1473,18 @@ export type AuthenticationData = {
   secret?: string;
   /** SSH public keys (for the `ssh_key` auth type): `KEY '<key>' TYPE '<type>'`. */
   sshKeys?: { key: string; type: string }[];
+  /**
+   * The auth-method keyword to re-emit after `WITH` (e.g. `'sha256_hash'`,
+   * `'kerberos'`), reconstructed from the native `auth_type` enum plus the
+   * `contains_hash`/`contains_password` flags. Absent for a bare
+   * `IDENTIFIED BY '...'` (no explicit method) and for SSH keys.
+   */
+  authType?: string;
+  /**
+   * The keyword that introduces {@link secret}: `'BY'` for passwords/hashes,
+   * `'REALM'` for `kerberos`, `'SERVER'` for `ldap`. Defaults to `'BY'`.
+   */
+  secretKeyword?: 'BY' | 'REALM' | 'SERVER';
 };
 
 // ── ALTER TABLE statement ─────────────────────────────────────────────────────
@@ -2398,7 +2411,7 @@ export const StorageNodeSchema: z.ZodType<WithoutLocations<StorageNode>> = z.laz
     sample_by: ExpressionSchema.optional(),
     ttl_table: ExpressionListNodeSchema.optional(),
     settings: SettingsNodeSchema.optional(),
-    _settings_after_order_by: z.boolean().optional(),
+    settings_after_order_by: z.boolean().optional(),
     ...ExprMetadataFields,
   }),
 );
@@ -2632,7 +2645,7 @@ export type StorageNode = {
   ttl_table?: ExpressionListNode;
   settings?: SettingsNode;
   /** Library-only: `true` when storage `SETTINGS` appeared after `ORDER BY`. */
-  _settings_after_order_by?: boolean;
+  settings_after_order_by?: boolean;
 } & NodeMetadata;
 
 /** A single descending storage ORDER BY element. */
@@ -3604,7 +3617,7 @@ export type DescribeQueryNode = {
   format?: string;
   settings?: SettingsNode;
   /** Library-only: true when SETTINGS appeared before FORMAT in the source. */
-  _settings_before_format?: boolean;
+  settings_before_format?: boolean;
 } & NodeMetadata;
 
 /** Zod schema for {@link DescribeQueryNode}. */
@@ -3614,7 +3627,7 @@ export const DescribeQueryNodeSchema: z.ZodType<WithoutLocations<DescribeQueryNo
     table_expression: TableExpressionSchema.optional(),
     format: z.string().optional(),
     settings: SettingsNodeSchema.optional(),
-    _settings_before_format: z.boolean().optional(),
+    settings_before_format: z.boolean().optional(),
     ...ExprMetadataFields,
   }),
 );