diff --git a/CLAUDE.md b/CLAUDE.md index 75f4dc24c..074aa767c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -18,5 +18,6 @@ ## Non-Semantic Fields for Explain Formatting -- Some underscore-prefixed fields (e.g. `_with_trailing`, `_agg_repeat`, `_settings_before_format`, `_settings_after_order_by`, `_no_parens`) are non-semantic and exist ONLY to let `formatExplain()` reproduce ClickHouse's `EXPLAIN AST` text dump, which exposes internal child ordering/duplication that the `json=1` reference AST discards. They are documented under "Non-Semantic Fields for Explain Formatting" in the README. -- These fields MUST be used only by `formatExplain()`. NEVER read them in `format()`; `format()` must emit a single canonical SQL form and treat the source variation these fields record as a semantics-preserving canonicalization. \ No newline at end of file +- Some library-only fields (e.g. `with_trailing`, `agg_repeat`, `settings_before_format`, `settings_after_order_by`, `no_parens`) are non-semantic and exist ONLY to let `formatExplain()` reproduce ClickHouse's `EXPLAIN AST` text dump, which exposes internal child ordering/duplication that the `json=1` reference AST discards. They are absent from the reference AST, so `formatExplainJson()` (used by the reference ast test) strips them along with node metadata. They are documented under "Non-Semantic Fields for Explain Formatting" in the README. +- These fields MUST be used only by `formatExplain()`. NEVER read them in `format()`; `format()` must emit a single canonical SQL form and treat the source variation these fields record as a semantics-preserving canonicalization. +- To add or change the set of library-only fields that the reference AST omits, update the `LIBRARY_ONLY_FIELDS` map in `src/json-explain.ts`, listing every node `type` each field is allowed on (so a same-named reference field on another node type is never dropped). Generally, strive to avoid doing this unless there is no way to capture or infer all semantic information from the reference fields alone. Semantics-preserving canonicalization of formatted SQL is preferable to introducing new library-only fields. \ No newline at end of file diff --git a/README.md b/README.md index 095a4577a..a047ec0c3 100644 --- a/README.md +++ b/README.md @@ -230,7 +230,7 @@ Comments are attached to the nearest node as arrays of their full source text These fields carry semantic information that the reference JSON AST discards but that `format()` needs to reproduce the source faithfully. -- `_nonfinite` — Found on `Literal` and `LiteralElement` nodes, this is a discriminator +- `nonfinite` — Found on `Literal` and `LiteralElement` nodes, this is a discriminator for `Float64` values that the native `value` float cannot represent in serialized JSON. Non-finite values (`inf`, `-inf`, `nan`, `-nan`) that each collapse to `null` can be recovered using this flag. @@ -239,30 +239,47 @@ which are not represented by the reference ClickHouse JSON AST. #### Non-Semantic Fields for Explain Formatting -These underscore-prefixed fields carry no semantic meaning and are **only** read -by `formatExplain()`. They exist because ClickHouse's `EXPLAIN AST`exposes internal -child-vector ordering and duplication that the structured JSON AST discards. +These fields carry no semantic meaning and are **only** read by `formatExplain()`. +They exist because ClickHouse's `EXPLAIN AST` exposes internal child-vector ordering +and duplication that the structured JSON AST discards. Like the semantic fields +above, they are absent from the reference AST, so `formatExplainJson()` strips them. -- `_with_trailing` — marks a `WITH` clause that ClickHouse appends *after* the +- `with_trailing` — marks a `WITH` clause that ClickHouse appends *after* the select body rather than before it (a `WITH` written before an enclosing `INSERT`, or propagated into a non-leftmost `UNION`/`INTERSECT` member). The reference AST stores the same `with` field in both positions, so the flag is the only record of the trailing placement. -- `_agg_repeat` — marks the synthetic `SelectQuery` produced when lowering +- `agg_repeat` — marks the synthetic `SelectQuery` produced when lowering `expr op ANY/ALL (subquery)`. ClickHouse's text dump emits this node's projection and tables twice (`SelectQuery (children 4)`); the reference AST keeps a single copy, indistinguishable from a user-written `(SELECT agg(*) FROM (sub))`. -- `_settings_before_format` — records that `SETTINGS` preceded `FORMAT` in the +- `settings_before_format` — records that `SETTINGS` preceded `FORMAT` in the source. `format()` canonicalizes to `FORMAT ... SETTINGS ...`; the flag lets the explain projection reproduce the original child order. -- `_settings_after_order_by` — records that a storage `SETTINGS` clause appeared +- `settings_after_order_by` — records that a storage `SETTINGS` clause appeared before the last clause in the source. `format()` canonicalizes it to the required final position; the flag preserves the original `Set` child order. -- `_no_parens` — records that a codec/engine function was written without +- `no_parens` — records that a codec/engine function was written without parentheses (e.g. `Delta` vs `Delta()`). `format()` canonicalizes to the empty-parens form; the flag reproduces ClickHouse's byte-exact AST. +#### Filtered Storage Settings + +When ClickHouse emits `EXPLAIN AST json=1`, it filters a +`CREATE TABLE`/`CREATE DATABASE ... SETTINGS` clause down to only the settings +that belong to the target engine's own registry (a version- and +engine-specific set). Session- or query-level settings that were written into +the clause — e.g. `log_queries`, `allow_suspicious_low_cardinality_types`, or +`use_hive_partitioning` on a table, or `distributed_ddl_task_timeout` on a +database — are dropped from the `Storage > Set` `changes` map, and if nothing +remains the entire `Storage`/`settings` node is dropped. + +This library does **not** replicate that engine-specific registry: it keeps +**every** setting in the clause so `format()` can re-emit the original SQL +faithfully. The library AST may therefore carry extra `Storage` settings that +the reference AST omits. + ## Development ```bash diff --git a/eslint.config.mjs b/eslint.config.mjs index fa8f7985e..721987dd8 100644 --- a/eslint.config.mjs +++ b/eslint.config.mjs @@ -17,6 +17,7 @@ export default defineConfig( 'src/parser.d.ts', 'scripts/**', 'tmp/**', + 'playground/**', ], }, ); diff --git a/playground/src/App.tsx b/playground/src/App.tsx index b41536145..b99f57f7e 100644 --- a/playground/src/App.tsx +++ b/playground/src/App.tsx @@ -1,11 +1,12 @@ import { useCallback, useMemo, useRef, useState } from 'react'; -import { parse, format, formatExplain } from '@clickhouse/parser'; +import { parse, format, formatExplain, formatExplainJson } from '@clickhouse/parser'; import type { AstNode, SourceLocation } from './ast-utils'; import { SqlInput } from './SqlInput'; import { AstJson } from './AstJson'; import { AstViz } from './AstViz'; import { NodeDetail } from './NodeDetail'; import { TextOutput } from './TextOutput'; +import { JsonExplain } from './JsonExplain'; const DEFAULT_SQL = `SELECT user_id, @@ -19,13 +20,14 @@ HAVING events > {min_events:UInt32} ORDER BY events DESC LIMIT 100;`; -type Tab = 'json' | 'viz' | 'formatted' | 'explain'; +type Tab = 'json' | 'viz' | 'formatted' | 'jsonExplain' | 'explain'; const TABS: { id: Tab; label: string }[] = [ { id: 'json', label: 'AST (JSON)' }, { id: 'viz', label: 'AST (Visual)' }, { id: 'formatted', label: 'Formatted SQL' }, - { id: 'explain', label: 'Explain AST Output' }, + { id: 'explain', label: 'Explain AST' }, + { id: 'jsonExplain', label: 'Explain AST JSON (v2)' }, ]; type ParseResult = { ok: true; statements: AstNode[] } | { ok: false; error: string }; @@ -95,6 +97,13 @@ export function App() { () => (parsed.ok ? tryRun(() => formatExplain(parsed.statements as never)) : null), [parsed], ); + const jsonExplained = useMemo( + () => + parsed.ok + ? tryRun(() => JSON.stringify(formatExplainJson(parsed.statements as never, 2), null, 2)) + : null, + [parsed], + ); return (
@@ -160,6 +169,8 @@ export function App() { /> ) : tab === 'formatted' ? ( + ) : tab === 'jsonExplain' ? ( + ) : ( )} diff --git a/playground/src/JsonExplain.tsx b/playground/src/JsonExplain.tsx new file mode 100644 index 000000000..e5a5e4ed3 --- /dev/null +++ b/playground/src/JsonExplain.tsx @@ -0,0 +1,22 @@ +type Result = { ok: true; value: string } | { ok: false; error: string } | null; + +type Props = { + result: Result; +}; + +/** + * Renders the `formatExplainJson` output (the ClickHouse-native `EXPLAIN AST + * json = 2` view, `{ version, ast }` per statement) as formatted JSON, or its + * error. + */ +export function JsonExplain({ result }: Props) { + if (!result) return
No output.
; + if (!result.ok) { + return
{`Error:\n${result.error}`}
; + } + return ( +
+
{result.value}
+
+ ); +} diff --git a/scripts/diff-lib.ts b/scripts/diff-lib.ts index dcdf33c88..c5a4de413 100644 --- a/scripts/diff-lib.ts +++ b/scripts/diff-lib.ts @@ -13,8 +13,8 @@ import * as path from 'path'; import { createTwoFilesPatch } from 'diff'; import { minimatch } from 'minimatch'; import pc from 'picocolors'; -import { format, formatExplain, parse } from '../src/index.js'; -import { stripAstMeta, stripVolatile } from '../src/meta.js'; +import { format, formatExplain, formatExplainJson, parse } from '../src/index.js'; +import { stripVolatile } from '../src/meta.js'; import { substituteQueryParameters } from '../src/query-parameters.js'; /** @@ -43,7 +43,7 @@ const AST_ERROR = ''; * don't surface as spurious diffs (mirroring the ast reference test). */ export function computeAst(sql: string, expected: string | null = null): string { - const actual = stripAstMeta(parse(sql)) as unknown[]; + const actual = formatExplainJson(parse(sql), 2) as unknown[]; if (expected !== null) { try { const expectedEntries = JSON.parse(expected) as unknown[]; diff --git a/src/ast.ts b/src/ast.ts index 79fa4af96..a42aecd43 100644 --- a/src/ast.ts +++ b/src/ast.ts @@ -60,10 +60,11 @@ const ExprMetadataFields = { // ── ClickHouse-native expression nodes ──────────────────────────────────────── // These node types mirror ClickHouse's own AST: the `type` discriminator and all -// non-underscore fields must match `EXPLAIN AST json = 1` output exactly (the -// reference ast suite compares them via `stripAstMeta`). Underscore-prefixed -// fields carry library-only data the native JSON loses but that format() and -// formatExplain() need. See CLAUDE.md and src/meta.ts. +// reference AST fields must match `EXPLAIN AST json = 1` output exactly (the +// reference ast suite compares them via `formatExplainJson`, which drops the +// library-only fields). A few fields carry library-only data the native JSON +// loses but that format() and formatExplain() need; these are enumerated in +// src/json-explain.ts and documented in the README. /** * A query parameter placeholder: `{name:Type}`. Library-extension node type — @@ -105,14 +106,14 @@ export const IdentifierPartSchema: z.ZodType> = * {@link LiteralNode} without the `type`/`alias`/metadata: just the element's * `value_type`, its decoded `value` (recursively a list for nested * collections), and—for non-finite/`-0` `Float64` elements—the same - * `_nonfinite` discriminator a top-level Float64 literal carries (the native + * `nonfinite` discriminator a top-level Float64 literal carries (the native * JSON collapses those to `null`/`0`, so it is needed to re-spell `inf`/`-0` * in `format()`/`formatExplain()`). */ export type LiteralElement = { value_type: LiteralNode['value_type']; value: number | string | boolean | null | LiteralElement[]; - _nonfinite?: LiteralNode['_nonfinite']; + nonfinite?: LiteralNode['nonfinite']; }; /** Zod schema for {@link LiteralElement}. */ @@ -136,7 +137,7 @@ export const LiteralElementSchema: z.ZodType> = z.null(), z.array(LiteralElementSchema), ]), - _nonfinite: z + nonfinite: z .union([ z.literal('inf'), z.literal('-inf'), @@ -186,7 +187,7 @@ export type LiteralNode = { * Hex/large-integer source spelling is intentionally not preserved * (`0xFF` formats as `255`). */ - _nonfinite?: 'inf' | '-inf' | 'nan' | '-nan' | '-0'; + nonfinite?: 'inf' | '-inf' | 'nan' | '-nan' | '-0'; } & NodeMetadata; /** Zod schema for {@link LiteralNode}. */ @@ -214,7 +215,7 @@ export const LiteralNodeSchema: z.ZodType> = z.laz z.array(LiteralElementSchema), ]), alias: z.string().optional(), - _nonfinite: z + nonfinite: z .union([ z.literal('inf'), z.literal('-inf'), @@ -293,7 +294,7 @@ export type FunctionNode = { * `arguments: []` for both the no-parens and empty-parens forms, but its * EXPLAIN/SHOW CREATE output distinguishes them. */ - _no_parens?: boolean; + no_parens?: boolean; } & NodeMetadata; /** Zod schema for {@link FunctionNode}. */ @@ -321,7 +322,7 @@ export const FunctionNodeSchema: z.ZodType> = z.l window_name: z.string().optional(), nulls_action: z.union([z.literal('RESPECT NULLS'), z.literal('IGNORE NULLS')]).optional(), alias: z.string().optional(), - _no_parens: z.boolean().optional(), + no_parens: z.boolean().optional(), ...ExprMetadataFields, }), ); @@ -1028,13 +1029,13 @@ export type SelectQueryNode = { * WITH ExpressionList after the select body in EXPLAIN AST; this flag lets * the explain projection reproduce that ordering. */ - _with_trailing?: boolean; + with_trailing?: boolean; /** * Library-only: marks the synthetic SelectQuery produced when lowering * `expr op ANY/ALL (subquery)`. ClickHouse's EXPLAIN AST text dumps the * projection + tables twice for this node; the flag drives that doubling. */ - _agg_repeat?: boolean; + agg_repeat?: boolean; distinct?: boolean; select: Expression[]; from?: TablesInSelectQueryNode; @@ -1083,8 +1084,8 @@ export const SelectQuerySchema: z.ZodType> = z type: z.literal('SelectQuery'), with: z.array(WithItemSchema).optional(), recursive_with: z.boolean().optional(), - _with_trailing: z.boolean().optional(), - _agg_repeat: z.boolean().optional(), + with_trailing: z.boolean().optional(), + agg_repeat: z.boolean().optional(), distinct: z.boolean().optional(), select: z.array(ExpressionSchema), from: TablesInSelectQuerySchema.optional(), @@ -1144,7 +1145,7 @@ export type QueryTrailingFields = { * after FORMAT) — so the flag may flip across a reformat and is treated as * volatile in round-trip comparisons. */ - _settings_before_format?: boolean; + settings_before_format?: boolean; }; const QueryTrailingSchemaFields = { @@ -1152,7 +1153,7 @@ const QueryTrailingSchemaFields = { outfile_truncate: z.boolean().optional(), format: z.string().optional(), settings: SettingsNodeSchema.optional(), - _settings_before_format: z.boolean().optional(), + settings_before_format: z.boolean().optional(), }; /** @@ -1472,6 +1473,18 @@ export type AuthenticationData = { secret?: string; /** SSH public keys (for the `ssh_key` auth type): `KEY '' TYPE ''`. */ sshKeys?: { key: string; type: string }[]; + /** + * The auth-method keyword to re-emit after `WITH` (e.g. `'sha256_hash'`, + * `'kerberos'`), reconstructed from the native `auth_type` enum plus the + * `contains_hash`/`contains_password` flags. Absent for a bare + * `IDENTIFIED BY '...'` (no explicit method) and for SSH keys. + */ + authType?: string; + /** + * The keyword that introduces {@link secret}: `'BY'` for passwords/hashes, + * `'REALM'` for `kerberos`, `'SERVER'` for `ldap`. Defaults to `'BY'`. + */ + secretKeyword?: 'BY' | 'REALM' | 'SERVER'; }; // ── ALTER TABLE statement ───────────────────────────────────────────────────── @@ -2398,7 +2411,7 @@ export const StorageNodeSchema: z.ZodType> = z.laz sample_by: ExpressionSchema.optional(), ttl_table: ExpressionListNodeSchema.optional(), settings: SettingsNodeSchema.optional(), - _settings_after_order_by: z.boolean().optional(), + settings_after_order_by: z.boolean().optional(), ...ExprMetadataFields, }), ); @@ -2632,7 +2645,7 @@ export type StorageNode = { ttl_table?: ExpressionListNode; settings?: SettingsNode; /** Library-only: `true` when storage `SETTINGS` appeared after `ORDER BY`. */ - _settings_after_order_by?: boolean; + settings_after_order_by?: boolean; } & NodeMetadata; /** A single descending storage ORDER BY element. */ @@ -3604,7 +3617,7 @@ export type DescribeQueryNode = { format?: string; settings?: SettingsNode; /** Library-only: true when SETTINGS appeared before FORMAT in the source. */ - _settings_before_format?: boolean; + settings_before_format?: boolean; } & NodeMetadata; /** Zod schema for {@link DescribeQueryNode}. */ @@ -3614,7 +3627,7 @@ export const DescribeQueryNodeSchema: z.ZodType exprList(stmt.select); const tables = (): ExplainNode => tablesExplainNode(stmt.from!); return n('SelectQuery', [projection(), tables(), projection(), tables()]); @@ -413,9 +413,9 @@ function selectQueryNode(stmt: SelectQueryNode): ExplainNode { const children: ExplainNode[] = []; // CTEs from WITH clause go before the select columns, except when the WITH - // was written before an enclosing INSERT (`_with_trailing`), in which case + // was written before an enclosing INSERT (`with_trailing`), in which case // ClickHouse appends the WITH ExpressionList after the select body. - if (stmt.with && stmt.with.length > 0 && stmt._with_trailing !== true) { + if (stmt.with && stmt.with.length > 0 && stmt.with_trailing !== true) { children.push(n('ExpressionList', stmt.with.map(withItemNode))); } @@ -484,7 +484,7 @@ function selectQueryNode(stmt: SelectQueryNode): ExplainNode { // A WITH written before an enclosing INSERT, or propagated into a later UNION // member, is emitted after the select body. Within this propagated copy // ClickHouse also flips each joined element's TableJoin/TableExpression order. - if (stmt.with && stmt.with.length > 0 && stmt._with_trailing === true) { + if (stmt.with && stmt.with.length > 0 && stmt.with_trailing === true) { const prev = reverseTrailingJoins; reverseTrailingJoins = true; children.push(n('ExpressionList', stmt.with.map(withItemNode))); @@ -562,7 +562,7 @@ function jsonArgToExplain(node: ASTNode): ExplainNode { /** Native `CODEC(...)` / `STATISTICS(...)` function node → explain. */ function codecExplainNative(fnNode: FunctionNode): ExplainNode { const children = (fnNode.arguments as FunctionNode[]).map((c) => - c._no_parens === true ? n(`Function ${c.name}`) : functionNode(c.name, c.arguments), + c.no_parens === true ? n(`Function ${c.name}`) : functionNode(c.name, c.arguments), ); return n(`Function ${fnNode.name}`, [n('ExpressionList', children)]); } @@ -624,7 +624,7 @@ function projectionExplainNative(proj: ProjectionNode): ExplainNode { /** Native engine `Function` node → explain (no-parens engines omit ExpressionList). */ function engineExplainNative(engine: FunctionNode): ExplainNode { - if (engine._no_parens === true) return n(`Function ${engine.name}`); + if (engine.no_parens === true) return n(`Function ${engine.name}`); return functionNode(engine.name, engine.arguments); } @@ -663,7 +663,7 @@ function pushStorageOrderBy(children: ExplainNode[], storage: StorageNode): void function storageExplainNative(storage: StorageNode, columnsHavePk: boolean): ExplainNode | null { const children: ExplainNode[] = []; if (storage.engine) children.push(engineExplainNative(storage.engine)); - const settingsAfterOb = storage._settings_after_order_by === true; + const settingsAfterOb = storage.settings_after_order_by === true; if (storage.settings && !settingsAfterOb) children.push(SET); if (storage.partition_by !== undefined) children.push(exprNode(storage.partition_by)); // In ClickHouse's storage AST the PRIMARY KEY child is placed after ORDER BY @@ -1461,12 +1461,12 @@ function stmtNode(anyStmt: Statement): ExplainNode { const dq = anyStmt as DescribeQueryNode; // Rebuild children: TableExpression, then the optional FORMAT // Identifier / Settings in source order (preserved via - // `_settings_before_format`). + // `settings_before_format`). const children: ExplainNode[] = []; if (dq.table_expression !== undefined) { children.push(statementChildNode(dq.table_expression)); } - if (dq._settings_before_format === true) { + if (dq.settings_before_format === true) { if (dq.settings !== undefined) children.push(SET); if (dq.format !== undefined) children.push(identifier(dq.format)); } else { @@ -1863,11 +1863,11 @@ function queryWrapperNode(q: SelectWithUnionQueryNode): ExplainNode { children.push(n(`Literal '${escapeStringValue(String(q.out_file.value))}'`)); } // ClickHouse's AST child order follows the source order of SETTINGS vs FORMAT - // (`_settings_before_format`), so reproduce it here even though format() + // (`settings_before_format`), so reproduce it here even though format() // canonicalizes the order. - if (q.settings !== undefined && q._settings_before_format) children.push(SET); + if (q.settings !== undefined && q.settings_before_format) children.push(SET); if (q.format !== undefined) children.push(identifier(q.format)); - if (q.settings !== undefined && !q._settings_before_format) children.push(SET); + if (q.settings !== undefined && !q.settings_before_format) children.push(SET); return n('SelectWithUnionQuery', children); } diff --git a/src/format.ts b/src/format.ts index aaeeae645..0a187097d 100644 --- a/src/format.ts +++ b/src/format.ts @@ -555,9 +555,29 @@ function nativeAuthToAuth( })), }; } - const first = m.arguments?.[0]; - if (first && first.value !== undefined) return { secret: first.value }; - return {}; + const secret = m.arguments?.[0]?.value; + if (secret === undefined) return {}; // NO_PASSWORD / NOT IDENTIFIED + // Reconstruct the source auth keyword and secret introducer from the native + // `auth_type` enum plus the `contains_hash`/`contains_password` flags (the + // inverse of the grammar's mapping), so `format()` re-emits the `WITH` + // qualifier and the field round-trips exactly. + switch (m.auth_type) { + case 'PLAINTEXT_PASSWORD': + return { secret, authType: 'plaintext_password' }; + case 'SHA256_PASSWORD': + return { secret, authType: m.contains_hash ? 'sha256_hash' : 'sha256_password' }; + case 'DOUBLE_SHA1_PASSWORD': + return { secret, authType: m.contains_hash ? 'double_sha1_hash' : 'double_sha1_password' }; + case 'BCRYPT_PASSWORD': + return { secret, authType: m.contains_hash ? 'bcrypt_hash' : 'bcrypt_password' }; + case 'KERBEROS': + return { secret, authType: 'kerberos', secretKeyword: 'REALM' as const }; + case 'LDAP': + return { secret, authType: 'ldap', secretKeyword: 'SERVER' as const }; + default: + // No `auth_type` — a bare `IDENTIFIED BY '...'` (server-default method). + return { secret }; + } }); } @@ -1012,7 +1032,7 @@ function formatSettingScalar( if (typeof v === 'object') { const pairs = Object.entries(v).map( ([k, el]) => - `('${escapeString(k)}', ${formatLiteralValue(el.value_type, el.value, el._nonfinite)})`, + `('${escapeString(k)}', ${formatLiteralValue(el.value_type, el.value, el.nonfinite)})`, ); return `[${pairs.join(', ')}]`; } @@ -1504,9 +1524,11 @@ function formatShowFamilyQuery(stmt: ShowFamilyQueryNode, indent: string): strin // Formats the text after the IDENTIFIED keyword from an auth-method array. // Shared by CREATE USER and ALTER USER (the caller supplies the IDENTIFIED keyword). // -// Note: for password methods the auth-method *type* (e.g. `WITH sha256_password`) -// is not preserved in the AST — only the secret is stored — so it is not echoed -// here. SSH keys, by contrast, are reproduced verbatim. +// The auth-method type is re-emitted as `WITH ` (e.g. +// `WITH sha256_hash`) whenever it was recorded, so the qualifier round-trips; +// a bare `IDENTIFIED BY '...'` (no explicit method) omits it. The secret is +// introduced by `BY`/`REALM`/`SERVER` per {@link AuthenticationData.secretKeyword}, +// and SSH keys are reproduced verbatim. function formatAuthMethods(auth: AuthenticationData[]): string { const parts = auth .map((a) => { @@ -1516,8 +1538,9 @@ function formatAuthMethods(auth: AuthenticationData[]): string { ); return `WITH ssh_key BY ${keys.join(', ')}`; } - if (a.secret) return `BY '${escapeString(a.secret)}'`; - return ''; + if (a.secret === undefined) return ''; + const secretText = `${a.secretKeyword ?? 'BY'} '${escapeString(a.secret)}'`; + return a.authType !== undefined ? `WITH ${a.authType} ${secretText}` : secretText; }) .filter(Boolean); return parts.join(', '); @@ -2298,7 +2321,7 @@ function formatNativeCodecInner(fnNode: FunctionNode): string { .map((c) => { const fmtName = /^[a-zA-Z_][a-zA-Z0-9_]*$/.test(c.name) ? c.name : quoteIdent(c.name); // Canonicalize the no-argument form to empty parens (`Delta()`, not - // `Delta`); the two are semantically identical. `_no_parens` is retained + // `Delta`); the two are semantically identical. `no_parens` is retained // only so `formatExplain()` can reproduce ClickHouse's byte-exact AST. if (c.arguments !== undefined && c.arguments.length > 0) { return `${fmtName}(${c.arguments.map((a) => formatExpr(a, '')).join(', ')})`; @@ -2335,7 +2358,7 @@ function formatColumnDeclNode(col: ColumnDeclarationNode, indent: string): strin /** Render a native `TYPE name[(args)]` index-type function. */ function formatTypeFn(it: FunctionNode, indent: string): string { // Canonicalize the no-argument form to empty parens (`minmax()`, not - // `minmax`); `_no_parens` is retained only for `formatExplain()`. + // `minmax`); `no_parens` is retained only for `formatExplain()`. if (it.arguments.length > 0) { return `${it.name}(${it.arguments.map((a) => formatExpr(a, indent)).join(', ')})`; } @@ -2428,7 +2451,7 @@ function formatColumnsBlockNode(cols: ColumnsNode, indent: string, sep: string = /** Render a native engine `Function` node (`ENGINE = name[(args)]`). */ function formatEngineNode(engine: FunctionNode, indent: string): string { // Canonicalize the no-argument form to empty parens (`ENGINE = Memory()`, not - // `Memory`); `_no_parens` is retained only for `formatExplain()`. + // `Memory`); `no_parens` is retained only for `formatExplain()`. if (engine.arguments !== undefined && engine.arguments.length > 0) { return `${engine.name}(${engine.arguments.map((a) => formatExpr(a, indent)).join(', ')})`; } @@ -2506,7 +2529,7 @@ function formatStorageClausesNode( result += `\n${indent}TTL ${ttlStr}`; } // ClickHouse requires storage `SETTINGS` to be the last clause; canonicalize - // to that position. `_settings_after_order_by` is retained only so + // to that position. `settings_after_order_by` is retained only so // `formatExplain()` can reproduce ClickHouse's source-order `Set` child. if (storage.settings) { result += `\n${indent}SETTINGS ${formatSetPairs(storage.settings).join(', ')}`; @@ -3380,18 +3403,18 @@ function formatExpr(expr: Expression, indent: string): string { // wrapNaryOperandCore). function formatLiteral(expr: LiteralNode): string { - return formatLiteralValue(expr.value_type, expr.value, expr._nonfinite); + return formatLiteralValue(expr.value_type, expr.value, expr.nonfinite); } // Shared rendering for a Literal node and for the `{value_type, value, -// _nonfinite?}` elements of an `Array`/`Tuple` literal `value` list (the +// nonfinite?}` elements of an `Array`/`Tuple` literal `value` list (the // scalar cases are identical; collections recurse into their element lists). // Inline comments are not represented in the typed-element model, so they are // dropped (an accepted canonicalization of array/tuple literals). function formatLiteralValue( valueType: LiteralNode['value_type'], value: LiteralNode['value'], - nonfinite: LiteralNode['_nonfinite'], + nonfinite: LiteralNode['nonfinite'], ): string { switch (valueType) { case 'String': @@ -3402,7 +3425,7 @@ function formatLiteralValue( return 'NULL'; case 'Float64': // `value` cannot carry non-finite forms (`null`) or negative zero (`0`); - // `_nonfinite` supplies those. Finite values reconstruct from the double: + // `nonfinite` supplies those. Finite values reconstruct from the double: // `canonicalFloatText` guarantees a `.`/`e` so the output reparses as a // Float and not as a UInt64. return value === null @@ -3417,7 +3440,7 @@ function formatLiteralValue( const els = Array.isArray(value) ? value : []; return ( open + - els.map((el) => formatLiteralValue(el.value_type, el.value, el._nonfinite)).join(', ') + + els.map((el) => formatLiteralValue(el.value_type, el.value, el.nonfinite)).join(', ') + close ); } @@ -3751,7 +3774,7 @@ function formatDescribeQuery(stmt: DescribeQueryNode, indent: string): string { let result = `${indent}DESCRIBE TABLE ${target}`; if (te?.final) result += ' FINAL'; // Canonicalize `SETTINGS ... FORMAT ...` to `FORMAT ... SETTINGS ...` (a - // syntactic no-op). `_settings_before_format` is retained only so + // syntactic no-op). `settings_before_format` is retained only so // `formatExplain()` can reproduce ClickHouse's source-order child list. if (stmt.format !== undefined) result += ` FORMAT ${stmt.format}`; result += settingsTrailerSuffix(stmt); @@ -4244,10 +4267,10 @@ function formatFunction(expr: FunctionNode, indent: string): string { // Function Array reaches this case for two reasons: at least one // non-dumpable element (always Function on reparse), or the original // source had a parenthesized element (`[(NULL)]`) whose - // `_parenthesized` marker is stripped from the public AST. To keep + // `parenthesized` marker is stripped from the public AST. To keep // the Function shape across `parse → format → parse` in the second // case, wrap the first dump-eligible Literal in parens so the - // reparsed first element carries `_parenthesized` and `plainElem` + // reparsed first element carries `parenthesized` and `plainElem` // again returns null — forcing the Function form. result = `[${formatFunctionArrayArgs(expr.arguments, indent)}]`; break; diff --git a/src/grammar.pegjs b/src/grammar.pegjs index 172de9a07..19a343a30 100644 --- a/src/grammar.pegjs +++ b/src/grammar.pegjs @@ -154,12 +154,12 @@ // is the IEEE double (a lossy JS number). Two forms cannot be recovered // from the native AST's `value`: non-finite values collapse to `null`, and // negative zero collapses to `0`. Both are recorded in a library-only - // `_nonfinite` discriminator so format()/formatExplain() can reproduce them. + // `nonfinite` discriminator so format()/formatExplain() can reproduce them. function floatLit(raw, extra) { const v = floatValue(raw); const node = lit('Float64', v, extra); - if (v === null) node._nonfinite = nonfiniteToken(raw); - else if (v === 0 && Object.is(Number(raw), -0)) node._nonfinite = '-0'; + if (v === null) node.nonfinite = nonfiniteToken(raw); + else if (v === 0 && Object.is(Number(raw), -0)) node.nonfinite = '-0'; return node; } @@ -188,7 +188,7 @@ // Negate a Literal's `value` field, preserving its native shape: UInt64/ // Int64 are decimal-digit strings (use BigInt for precision), Float64 is // a JS number (non-finite `null` is left untouched — see `negateNumericLit` - // for the `_nonfinite` flip). Used by the `::` cast-fold negation path; + // for the `nonfinite` flip). Used by the `::` cast-fold negation path; // other literal types should never reach this helper. function negateLitValue(node) { if (node.value_type === 'UInt64' || node.value_type === 'Int64') { @@ -209,28 +209,28 @@ if (n.value_type !== 'UInt64' && n.value_type !== 'Int64' && n.value_type !== 'Float64') { return false; } - if (n._nonfinite !== undefined) return n._nonfinite.charAt(0) !== '-'; + if (n.nonfinite !== undefined) return n.nonfinite.charAt(0) !== '-'; if (typeof n.value === 'number') return !(n.value < 0); return n.value.charAt(0) !== '-'; } // Return a negated copy of a numeric literal. For Float64 this also flips - // the `_nonfinite` sign discriminator: inf↔-inf, nan↔-nan, +0↔-0 (the cases + // the `nonfinite` sign discriminator: inf↔-inf, nan↔-nan, +0↔-0 (the cases // `value` alone can't represent). function negateNumericLit(n) { const out = { ...n, value: negateLitValue(n) }; if (n.value_type === 'Float64') { - if (n._nonfinite !== undefined) { - const flipped = n._nonfinite === '-0' + if (n.nonfinite !== undefined) { + const flipped = n.nonfinite === '-0' ? undefined - : n._nonfinite.charAt(0) === '-' - ? n._nonfinite.substring(1) - : '-' + n._nonfinite; - if (flipped === undefined) delete out._nonfinite; - else out._nonfinite = flipped; + : n.nonfinite.charAt(0) === '-' + ? n.nonfinite.substring(1) + : '-' + n.nonfinite; + if (flipped === undefined) delete out.nonfinite; + else out.nonfinite = flipped; } else if (out.value === 0) { // Negating +0 yields -0. - out._nonfinite = '-0'; + out.nonfinite = '-0'; } } return out; @@ -246,12 +246,12 @@ } // Plain dump form of a Float64 literal node. Non-finite values (`value` is - // null) come from `_nonfinite`; `nan`/`-nan` both dump as `nan`. + // null) come from `nonfinite`; `nan`/`-nan` both dump as `nan`. function floatDump(node) { if (node.value === null) { - return node._nonfinite === 'nan' || node._nonfinite === '-nan' ? 'nan' : node._nonfinite; + return node.nonfinite === 'nan' || node.nonfinite === '-nan' ? 'nan' : node.nonfinite; } - if (node._nonfinite === '-0') return '-0.'; + if (node.nonfinite === '-0') return '-0.'; return floatNumText(node.value); } @@ -317,7 +317,7 @@ // ── Literal Array/Tuple typed-element model ────────────────────────────────── // ClickHouse serializes all-literal array/tuple syntax as a single Literal // node whose `value` is the list of its elements, each a {value_type, value} - // object (recursively for nested Array/Tuple, with `_nonfinite` carried for + // object (recursively for nested Array/Tuple, with `nonfinite` carried for // non-finite/`-0` Float64 elements). Elements that can't be folded force the // Function array/tuple form instead. @@ -338,7 +338,7 @@ // collection type is permitted as an element: 'array' allows nested Array // (not Tuple), 'tuple' allows nested Tuple (not Array). function litElem(node, allowNested) { - if (node._parenthesized) return null; + if (node.parenthesized) return null; if (node.alias !== undefined) return null; if (node.type !== 'Literal') return null; switch (node.value_type) { @@ -359,7 +359,7 @@ return null; } const el = { value_type: node.value_type, value: node.value }; - if (node._nonfinite !== undefined) el._nonfinite = node._nonfinite; + if (node.nonfinite !== undefined) el.nonfinite = node.nonfinite; return el; } @@ -373,11 +373,11 @@ return litElem(node, 'array'); } - // Rebuild a Literal Array/Tuple element ({value_type, value, _nonfinite?}) + // Rebuild a Literal Array/Tuple element ({value_type, value, nonfinite?}) // into a Literal Expression node (used when a folded Literal collection must // be expanded back into Function array/tuple arguments). function elemToExpr(el) { - const extra = el._nonfinite !== undefined ? { _nonfinite: el._nonfinite } : undefined; + const extra = el.nonfinite !== undefined ? { nonfinite: el.nonfinite } : undefined; return lit(el.value_type, el.value, extra); } @@ -476,7 +476,7 @@ } const node = { type: typeName || 'Settings' }; if (hasChanges) node.changes = changes; - if (Object.keys(changeValueTypes).length > 0) node._change_value_types = changeValueTypes; + if (Object.keys(changeValueTypes).length > 0) node.change_value_types = changeValueTypes; if (defaults.length > 0) node.default_settings = defaults; return withLoc(node, spanOf(items) ?? spanOf(items.map((i) => i.value))); } @@ -795,7 +795,7 @@ // EXPLAIN AST text dump shows `SelectQuery (children 4)`. The JSON AST // (and our named `select`/`from` fields) keep a single copy; this flag // tells the explain projection to emit the doubled child list. - _agg_repeat: true, + agg_repeat: true, }, L); return withLoc({ type: 'Subquery', query: withLoc(wrapSWU([sel]), L) }, L); } @@ -880,7 +880,7 @@ // Canonical dump text of one typed-list element, or null when it is not a // pure-cast literal (Null/Bool). Recurses through nested Array/Tuple. Typed - // elements share the Literal node's `value`/`_nonfinite` shape, so `floatDump` + // elements share the Literal node's `value`/`nonfinite` shape, so `floatDump` // handles the Float64 case directly. function elemCastText(el) { switch (el.value_type) { @@ -970,7 +970,7 @@ // Build a CAST Function node for the `::` operator form. Pure-literal // operands are stored as their String literal text, as ClickHouse does. - // The structured operand is parked on a parse-time-only `_cast_operand` + // The structured operand is parked on a parse-time-only `cast_operand` // marker so the unary-minus negate-fold can distinguish a folded `::` // cast (which folds `-1::Int8` → `CAST('-1','Int8')`) from a user-written // `CAST('1','Int8')` (which must stay `negate(CAST('1','Int8'))`). The @@ -992,7 +992,7 @@ } if (text !== null) { // Stringified pure-literal casts are plain CAST calls in ClickHouse's AST - const stringified = withLoc(strLit(text, { _cast_operand: operand }), operand.location); + const stringified = withLoc(strLit(text, { cast_operand: operand }), operand.location); return withLoc(fn('CAST', [stringified, withLoc(typeLit(rawType), operand.location)]), operand.location); } return withLoc(fn('CAST', [operand, withLoc(typeLit(rawType), operand.location)], { is_operator: true }), operand.location); @@ -1142,7 +1142,7 @@ if (node.union_mode === 'UNION_DISTINCT') { // Explicitly parenthesized DISTINCT groups keep their serialized mode; // precedence-implied groups have it removed (ClickHouse mode REMOVED). - if (node._parenthesized === true) return [node]; + if (node.parenthesized === true) return [node]; const group = { ...node }; delete group.union_mode; return [group]; @@ -1165,7 +1165,7 @@ // Wrap an intersect/except child in SelectWithUnionQuery when required. function intersectChild(node, wrap) { if (node.type === 'SelectWithUnionQuery') return node; - if (node._parenthesized) wrap = true; + if (node.parenthesized) wrap = true; return wrap ? wrapSWU([node]) : node; } @@ -1208,7 +1208,7 @@ // A WITH that scopes the whole union is emitted first on the member that // declared it but appended last on every member it propagates into, so // the distributed copies are tagged trailing. - return { ...m, with: first.with, _with_trailing: true }; + return { ...m, with: first.with, with_trailing: true }; }); } @@ -1227,7 +1227,7 @@ if (isLeftmostPath) return q; if (q.with !== undefined && q.with.length > 0) return q; // Propagated (non-leftmost) copies are appended last by ClickHouse. - return { ...q, with: withItems, _with_trailing: true }; + return { ...q, with: withItems, with_trailing: true }; } if (q.type === 'SelectIntersectExceptQuery') { return { @@ -1258,9 +1258,9 @@ // A WITH written before INSERT (`WITH ... INSERT INTO ... SELECT ...`) // is appended to the inner SELECT's child list by ClickHouse, so its // EXPLAIN AST emits the WITH ExpressionList *after* the select body. - // `_with_trailing` records that source position for the explain + // `with_trailing` records that source position for the explain // projection (format() still re-emits it before the statement). - return { ...q, with: withItems, _with_trailing: true }; + return { ...q, with: withItems, with_trailing: true }; } if (q.type === 'SelectWithUnionQuery') { const selects = q.selects.slice(); @@ -1328,7 +1328,7 @@ if (preSet !== undefined) { if (preSet.changes !== undefined) { for (const key of Object.keys(preSet.changes)) { - preEntries.push(key + ' = ' + explainSettingText(preSet.changes[key], preSet._change_value_types && preSet._change_value_types[key])); + preEntries.push(key + ' = ' + explainSettingText(preSet.changes[key], preSet.change_value_types && preSet.change_value_types[key])); } } if (preSet.default_settings !== undefined) { @@ -1689,11 +1689,11 @@ ...(setN.changes || {}), }; const mergedValueTypes = { - ...(selectInner._change_value_types || {}), - ...(setN._change_value_types || {}), + ...(selectInner.change_value_types || {}), + ...(setN.change_value_types || {}), }; if (Object.keys(mergedChanges).length > 0) setN.changes = mergedChanges; - if (Object.keys(mergedValueTypes).length > 0) setN._change_value_types = mergedValueTypes; + if (Object.keys(mergedValueTypes).length > 0) setN.change_value_types = mergedValueTypes; const insertKeys = new Set(Object.keys(setN.changes || {})); const seen = new Set(); const mergedDefaults = []; @@ -1719,8 +1719,8 @@ // SELECT's own settings, so no origin marker is needed. const setN = { type: 'Settings' }; if (selectInner.changes !== undefined) setN.changes = { ...selectInner.changes }; - if (selectInner._change_value_types !== undefined) { - setN._change_value_types = { ...selectInner._change_value_types }; + if (selectInner.change_value_types !== undefined) { + setN.change_value_types = { ...selectInner.change_value_types }; } if (selectInner.default_settings !== undefined) { setN.default_settings = [...selectInner.default_settings]; @@ -1815,7 +1815,7 @@ function cloneAst(v) { if (Array.isArray(v)) { const a = v.map(cloneAst); - // Preserve custom non-index array props (e.g. ORDER BY `_parenthesized`). + // Preserve custom non-index array props (e.g. ORDER BY `parenthesized`). for (const k of Object.keys(v)) if (!/^\d+$/.test(k)) a[k] = cloneAst(v[k]); return a; } @@ -2050,7 +2050,7 @@ // (`ENGINE = MergeTree`) and the empty-parens form (`MergeTree()`), but its // EXPLAIN text and SHOW CREATE distinguish them. Mark the no-parens form so // format()/explain can re-emit it. - if (args === undefined) node._no_parens = true; + if (args === undefined) node.no_parens = true; return withLoc(node, l ?? spanOf(args)); } @@ -2170,7 +2170,7 @@ // EXPLAIN (the native `Set` child order follows the source), so keep that // flag. `PRIMARY KEY` position has no such effect: format() canonicalizes // it to ClickHouse's `PRIMARY KEY` … `ORDER BY` order. - if (stmt.settingsAfterOrderBy === true) node._settings_after_order_by = true; + if (stmt.settingsAfterOrderBy === true) node.settings_after_order_by = true; // Empty when none of the optional fields applied. if ( node.engine === undefined && @@ -2255,7 +2255,7 @@ // formatter. function storageOrderByNode(orderBy) { const hasDesc = orderBy.some((o) => o.dir === 'DESC'); - const parenthesized = !!orderBy._parenthesized; + const parenthesized = !!orderBy.parenthesized; const sobe = (expr, dir) => withLoc({ type: 'StorageOrderByElement', expression: expr, @@ -2775,10 +2775,10 @@ switch (cmd.commandType) { case 'ADD_COLUMN': if (cmd.column) node.column_declaration = columnDeclNode(cmd.column); - if (cmd.afterColumn) node.column = colRefIdent(cmd.afterColumn, cmd._afterColumnParts, cmd.location); + if (cmd.afterColumn) node.column = colRefIdent(cmd.afterColumn, cmd.afterColumnParts, cmd.location); break; case 'DROP_COLUMN': - if (cmd.columnName) node.column = colRefIdent(cmd.columnName, cmd._columnNameParts, cmd.location); + if (cmd.columnName) node.column = colRefIdent(cmd.columnName, cmd.columnNameParts, cmd.location); if (cmd.clear === true) node.clear_column = true; if (cmd.partition) node.partition = alterPartitionNode(cmd.partition); break; @@ -2792,20 +2792,20 @@ node.settings_changes = setNode(op.settings || []); } } - if (cmd.afterColumn) node.column = colRefIdent(cmd.afterColumn, cmd._afterColumnParts, cmd.location); + if (cmd.afterColumn) node.column = colRefIdent(cmd.afterColumn, cmd.afterColumnParts, cmd.location); if (cmd.removeProperty) node.remove_property = cmd.removeProperty; break; } case 'RENAME_COLUMN': - if (cmd.oldName) node.column = colRefIdent(cmd.oldName, cmd._oldNameParts, cmd.location); - if (cmd.newName) node.rename_to = colRefIdent(cmd.newName, cmd._newNameParts, cmd.location); + if (cmd.oldName) node.column = colRefIdent(cmd.oldName, cmd.oldNameParts, cmd.location); + if (cmd.newName) node.rename_to = colRefIdent(cmd.newName, cmd.newNameParts, cmd.location); break; case 'COMMENT_COLUMN': - if (cmd.columnName) node.column = colRefIdent(cmd.columnName, cmd._columnNameParts, cmd.location); + if (cmd.columnName) node.column = colRefIdent(cmd.columnName, cmd.columnNameParts, cmd.location); if (cmd.comment) node.comment = cmd.comment; break; case 'MATERIALIZE_COLUMN': - if (cmd.columnName) node.column = colRefIdent(cmd.columnName, cmd._columnNameParts, cmd.location); + if (cmd.columnName) node.column = colRefIdent(cmd.columnName, cmd.columnNameParts, cmd.location); if (cmd.partition) node.partition = alterPartitionNode(cmd.partition); break; case 'ADD_INDEX': @@ -4273,7 +4273,7 @@ TopLevelStatement // SETTINGS before and/or after FORMAT collapse onto the query wrapper's // native `settings` field. The pre-/post-FORMAT position is a purely // syntactic no-op, but ClickHouse's EXPLAIN AST child order preserves it, - // so we record a library-only `_settings_before_format` hint (used only + // so we record a library-only `settings_before_format` hint (used only // by formatExplain(); format() canonicalizes to SETTINGS after FORMAT). const settingItems = [ ...(preSettings !== null ? preSettings[1] : []), @@ -4282,7 +4282,7 @@ TopLevelStatement if (settingItems.length > 0) { if (isNewNode) { result = { ...result, settings: setNode(settingItems) }; - if (preSettings !== null) result = { ...result, _settings_before_format: true }; + if (preSettings !== null) result = { ...result, settings_before_format: true }; } else { result = { ...result, postFormatSettings: settingItems }; } @@ -4440,7 +4440,7 @@ AlterCommandAddColumn const result = loc({ kind: 'alterCommand', commandType: 'ADD_COLUMN', column: col }); if (after !== null && after[1] && after[4] && typeof after[4] === 'object') { result.afterColumn = after[4].name; - result._afterColumnParts = after[4].parts; + result.afterColumnParts = after[4].parts; } else if (after !== null && after[1] && after[1].toUpperCase() === 'FIRST') { result.first = true; } @@ -4453,7 +4453,7 @@ AlterCommandDropColumn ifExists:( _ "IF"i ![a-zA-Z0-9_] _ "EXISTS"i ![a-zA-Z0-9_] )? _ col:AlterColumnRef partition:( _ AlterInPartitionClause )? { - const result = loc({ kind: 'alterCommand', commandType: 'DROP_COLUMN', columnName: col.name, _columnNameParts: col.parts }); + const result = loc({ kind: 'alterCommand', commandType: 'DROP_COLUMN', columnName: col.name, columnNameParts: col.parts }); if (ifExists !== null) result.ifExists = true; if (partition !== null) result.partition = partition[1]; return result; @@ -4468,7 +4468,7 @@ AlterCommandClearColumn kind: 'alterCommand', commandType: 'DROP_COLUMN', columnName: col.name, - _columnNameParts: col.parts, + columnNameParts: col.parts, clear: true, }); if (ifExists !== null) result.ifExists = true; @@ -4495,7 +4495,7 @@ AlterCommandModifyColumn if (modifySetReset !== null) result.columnSettingOp = modifySetReset[1]; if (after !== null && after[1] && after[4] && typeof after[4] === 'object') { result.afterColumn = after[4].name; - result._afterColumnParts = after[4].parts; + result.afterColumnParts = after[4].parts; } else if (after !== null && after[1] && after[1].toUpperCase() === 'FIRST') { result.first = true; } @@ -4571,8 +4571,8 @@ AlterCommandRenameColumn commandType: 'RENAME_COLUMN', oldName: oldName.name, newName: newName.name, - _oldNameParts: oldName.parts, - _newNameParts: newName.parts, + oldNameParts: oldName.parts, + newNameParts: newName.parts, }); if (ifExists !== null) result.ifExists = true; return result; @@ -4586,7 +4586,7 @@ AlterCommandCommentColumn kind: 'alterCommand', commandType: 'COMMENT_COLUMN', columnName: col.name, - _columnNameParts: col.parts, + columnNameParts: col.parts, comment, }); if (ifExists !== null) result.ifExists = true; @@ -4602,7 +4602,7 @@ AlterCommandMaterializeColumn kind: 'alterCommand', commandType: 'MATERIALIZE_COLUMN', columnName: col.name, - _columnNameParts: col.parts, + columnNameParts: col.parts, }); if (ifExists !== null) result.ifExists = true; if (partition !== null) result.partition = partition[1]; @@ -5902,7 +5902,7 @@ DescribeStatement if (format !== null) node.format = format[1]; // Preserve source ordering: `SETTINGS ... FORMAT name` vs // `FORMAT name SETTINGS ...`. - if (preSettings !== null && format !== null) node._settings_before_format = true; + if (preSettings !== null && format !== null) node.settings_before_format = true; return loc(node); } @@ -6987,10 +6987,10 @@ CreateTableStatement if (fromPath !== null) result.attachFromPath = fromPath[4].value; if (format !== null) result.format = format[1]; // Promote column-level PRIMARY KEY to primaryKey if no explicit PRIMARY KEY clause - if (result._columnPrimaryKeys && !result.primaryKey) { - result.primaryKey = result._columnPrimaryKeys; + if (result.columnPrimaryKeys && !result.primaryKey) { + result.primaryKey = result.columnPrimaryKeys; } - delete result._columnPrimaryKeys; + delete result.columnPrimaryKeys; return loc(createTableNode(result)); } @@ -7145,8 +7145,8 @@ CreateTableSchema let primaryKeyInSchema = null; const columnPrimaryKeys = []; for (const el of elements) { - if (el._primaryKey) { - primaryKeyInSchema = el._primaryKey; + if (el.primaryKeyExprs) { + primaryKeyInSchema = el.primaryKeyExprs; } else { tableElements.push(el); if (el.kind === 'columnDef' && el.primaryKey) { @@ -7157,7 +7157,7 @@ CreateTableSchema const result = {}; if (tableElements.length > 0) result.tableElements = tableElements; if (primaryKeyInSchema !== null) result.primaryKeyInSchema = primaryKeyInSchema; - if (columnPrimaryKeys.length > 0) result._columnPrimaryKeys = columnPrimaryKeys; + if (columnPrimaryKeys.length > 0) result.columnPrimaryKeys = columnPrimaryKeys; return result; } @@ -7328,7 +7328,7 @@ ProjectionElement PrimaryKeyElement = "PRIMARY"i ![a-zA-Z0-9_] _ "KEY"i ![a-zA-Z0-9_] _ exprs:PrimaryKeyExprs { exprs.location = location(); - return { _primaryKey: exprs }; + return { primaryKeyExprs: exprs }; } PrimaryKeyExprs @@ -7386,7 +7386,7 @@ CreateTableClause CreateOrderByClause = "ORDER"i ![a-zA-Z0-9_] _ "BY"i ![a-zA-Z0-9_] _ "(" _ head:CreateOrderByItem tail:(_ "," _ CreateOrderByItem)* _ ")" !( _ [*/%+\-] ) { const items = [head, ...tail.map(t => t[3])]; - items._parenthesized = true; + items.parenthesized = true; return items; } / "ORDER"i ![a-zA-Z0-9_] _ "BY"i ![a-zA-Z0-9_] _ head:CreateOrderByItem tail:(_ "," _ CreateOrderByItem)* { @@ -7608,13 +7608,13 @@ UnionQueryAtom query.selects.length === 1 && (query.selects[0].type === 'SelectQuery' || query.selects[0].type === 'SelectIntersectExceptQuery') ) { - let result = { ...query.selects[0], _parenthesized: true }; + let result = { ...query.selects[0], parenthesized: true }; if (query.leadingComments !== undefined) result = addLeading(result, query.leadingComments); if (query.trailingComments !== undefined) result = addTrailing(result, query.trailingComments); return result; } if (query.type === 'SelectWithUnionQuery' || query.type === 'SelectIntersectExceptQuery') { - return { ...query, _parenthesized: true }; + return { ...query, parenthesized: true }; } return query; } @@ -8350,7 +8350,7 @@ CompareRightSuffix return (left) => (loc(fn('match', [left, right], { is_operator: true }))); } // BETWEEN expands to operator-form comparisons, as ClickHouse stores it. - // The synthesized n-ary wrapper is marked `_parenthesized: true` so the + // The synthesized n-ary wrapper is marked `parenthesized: true` so the // formatted output `x >= a AND x <= b` reparses to an equivalent node // structure (the explicit parens that wrap_nary would emit are recorded // here). @@ -8358,13 +8358,13 @@ CompareRightSuffix return (left) => (loc(fn('or', [ loc(opFn('<', [left, low])), loc(opFn('>', [left, high])) - ], { is_operator: true, _parenthesized: true }))); + ], { is_operator: true, parenthesized: true }))); } / _ KW_BETWEEN _ low:AddExpr _ KW_AND _ high:AddExpr { return (left) => (loc(fn('and', [ loc(opFn('>=', [left, low])), loc(opFn('<=', [left, high])) - ], { is_operator: true, _parenthesized: true }))); + ], { is_operator: true, parenthesized: true }))); } // :: cast operator at comparison level so `x IS NULL :: Type` works as cast(isNull(x), Type). // (At PrimaryExpr level, `x::Type` is also handled for tighter-binding casts on bare values.) @@ -8494,40 +8494,40 @@ UnaryExpr // Don't fold across explicit parentheses: -(1) must produce negate(1), not Int64_-1. // (-1) without an outer minus stays as a UInt64 literal — only `-` immediately before a // bare literal folds. A parenthesized inner expression always wraps in negate(). - if (expr._parenthesized) { + if (expr.parenthesized) { return loc(fn('negate', [expr], { is_operator: true })); } - if (expr.type === 'Literal' && expr.value_type === 'UInt64' && expr._neg_folded === undefined) { + if (expr.type === 'Literal' && expr.value_type === 'UInt64' && expr.neg_folded === undefined) { // Negate a non-negative integer literal: compute decimal value using BigInt for precision. - // A UInt64 carrying `_neg_folded` is itself a folded `-0` — `- -0` stays negate(0). + // A UInt64 carrying `neg_folded` is itself a folded `-0` — `- -0` stays negate(0). const bigNeg = -BigInt(expr.value); // Check if fits in Int64 range [-2^63, 0] const INT64_MIN = BigInt('-9223372036854775808'); if (bigNeg >= INT64_MIN) { - // -0 stays a UInt64 in ClickHouse; mark with `_neg_folded` so the + // -0 stays a UInt64 in ClickHouse; mark with `neg_folded` so the // outer minus of `- -0` doesn't re-fold. The marker is parse-time // only and is stripped before the AST is returned (see // `stripParseTimeMarkers` in src/index.ts). const intResult = loc(bigNeg === BigInt(0) ? uintLit('0') : intLit(String(bigNeg))); - if (bigNeg === BigInt(0)) intResult._neg_folded = true; + if (bigNeg === BigInt(0)) intResult.neg_folded = true; return intResult; } // Overflows Int64: use Float64 (loses precision like ClickHouse does) return loc(floatLit(String(Number(bigNeg)))); } if (expr.type === 'Literal' && expr.value_type === 'Float64' && isNonNegNumericLit(expr)) { - // Negate a positive float literal (works from `value`/`_nonfinite`). + // Negate a positive float literal (works from `value`/`nonfinite`). return loc(negateNumericLit(expr)); } // Identify a `::`-form CAST: either the structured-operand form // (is_operator: true) or the folded pure-literal form (operand is a - // String literal carrying the parse-time `_cast_operand` marker — + // String literal carrying the parse-time `cast_operand` marker — // stripped from the final AST, but visible during parsing). const isOpCast = (e) => { if (e.type !== 'Function' || e.name !== 'CAST') return false; if (e.is_operator === true) return true; const a0 = e.arguments[0]; - return a0 !== undefined && a0.type === 'Literal' && a0._cast_operand !== undefined; + return a0 !== undefined && a0.type === 'Literal' && a0.cast_operand !== undefined; }; if (isOpCast(expr)) { // -value::Type: fold the minus sign into the cast's innermost literal (for :: operator casts) @@ -8538,11 +8538,11 @@ UnaryExpr } const inner = innermost.arguments[0]; // The operand may have been stringified (pure-literal :: casts) - if (inner.type === 'Literal' && inner.value_type === 'String' && inner._cast_operand !== undefined) { - const orig = inner._cast_operand; + if (inner.type === 'Literal' && inner.value_type === 'String' && inner.cast_operand !== undefined) { + const orig = inner.cast_operand; if (isNonNegNumericLit(orig)) { const negOrig = negateNumericLit(orig); - const negInner = { ...inner, value: '-' + inner.value, _cast_operand: negOrig }; + const negInner = { ...inner, value: '-' + inner.value, cast_operand: negOrig }; let result = { ...innermost, arguments: [negInner, innermost.arguments[1]] }; const stack = []; let cur = expr; @@ -8993,7 +8993,7 @@ ParenGroup if (rest.length === 0 && trailing === null) { // (expr) — parenthesized expression first = addTrailing(first, flattenWs(afterLast)); - return { ...first, _parenthesized: true }; + return { ...first, parenthesized: true }; } else if (rest.length === 0) { // (expr,) — single-element tuple return loc(fn('tuple', [first], { is_operator: true })); diff --git a/src/index.ts b/src/index.ts index ea532dd11..984d20966 100644 --- a/src/index.ts +++ b/src/index.ts @@ -84,7 +84,7 @@ export type { * * Three markers are handled: * - * - `_parenthesized: true` — set on (a) expression nodes (Literal, + * - `parenthesized: true` — set on (a) expression nodes (Literal, * Identifier, Function, Asterisk, SelectQuery, SelectWithUnionQuery, * SelectIntersectExceptQuery) and on (b) the storage `orderBy` array. * Read at parse time by the unary-minus folding rule, tuple/array element @@ -96,7 +96,7 @@ export type { * `orderByParenthesized` field on the parent statement (set by the * grammar itself); we just delete the array-property version here. * - * - `_cast_operand` — set on the stringified first argument of a + * - `cast_operand` — set on the stringified first argument of a * pure-literal `::` cast (`1::UInt8` folds the operand to `Literal * String '1'`, matching ClickHouse's native AST). Read at parse time * only, by the unary-minus negate-fold, to distinguish a folded `::` @@ -106,14 +106,14 @@ export type { * distinguish the two forms; the formatter accepts the canonicalization * `1::UInt8` → `CAST('1' AS UInt8)` at output time. * - * - `_neg_folded: true` — set on a `UInt64(0)` literal produced by folding + * - `neg_folded: true` — set on a `UInt64(0)` literal produced by folding * `-0`. Read at parse time only, by the unary-minus rule, so that * `- -0` falls through to `negate(0)` (matching ClickHouse) instead of * re-folding to a bare `UInt64(0)`. The public AST cannot distinguish * `0` from `-0` (both are `UInt64 0`); the marker only governs the * parse-time wrapping decision. * - * - `_change_value_types` — records the source literal type of each + * - `change_value_types` — records the source literal type of each * `Settings.changes` entry. Consumed only during parsing (by the grammar's * EXPLAIN-settings and INSERT/SELECT settings-merge rules); `format()` and * `formatExplain()` canonicalize numeric settings to the quoted form and @@ -123,17 +123,17 @@ function stripParseTimeMarkers(value: unknown): void { if (value === null || typeof value !== 'object') return; if (Array.isArray(value)) { - // Delete the non-index `_parenthesized` array property (storage ORDER BY). - delete (value as unknown as { _parenthesized?: boolean })._parenthesized; + // Delete the non-index `parenthesized` array property (storage ORDER BY). + delete (value as unknown as { parenthesized?: boolean }).parenthesized; for (const item of value) stripParseTimeMarkers(item); return; } const obj = value as Record; - if ('_parenthesized' in obj) delete obj._parenthesized; - if ('_cast_operand' in obj) delete obj._cast_operand; - if ('_neg_folded' in obj) delete obj._neg_folded; - if ('_change_value_types' in obj) delete obj._change_value_types; + if ('parenthesized' in obj) delete obj.parenthesized; + if ('cast_operand' in obj) delete obj.cast_operand; + if ('neg_folded' in obj) delete obj.neg_folded; + if ('change_value_types' in obj) delete obj.change_value_types; for (const v of Object.values(obj)) stripParseTimeMarkers(v); } @@ -250,6 +250,7 @@ export function parse( export { format, formatNode } from './format'; export { formatExplain } from './explain'; +export { formatExplainJson } from './json-explain'; export { findNodes } from './find-nodes'; export { transformNodes, type NodePositionMap } from './transform-nodes'; export { diff --git a/src/json-explain.ts b/src/json-explain.ts new file mode 100644 index 000000000..5ada0f526 --- /dev/null +++ b/src/json-explain.ts @@ -0,0 +1,63 @@ +import { Statement, WithoutLocations } from './ast'; +import { LIBRARY_ONLY_FIELDS, METADATA_KEYS } from './meta'; + +function stripNonReference(value: unknown): unknown { + if (value === null || value === undefined || typeof value !== 'object') { + return value; + } + + if (Array.isArray(value)) { + return value.map(stripNonReference); + } + + const obj = value as Record; + const nodeType = typeof obj.type === 'string' ? obj.type : undefined; + + const result: Record = {}; + for (const [k, v] of Object.entries(obj)) { + if (METADATA_KEYS.has(k)) continue; + const field = LIBRARY_ONLY_FIELDS[k]; + if (field) { + const stripHere = + nodeType === undefined ? field.onUntyped === true : field.types.has(nodeType); + if (stripHere) continue; + } + result[k] = stripNonReference(v); + } + return result; +} + +/** Supported `EXPLAIN AST json = N` schema versions. Only 2 is implemented. */ +export type JsonExplainVersion = 2; + +/** One statement's projected AST, wrapped in its schema-version envelope. */ +export type JsonExplainEnvelope = { + version: JsonExplainVersion; + ast: unknown; +}; + +/** + * Projects this library's AST onto the ClickHouse-native view emitted by + * `EXPLAIN AST json = `: each statement's AST is deep-copied with the + * library-only fields removed — node metadata ({@link METADATA_KEYS}) on every + * node, and each {@link LIBRARY_ONLY_FIELDS} entry only on the node types it is + * declared for — then wrapped in a `{ version, ast }` envelope matching the + * reference `.expected.ast.json` files. Both key sets live in `./meta` + * alongside the round-trip `VOLATILE_KEYS` they keep in sync with. + * + * `version` must be `2`, the only schema this library emits; any other value + * throws. The result of `formatExplainJson(parse(sql), 2)` must equal + * ClickHouse's `EXPLAIN AST json = 2` output for the same statements; the + * reference ast test relies on this. + */ +export function formatExplainJson( + statements: WithoutLocations[], + version: JsonExplainVersion, +): JsonExplainEnvelope[] { + if (version !== 2) { + throw new Error( + `formatExplainJson: unsupported version ${version}; only version 2 is supported`, + ); + } + return statements.map((statement) => ({ version, ast: stripNonReference(statement) })); +} diff --git a/src/meta.ts b/src/meta.ts index 122482aee..a90d8ca06 100644 --- a/src/meta.ts +++ b/src/meta.ts @@ -1,105 +1,80 @@ /** - * Helpers for stripping library-side metadata from AST nodes. - * - * The AST has two views: - * - The ClickHouse-native view: the fields that `EXPLAIN AST json = 1` emits. - * The reference ast test compares this view (via {@link stripAstMeta}) - * against the expected JSON byte-for-byte. - * - The library view: everything the native JSON loses but that format() - * and formatExplain() need — names/flags dropped by the native - * serialization (stored in underscore-prefixed fields), comments, - * locations, and parent references. + * Metadata this library attaches to (nearly) every AST node but that + * ClickHouse's native JSON never emits. Absent from the reference AST. */ +export const METADATA_KEYS = new Set(['location', 'parent', 'leadingComments', 'trailingComments']); -/** - * True for keys that hold library-only data the ClickHouse-native JSON does - * not contain (e.g. `_name`, `_settings`, `_nonfinite`). - */ -export function isLibraryOnlyKey(key: string): boolean { - return key.startsWith('_'); -} - -const METADATA_KEYS = new Set(['location', 'parent', 'leadingComments', 'trailingComments']); +export type LibraryOnlyField = { + /** Node `type` discriminators the field may appear on. */ + types: ReadonlySet; + /** + * When true, the field may also appear on untyped helper objects (objects + * with no `type` discriminator), e.g. `Array`/`Tuple` literal value elements. + */ + onUntyped?: boolean; + /** + * When true, the field carries semantic information `format()` needs, so it + * must survive a round trip (it is *not* volatile). When false/absent the + * field is a non-semantic explain-only flag: read only by `formatExplain()`, + * canonicalized away by `format()`, and therefore volatile across a reformat. + */ + semantic?: boolean; + /** + * Reason why the field is required and cannot be dropped in favor of an existing + * reference AST field. + */ + rationale: string; +}; /** - * Recursively removes everything the ClickHouse-native AST JSON does not - * contain: `location`, `parent`, comments, and underscore-prefixed - * library-only fields. Underscore keys are stripped on every object, not just - * AST nodes — the native JSON never emits a `_`-prefixed key (verified across - * the reference corpus), so this can only remove our library-only additions - * (e.g. the per-element `_nonfinite` on `Array`/`Tuple` literal `value` - * elements, which have no `type` discriminator). `Set.changes` keeps all of - * its entries because ClickHouse setting names are never `_`-prefixed. - * - * The result of `stripAstMeta(parse(sql))` must equal ClickHouse's - * `EXPLAIN AST json = 1` output for the same statements. + * Library-only node fields absent from ClickHouse's reference AST, keyed by + * field name and mapped to the exact node `type` discriminators the field is + * allowed to appear on. */ -export function stripAstMeta(value: unknown): unknown { - if (value === null || value === undefined || typeof value !== 'object') { - return value; - } - - if (Array.isArray(value)) { - return value.map(stripAstMeta); - } - - const result: Record = {}; - for (const [k, v] of Object.entries(value as Record)) { - if (METADATA_KEYS.has(k)) continue; - if (isLibraryOnlyKey(k)) continue; - result[k] = stripAstMeta(v); - } - return result; -} +export const LIBRARY_ONLY_FIELDS: Record = { + no_parens: { + types: new Set(['Function']), + rationale: 'ClickHouse JSON AST does not distinguish no-parens vs empty-parens function calls', + }, + with_trailing: { + types: new Set(['SelectQuery']), + rationale: + 'ClickHouse JSON AST does not preserve the original child order of a WITH clause that appeared before an enclosing INSERT or propagated into a later UNION member', + }, + agg_repeat: { + types: new Set(['SelectQuery']), + rationale: + 'ClickHouse JSON AST does not preserve the synthetic SelectQuery produced when lowering expr op ANY/ALL (subquery), whose projection/tables ClickHouse text dump emits twice', + }, + settings_before_format: { + types: new Set(['SelectWithUnionQuery', 'SelectIntersectExceptQuery', 'DescribeQuery']), + rationale: + 'ClickHouse JSON AST does not preserve the original child order of SETTINGS clauses before FORMAT, which is required in formatExplain', + }, + settings_after_order_by: { + types: new Set(['Storage']), + rationale: + 'ClickHouse JSON AST does not preserve the original child order of SETTINGS clauses after ORDER BY, which is required in formatExplain', + }, + nonfinite: { + types: new Set(['Literal']), + onUntyped: true, + semantic: true, + rationale: + 'ClickHouse JSON AST collapses non-finite Float64 values to null, losing the original source spelling', + }, +}; /** - * Recursively removes only the volatile keys (`location`, `parent`, and - * `_with_trailing`) that legitimately differ between two parses of equivalent - * SQL. Comments and other library-only fields are kept, so round-trip - * comparisons still exercise their fidelity. - * - * `_with_trailing` records whether a `WITH` clause appeared before an enclosing - * `INSERT` / propagated into a later UNION member, purely so the explain - * projection can reproduce ClickHouse's child ordering. `format()` canonicalizes - * `WITH` placement, so the flag may flip across a reformat — like `location`, it - * is not a semantic property and is excluded from round-trip comparison. - * - * The authentication-method keys (`auth_type`, `contains_password`, - * `contains_hash`) are likewise excluded: `format()` intentionally re-emits - * `IDENTIFIED BY '...'` without the `WITH ` qualifier (the native AST - * preserves it, but the canonical SQL drops it), so these fields legitimately - * differ after a reformat. - * - * `_settings_before_format` records that a query wrapper's (or `DescribeQuery`'s) - * trailing `SETTINGS` preceded its `FORMAT` clause, purely so `formatExplain()` - * can reproduce ClickHouse's AST child order. `format()` canonicalizes the order - * (SETTINGS after FORMAT), so the flag may flip across a reformat — like - * `_with_trailing`, it is not a semantic property and is excluded from round-trip - * comparison. - * - * `_no_parens` records that an engine/codec/index-type function was written - * without argument parens (`ENGINE = Memory`, not `Memory()`). `format()` - * canonicalizes the no-argument form to empty parens (`Memory()`), so the flag - * may flip across a reformat; it survives only so `formatExplain()` can - * reproduce ClickHouse's byte-exact `Function`/`ExpressionList` child shape. - * - * `_settings_after_order_by` records that a storage `SETTINGS` clause followed - * `ORDER BY` in the source. `format()` canonicalizes `SETTINGS` to the last - * storage clause (its only valid position), so the flag may flip across a - * reformat; it survives only so `formatExplain()` can reproduce ClickHouse's - * source-order `Set` child. + * Keys removed before a round-trip comparison because they may legitimately + * differ between two parses of equivalent SQL. */ -const VOLATILE_KEYS = new Set([ +export const VOLATILE_KEYS: ReadonlySet = new Set([ 'location', 'parent', - '_with_trailing', - '_agg_repeat', - '_settings_before_format', - '_no_parens', - '_settings_after_order_by', - 'auth_type', - 'contains_password', - 'contains_hash', + ...Object.entries(LIBRARY_ONLY_FIELDS) + .filter(([, field]) => !field.semantic) + .map(([name]) => name), ]); export function stripVolatile(value: unknown, keepLocation = false): unknown { diff --git a/src/parser.js b/src/parser.js index af8da8b95..9b39dca08 100644 --- a/src/parser.js +++ b/src/parser.js @@ -159,12 +159,12 @@ // is the IEEE double (a lossy JS number). Two forms cannot be recovered // from the native AST's `value`: non-finite values collapse to `null`, and // negative zero collapses to `0`. Both are recorded in a library-only - // `_nonfinite` discriminator so format()/formatExplain() can reproduce them. + // `nonfinite` discriminator so format()/formatExplain() can reproduce them. function floatLit(raw, extra) { const v = floatValue(raw); const node = lit('Float64', v, extra); - if (v === null) node._nonfinite = nonfiniteToken(raw); - else if (v === 0 && Object.is(Number(raw), -0)) node._nonfinite = '-0'; + if (v === null) node.nonfinite = nonfiniteToken(raw); + else if (v === 0 && Object.is(Number(raw), -0)) node.nonfinite = '-0'; return node; } @@ -193,7 +193,7 @@ // Negate a Literal's `value` field, preserving its native shape: UInt64/ // Int64 are decimal-digit strings (use BigInt for precision), Float64 is // a JS number (non-finite `null` is left untouched — see `negateNumericLit` - // for the `_nonfinite` flip). Used by the `::` cast-fold negation path; + // for the `nonfinite` flip). Used by the `::` cast-fold negation path; // other literal types should never reach this helper. function negateLitValue(node) { if (node.value_type === 'UInt64' || node.value_type === 'Int64') { @@ -214,28 +214,28 @@ if (n.value_type !== 'UInt64' && n.value_type !== 'Int64' && n.value_type !== 'Float64') { return false; } - if (n._nonfinite !== undefined) return n._nonfinite.charAt(0) !== '-'; + if (n.nonfinite !== undefined) return n.nonfinite.charAt(0) !== '-'; if (typeof n.value === 'number') return !(n.value < 0); return n.value.charAt(0) !== '-'; } // Return a negated copy of a numeric literal. For Float64 this also flips - // the `_nonfinite` sign discriminator: inf↔-inf, nan↔-nan, +0↔-0 (the cases + // the `nonfinite` sign discriminator: inf↔-inf, nan↔-nan, +0↔-0 (the cases // `value` alone can't represent). function negateNumericLit(n) { const out = { ...n, value: negateLitValue(n) }; if (n.value_type === 'Float64') { - if (n._nonfinite !== undefined) { - const flipped = n._nonfinite === '-0' + if (n.nonfinite !== undefined) { + const flipped = n.nonfinite === '-0' ? undefined - : n._nonfinite.charAt(0) === '-' - ? n._nonfinite.substring(1) - : '-' + n._nonfinite; - if (flipped === undefined) delete out._nonfinite; - else out._nonfinite = flipped; + : n.nonfinite.charAt(0) === '-' + ? n.nonfinite.substring(1) + : '-' + n.nonfinite; + if (flipped === undefined) delete out.nonfinite; + else out.nonfinite = flipped; } else if (out.value === 0) { // Negating +0 yields -0. - out._nonfinite = '-0'; + out.nonfinite = '-0'; } } return out; @@ -251,12 +251,12 @@ } // Plain dump form of a Float64 literal node. Non-finite values (`value` is - // null) come from `_nonfinite`; `nan`/`-nan` both dump as `nan`. + // null) come from `nonfinite`; `nan`/`-nan` both dump as `nan`. function floatDump(node) { if (node.value === null) { - return node._nonfinite === 'nan' || node._nonfinite === '-nan' ? 'nan' : node._nonfinite; + return node.nonfinite === 'nan' || node.nonfinite === '-nan' ? 'nan' : node.nonfinite; } - if (node._nonfinite === '-0') return '-0.'; + if (node.nonfinite === '-0') return '-0.'; return floatNumText(node.value); } @@ -322,7 +322,7 @@ // ── Literal Array/Tuple typed-element model ────────────────────────────────── // ClickHouse serializes all-literal array/tuple syntax as a single Literal // node whose `value` is the list of its elements, each a {value_type, value} - // object (recursively for nested Array/Tuple, with `_nonfinite` carried for + // object (recursively for nested Array/Tuple, with `nonfinite` carried for // non-finite/`-0` Float64 elements). Elements that can't be folded force the // Function array/tuple form instead. @@ -343,7 +343,7 @@ // collection type is permitted as an element: 'array' allows nested Array // (not Tuple), 'tuple' allows nested Tuple (not Array). function litElem(node, allowNested) { - if (node._parenthesized) return null; + if (node.parenthesized) return null; if (node.alias !== undefined) return null; if (node.type !== 'Literal') return null; switch (node.value_type) { @@ -364,7 +364,7 @@ return null; } const el = { value_type: node.value_type, value: node.value }; - if (node._nonfinite !== undefined) el._nonfinite = node._nonfinite; + if (node.nonfinite !== undefined) el.nonfinite = node.nonfinite; return el; } @@ -378,11 +378,11 @@ return litElem(node, 'array'); } - // Rebuild a Literal Array/Tuple element ({value_type, value, _nonfinite?}) + // Rebuild a Literal Array/Tuple element ({value_type, value, nonfinite?}) // into a Literal Expression node (used when a folded Literal collection must // be expanded back into Function array/tuple arguments). function elemToExpr(el) { - const extra = el._nonfinite !== undefined ? { _nonfinite: el._nonfinite } : undefined; + const extra = el.nonfinite !== undefined ? { nonfinite: el.nonfinite } : undefined; return lit(el.value_type, el.value, extra); } @@ -481,7 +481,7 @@ } const node = { type: typeName || 'Settings' }; if (hasChanges) node.changes = changes; - if (Object.keys(changeValueTypes).length > 0) node._change_value_types = changeValueTypes; + if (Object.keys(changeValueTypes).length > 0) node.change_value_types = changeValueTypes; if (defaults.length > 0) node.default_settings = defaults; return withLoc(node, spanOf(items) ?? spanOf(items.map((i) => i.value))); } @@ -800,7 +800,7 @@ // EXPLAIN AST text dump shows `SelectQuery (children 4)`. The JSON AST // (and our named `select`/`from` fields) keep a single copy; this flag // tells the explain projection to emit the doubled child list. - _agg_repeat: true, + agg_repeat: true, }, L); return withLoc({ type: 'Subquery', query: withLoc(wrapSWU([sel]), L) }, L); } @@ -885,7 +885,7 @@ // Canonical dump text of one typed-list element, or null when it is not a // pure-cast literal (Null/Bool). Recurses through nested Array/Tuple. Typed - // elements share the Literal node's `value`/`_nonfinite` shape, so `floatDump` + // elements share the Literal node's `value`/`nonfinite` shape, so `floatDump` // handles the Float64 case directly. function elemCastText(el) { switch (el.value_type) { @@ -975,7 +975,7 @@ // Build a CAST Function node for the `::` operator form. Pure-literal // operands are stored as their String literal text, as ClickHouse does. - // The structured operand is parked on a parse-time-only `_cast_operand` + // The structured operand is parked on a parse-time-only `cast_operand` // marker so the unary-minus negate-fold can distinguish a folded `::` // cast (which folds `-1::Int8` → `CAST('-1','Int8')`) from a user-written // `CAST('1','Int8')` (which must stay `negate(CAST('1','Int8'))`). The @@ -997,7 +997,7 @@ } if (text !== null) { // Stringified pure-literal casts are plain CAST calls in ClickHouse's AST - const stringified = withLoc(strLit(text, { _cast_operand: operand }), operand.location); + const stringified = withLoc(strLit(text, { cast_operand: operand }), operand.location); return withLoc(fn('CAST', [stringified, withLoc(typeLit(rawType), operand.location)]), operand.location); } return withLoc(fn('CAST', [operand, withLoc(typeLit(rawType), operand.location)], { is_operator: true }), operand.location); @@ -1147,7 +1147,7 @@ if (node.union_mode === 'UNION_DISTINCT') { // Explicitly parenthesized DISTINCT groups keep their serialized mode; // precedence-implied groups have it removed (ClickHouse mode REMOVED). - if (node._parenthesized === true) return [node]; + if (node.parenthesized === true) return [node]; const group = { ...node }; delete group.union_mode; return [group]; @@ -1170,7 +1170,7 @@ // Wrap an intersect/except child in SelectWithUnionQuery when required. function intersectChild(node, wrap) { if (node.type === 'SelectWithUnionQuery') return node; - if (node._parenthesized) wrap = true; + if (node.parenthesized) wrap = true; return wrap ? wrapSWU([node]) : node; } @@ -1213,7 +1213,7 @@ // A WITH that scopes the whole union is emitted first on the member that // declared it but appended last on every member it propagates into, so // the distributed copies are tagged trailing. - return { ...m, with: first.with, _with_trailing: true }; + return { ...m, with: first.with, with_trailing: true }; }); } @@ -1232,7 +1232,7 @@ if (isLeftmostPath) return q; if (q.with !== undefined && q.with.length > 0) return q; // Propagated (non-leftmost) copies are appended last by ClickHouse. - return { ...q, with: withItems, _with_trailing: true }; + return { ...q, with: withItems, with_trailing: true }; } if (q.type === 'SelectIntersectExceptQuery') { return { @@ -1263,9 +1263,9 @@ // A WITH written before INSERT (`WITH ... INSERT INTO ... SELECT ...`) // is appended to the inner SELECT's child list by ClickHouse, so its // EXPLAIN AST emits the WITH ExpressionList *after* the select body. - // `_with_trailing` records that source position for the explain + // `with_trailing` records that source position for the explain // projection (format() still re-emits it before the statement). - return { ...q, with: withItems, _with_trailing: true }; + return { ...q, with: withItems, with_trailing: true }; } if (q.type === 'SelectWithUnionQuery') { const selects = q.selects.slice(); @@ -1333,7 +1333,7 @@ if (preSet !== undefined) { if (preSet.changes !== undefined) { for (const key of Object.keys(preSet.changes)) { - preEntries.push(key + ' = ' + explainSettingText(preSet.changes[key], preSet._change_value_types && preSet._change_value_types[key])); + preEntries.push(key + ' = ' + explainSettingText(preSet.changes[key], preSet.change_value_types && preSet.change_value_types[key])); } } if (preSet.default_settings !== undefined) { @@ -1694,11 +1694,11 @@ ...(setN.changes || {}), }; const mergedValueTypes = { - ...(selectInner._change_value_types || {}), - ...(setN._change_value_types || {}), + ...(selectInner.change_value_types || {}), + ...(setN.change_value_types || {}), }; if (Object.keys(mergedChanges).length > 0) setN.changes = mergedChanges; - if (Object.keys(mergedValueTypes).length > 0) setN._change_value_types = mergedValueTypes; + if (Object.keys(mergedValueTypes).length > 0) setN.change_value_types = mergedValueTypes; const insertKeys = new Set(Object.keys(setN.changes || {})); const seen = new Set(); const mergedDefaults = []; @@ -1724,8 +1724,8 @@ // SELECT's own settings, so no origin marker is needed. const setN = { type: 'Settings' }; if (selectInner.changes !== undefined) setN.changes = { ...selectInner.changes }; - if (selectInner._change_value_types !== undefined) { - setN._change_value_types = { ...selectInner._change_value_types }; + if (selectInner.change_value_types !== undefined) { + setN.change_value_types = { ...selectInner.change_value_types }; } if (selectInner.default_settings !== undefined) { setN.default_settings = [...selectInner.default_settings]; @@ -1820,7 +1820,7 @@ function cloneAst(v) { if (Array.isArray(v)) { const a = v.map(cloneAst); - // Preserve custom non-index array props (e.g. ORDER BY `_parenthesized`). + // Preserve custom non-index array props (e.g. ORDER BY `parenthesized`). for (const k of Object.keys(v)) if (!/^\d+$/.test(k)) a[k] = cloneAst(v[k]); return a; } @@ -2055,7 +2055,7 @@ // (`ENGINE = MergeTree`) and the empty-parens form (`MergeTree()`), but its // EXPLAIN text and SHOW CREATE distinguish them. Mark the no-parens form so // format()/explain can re-emit it. - if (args === undefined) node._no_parens = true; + if (args === undefined) node.no_parens = true; return withLoc(node, l ?? spanOf(args)); } @@ -2175,7 +2175,7 @@ // EXPLAIN (the native `Set` child order follows the source), so keep that // flag. `PRIMARY KEY` position has no such effect: format() canonicalizes // it to ClickHouse's `PRIMARY KEY` … `ORDER BY` order. - if (stmt.settingsAfterOrderBy === true) node._settings_after_order_by = true; + if (stmt.settingsAfterOrderBy === true) node.settings_after_order_by = true; // Empty when none of the optional fields applied. if ( node.engine === undefined && @@ -2260,7 +2260,7 @@ // formatter. function storageOrderByNode(orderBy) { const hasDesc = orderBy.some((o) => o.dir === 'DESC'); - const parenthesized = !!orderBy._parenthesized; + const parenthesized = !!orderBy.parenthesized; const sobe = (expr, dir) => withLoc({ type: 'StorageOrderByElement', expression: expr, @@ -2780,10 +2780,10 @@ switch (cmd.commandType) { case 'ADD_COLUMN': if (cmd.column) node.column_declaration = columnDeclNode(cmd.column); - if (cmd.afterColumn) node.column = colRefIdent(cmd.afterColumn, cmd._afterColumnParts, cmd.location); + if (cmd.afterColumn) node.column = colRefIdent(cmd.afterColumn, cmd.afterColumnParts, cmd.location); break; case 'DROP_COLUMN': - if (cmd.columnName) node.column = colRefIdent(cmd.columnName, cmd._columnNameParts, cmd.location); + if (cmd.columnName) node.column = colRefIdent(cmd.columnName, cmd.columnNameParts, cmd.location); if (cmd.clear === true) node.clear_column = true; if (cmd.partition) node.partition = alterPartitionNode(cmd.partition); break; @@ -2797,20 +2797,20 @@ node.settings_changes = setNode(op.settings || []); } } - if (cmd.afterColumn) node.column = colRefIdent(cmd.afterColumn, cmd._afterColumnParts, cmd.location); + if (cmd.afterColumn) node.column = colRefIdent(cmd.afterColumn, cmd.afterColumnParts, cmd.location); if (cmd.removeProperty) node.remove_property = cmd.removeProperty; break; } case 'RENAME_COLUMN': - if (cmd.oldName) node.column = colRefIdent(cmd.oldName, cmd._oldNameParts, cmd.location); - if (cmd.newName) node.rename_to = colRefIdent(cmd.newName, cmd._newNameParts, cmd.location); + if (cmd.oldName) node.column = colRefIdent(cmd.oldName, cmd.oldNameParts, cmd.location); + if (cmd.newName) node.rename_to = colRefIdent(cmd.newName, cmd.newNameParts, cmd.location); break; case 'COMMENT_COLUMN': - if (cmd.columnName) node.column = colRefIdent(cmd.columnName, cmd._columnNameParts, cmd.location); + if (cmd.columnName) node.column = colRefIdent(cmd.columnName, cmd.columnNameParts, cmd.location); if (cmd.comment) node.comment = cmd.comment; break; case 'MATERIALIZE_COLUMN': - if (cmd.columnName) node.column = colRefIdent(cmd.columnName, cmd._columnNameParts, cmd.location); + if (cmd.columnName) node.column = colRefIdent(cmd.columnName, cmd.columnNameParts, cmd.location); if (cmd.partition) node.partition = alterPartitionNode(cmd.partition); break; case 'ADD_INDEX': @@ -5332,7 +5332,7 @@ function peg$parse(input, options) { // SETTINGS before and/or after FORMAT collapse onto the query wrapper's // native `settings` field. The pre-/post-FORMAT position is a purely // syntactic no-op, but ClickHouse's EXPLAIN AST child order preserves it, - // so we record a library-only `_settings_before_format` hint (used only + // so we record a library-only `settings_before_format` hint (used only // by formatExplain(); format() canonicalizes to SETTINGS after FORMAT). const settingItems = [ ...(preSettings !== null ? preSettings[1] : []), @@ -5341,7 +5341,7 @@ function peg$parse(input, options) { if (settingItems.length > 0) { if (isNewNode) { result = { ...result, settings: setNode(settingItems) }; - if (preSettings !== null) result = { ...result, _settings_before_format: true }; + if (preSettings !== null) result = { ...result, settings_before_format: true }; } else { result = { ...result, postFormatSettings: settingItems }; } @@ -5388,7 +5388,7 @@ function peg$parse(input, options) { const result = loc({ kind: 'alterCommand', commandType: 'ADD_COLUMN', column: col }); if (after !== null && after[1] && after[4] && typeof after[4] === 'object') { result.afterColumn = after[4].name; - result._afterColumnParts = after[4].parts; + result.afterColumnParts = after[4].parts; } else if (after !== null && after[1] && after[1].toUpperCase() === 'FIRST') { result.first = true; } @@ -5396,7 +5396,7 @@ function peg$parse(input, options) { return result; } function peg$f23(ifExists, col, partition) { - const result = loc({ kind: 'alterCommand', commandType: 'DROP_COLUMN', columnName: col.name, _columnNameParts: col.parts }); + const result = loc({ kind: 'alterCommand', commandType: 'DROP_COLUMN', columnName: col.name, columnNameParts: col.parts }); if (ifExists !== null) result.ifExists = true; if (partition !== null) result.partition = partition[1]; return result; @@ -5406,7 +5406,7 @@ function peg$parse(input, options) { kind: 'alterCommand', commandType: 'DROP_COLUMN', columnName: col.name, - _columnNameParts: col.parts, + columnNameParts: col.parts, clear: true, }); if (ifExists !== null) result.ifExists = true; @@ -5424,7 +5424,7 @@ function peg$parse(input, options) { if (modifySetReset !== null) result.columnSettingOp = modifySetReset[1]; if (after !== null && after[1] && after[4] && typeof after[4] === 'object') { result.afterColumn = after[4].name; - result._afterColumnParts = after[4].parts; + result.afterColumnParts = after[4].parts; } else if (after !== null && after[1] && after[1].toUpperCase() === 'FIRST') { result.first = true; } @@ -5471,8 +5471,8 @@ function peg$parse(input, options) { commandType: 'RENAME_COLUMN', oldName: oldName.name, newName: newName.name, - _oldNameParts: oldName.parts, - _newNameParts: newName.parts, + oldNameParts: oldName.parts, + newNameParts: newName.parts, }); if (ifExists !== null) result.ifExists = true; return result; @@ -5482,7 +5482,7 @@ function peg$parse(input, options) { kind: 'alterCommand', commandType: 'COMMENT_COLUMN', columnName: col.name, - _columnNameParts: col.parts, + columnNameParts: col.parts, comment, }); if (ifExists !== null) result.ifExists = true; @@ -5493,7 +5493,7 @@ function peg$parse(input, options) { kind: 'alterCommand', commandType: 'MATERIALIZE_COLUMN', columnName: col.name, - _columnNameParts: col.parts, + columnNameParts: col.parts, }); if (ifExists !== null) result.ifExists = true; if (partition !== null) result.partition = partition[1]; @@ -6299,7 +6299,7 @@ function peg$parse(input, options) { if (format !== null) node.format = format[1]; // Preserve source ordering: `SETTINGS ... FORMAT name` vs // `FORMAT name SETTINGS ...`. - if (preSettings !== null && format !== null) node._settings_before_format = true; + if (preSettings !== null && format !== null) node.settings_before_format = true; return loc(node); } function peg$f269(query) { return { kind: 'subquery', query }; } @@ -6915,10 +6915,10 @@ function peg$parse(input, options) { if (fromPath !== null) result.attachFromPath = fromPath[4].value; if (format !== null) result.format = format[1]; // Promote column-level PRIMARY KEY to primaryKey if no explicit PRIMARY KEY clause - if (result._columnPrimaryKeys && !result.primaryKey) { - result.primaryKey = result._columnPrimaryKeys; + if (result.columnPrimaryKeys && !result.primaryKey) { + result.primaryKey = result.columnPrimaryKeys; } - delete result._columnPrimaryKeys; + delete result.columnPrimaryKeys; return loc(createTableNode(result)); } function peg$f451(temp, table, cluster) { @@ -7030,8 +7030,8 @@ function peg$parse(input, options) { let primaryKeyInSchema = null; const columnPrimaryKeys = []; for (const el of elements) { - if (el._primaryKey) { - primaryKeyInSchema = el._primaryKey; + if (el.primaryKeyExprs) { + primaryKeyInSchema = el.primaryKeyExprs; } else { tableElements.push(el); if (el.kind === 'columnDef' && el.primaryKey) { @@ -7042,7 +7042,7 @@ function peg$parse(input, options) { const result = {}; if (tableElements.length > 0) result.tableElements = tableElements; if (primaryKeyInSchema !== null) result.primaryKeyInSchema = primaryKeyInSchema; - if (columnPrimaryKeys.length > 0) result._columnPrimaryKeys = columnPrimaryKeys; + if (columnPrimaryKeys.length > 0) result.columnPrimaryKeys = columnPrimaryKeys; return result; } function peg$f479(head, tail, trailing_comma) { @@ -7133,7 +7133,7 @@ function peg$parse(input, options) { } function peg$f506(exprs) { exprs.location = location(); - return { _primaryKey: exprs }; + return { primaryKeyExprs: exprs }; } function peg$f507() { return []; } function peg$f508(head, tail) { @@ -7176,7 +7176,7 @@ function peg$parse(input, options) { function peg$f519(c) { return { comment: c }; } function peg$f520(head, tail) { const items = [head, ...tail.map(t => t[3])]; - items._parenthesized = true; + items.parenthesized = true; return items; } function peg$f521(head, tail) { @@ -7312,13 +7312,13 @@ function peg$parse(input, options) { query.selects.length === 1 && (query.selects[0].type === 'SelectQuery' || query.selects[0].type === 'SelectIntersectExceptQuery') ) { - let result = { ...query.selects[0], _parenthesized: true }; + let result = { ...query.selects[0], parenthesized: true }; if (query.leadingComments !== undefined) result = addLeading(result, query.leadingComments); if (query.trailingComments !== undefined) result = addTrailing(result, query.trailingComments); return result; } if (query.type === 'SelectWithUnionQuery' || query.type === 'SelectIntersectExceptQuery') { - return { ...query, _parenthesized: true }; + return { ...query, parenthesized: true }; } return query; } @@ -7741,13 +7741,13 @@ function peg$parse(input, options) { return (left) => (loc(fn('or', [ loc(opFn('<', [left, low])), loc(opFn('>', [left, high])) - ], { is_operator: true, _parenthesized: true }))); + ], { is_operator: true, parenthesized: true }))); } function peg$f692(low, high) { return (left) => (loc(fn('and', [ loc(opFn('>=', [left, low])), loc(opFn('<=', [left, high])) - ], { is_operator: true, _parenthesized: true }))); + ], { is_operator: true, parenthesized: true }))); } function peg$f693(type) { return (left) => (loc(castOpFn(left, type))); @@ -7816,40 +7816,40 @@ function peg$parse(input, options) { // Don't fold across explicit parentheses: -(1) must produce negate(1), not Int64_-1. // (-1) without an outer minus stays as a UInt64 literal — only `-` immediately before a // bare literal folds. A parenthesized inner expression always wraps in negate(). - if (expr._parenthesized) { + if (expr.parenthesized) { return loc(fn('negate', [expr], { is_operator: true })); } - if (expr.type === 'Literal' && expr.value_type === 'UInt64' && expr._neg_folded === undefined) { + if (expr.type === 'Literal' && expr.value_type === 'UInt64' && expr.neg_folded === undefined) { // Negate a non-negative integer literal: compute decimal value using BigInt for precision. - // A UInt64 carrying `_neg_folded` is itself a folded `-0` — `- -0` stays negate(0). + // A UInt64 carrying `neg_folded` is itself a folded `-0` — `- -0` stays negate(0). const bigNeg = -BigInt(expr.value); // Check if fits in Int64 range [-2^63, 0] const INT64_MIN = BigInt('-9223372036854775808'); if (bigNeg >= INT64_MIN) { - // -0 stays a UInt64 in ClickHouse; mark with `_neg_folded` so the + // -0 stays a UInt64 in ClickHouse; mark with `neg_folded` so the // outer minus of `- -0` doesn't re-fold. The marker is parse-time // only and is stripped before the AST is returned (see // `stripParseTimeMarkers` in src/index.ts). const intResult = loc(bigNeg === BigInt(0) ? uintLit('0') : intLit(String(bigNeg))); - if (bigNeg === BigInt(0)) intResult._neg_folded = true; + if (bigNeg === BigInt(0)) intResult.neg_folded = true; return intResult; } // Overflows Int64: use Float64 (loses precision like ClickHouse does) return loc(floatLit(String(Number(bigNeg)))); } if (expr.type === 'Literal' && expr.value_type === 'Float64' && isNonNegNumericLit(expr)) { - // Negate a positive float literal (works from `value`/`_nonfinite`). + // Negate a positive float literal (works from `value`/`nonfinite`). return loc(negateNumericLit(expr)); } // Identify a `::`-form CAST: either the structured-operand form // (is_operator: true) or the folded pure-literal form (operand is a - // String literal carrying the parse-time `_cast_operand` marker — + // String literal carrying the parse-time `cast_operand` marker — // stripped from the final AST, but visible during parsing). const isOpCast = (e) => { if (e.type !== 'Function' || e.name !== 'CAST') return false; if (e.is_operator === true) return true; const a0 = e.arguments[0]; - return a0 !== undefined && a0.type === 'Literal' && a0._cast_operand !== undefined; + return a0 !== undefined && a0.type === 'Literal' && a0.cast_operand !== undefined; }; if (isOpCast(expr)) { // -value::Type: fold the minus sign into the cast's innermost literal (for :: operator casts) @@ -7860,11 +7860,11 @@ function peg$parse(input, options) { } const inner = innermost.arguments[0]; // The operand may have been stringified (pure-literal :: casts) - if (inner.type === 'Literal' && inner.value_type === 'String' && inner._cast_operand !== undefined) { - const orig = inner._cast_operand; + if (inner.type === 'Literal' && inner.value_type === 'String' && inner.cast_operand !== undefined) { + const orig = inner.cast_operand; if (isNonNegNumericLit(orig)) { const negOrig = negateNumericLit(orig); - const negInner = { ...inner, value: '-' + inner.value, _cast_operand: negOrig }; + const negInner = { ...inner, value: '-' + inner.value, cast_operand: negOrig }; let result = { ...innermost, arguments: [negInner, innermost.arguments[1]] }; const stack = []; let cur = expr; @@ -8130,7 +8130,7 @@ function peg$parse(input, options) { if (rest.length === 0 && trailing === null) { // (expr) — parenthesized expression first = addTrailing(first, flattenWs(afterLast)); - return { ...first, _parenthesized: true }; + return { ...first, parenthesized: true }; } else if (rest.length === 0) { // (expr,) — single-element tuple return loc(fn('tuple', [first], { is_operator: true })); diff --git a/tests/README.md b/tests/README.md index cb88930b9..08e0cc372 100644 --- a/tests/README.md +++ b/tests/README.md @@ -6,10 +6,10 @@ The `clickhouse-reference-*.test.ts` suites run the parser and formatters agains For each input file in `tests/clickhouse-reference`, these tests -1. (`clickhouse-reference-ast.test.ts`) Run `parse()` on the input and compare the output AST to the expected AST — ClickHouse's own `EXPLAIN AST json = 1` output, generated by the `clickhouse` binary. The comparison uses `stripAstMeta` (see `src/meta.ts`), which removes node metadata (location/parent/comments) and underscore-prefixed library-only fields, leaving exactly the ClickHouse-native view. `` and `` entries in the expected file are skipped. +1. (`clickhouse-reference-ast.test.ts`) Run `parse()` on the input and compare the output AST to the expected AST — ClickHouse's own `EXPLAIN AST json = 1` output, generated by the `clickhouse` binary. The comparison uses `formatExplainJson` (see `src/json-explain.ts`), which removes node metadata (location/parent/comments) and the library-only fields absent from the reference AST, leaving exactly the ClickHouse-native view. `` and `` entries in the expected file are skipped. 2. (`clickhouse-reference-explain.test.ts`) Run `formatExplain()` on the input and compare it to the canonical explain output. This ensures that the parser is parsing all of the information that ClickHouse does. 3. (`clickhouse-reference-format.test.ts`) Run `format()` on the AST and compare the output to the expected format. -4. (`clickhouse-reference-round-trip.test.ts`) Run `parse()` on the formatted SQL and compare the output AST to the AST from parsing the original input. The comparison uses `stripVolatile` (only location/parent removed), so comments and underscore-prefixed library-only fields must survive the round trip too. +4. (`clickhouse-reference-round-trip.test.ts`) Run `parse()` on the formatted SQL and compare the output AST to the AST from parsing the original input. The comparison uses `stripVolatile` (only location/parent and the non-semantic explain-only flags removed), so comments and the semantic library-only fields must survive the round trip too. ### Related Scripts diff --git a/tests/clickhouse-reference-ast.test.ts b/tests/clickhouse-reference-ast.test.ts index b7746f978..8399389ae 100644 --- a/tests/clickhouse-reference-ast.test.ts +++ b/tests/clickhouse-reference-ast.test.ts @@ -4,10 +4,10 @@ import { parse } from '../src/index'; import { CLICKHOUSE_DIR, discoverCases, + formatExplainJson, pruneFilteredStorageSettings, pruneLibraryOnlyParamSettings, readReferenceSql, - stripAstMeta, } from './helpers'; const AST_ERROR = ''; @@ -23,30 +23,21 @@ describe('clickhouse reference - ast', () => { const sql = readReferenceSql(filePath); const statements = parse(sql); - const actual = stripAstMeta(statements) as unknown[]; + const actual = formatExplainJson(statements, 2) as unknown[]; // Each entry in the reference JSON is either a sentinel string - // (``, ``) or a `{ version, ast }` - // envelope wrapping the actual AST. Unwrap envelopes so the rest of - // the comparison can treat them like raw AST nodes. - const rawExpected = JSON.parse(fs.readFileSync(astPath, 'utf-8')) as unknown[]; - const expectedAst = rawExpected.map((entry) => { - if (entry && typeof entry === 'object' && !Array.isArray(entry) && 'ast' in entry) { - return (entry as { ast: unknown }).ast; - } - return entry; - }); + // (``, ``) or a `{ version, ast }` envelope. + const expected = JSON.parse(fs.readFileSync(astPath, 'utf-8')) as unknown[]; // Tolerate ClickHouse's per-engine filtering of `Storage > Set` settings // (extra keys in actual only); see pruneFilteredStorageSettings. - pruneFilteredStorageSettings(actual, expectedAst); + pruneFilteredStorageSettings(actual, expected); // Tolerate `param_*` SET entries that this library keeps in changes // but the native AST drops (see pruneLibraryOnlyParamSettings). - pruneLibraryOnlyParamSettings(actual, expectedAst); - for (let i = 0; i < expectedAst.length; i++) { - const expected = expectedAst[i]; - if (expected === AST_ERROR) continue; - if (expected === QUERY_PARAMS) continue; - expect(actual[i]).toEqual(expected); + pruneLibraryOnlyParamSettings(actual, expected); + for (let i = 0; i < expected.length; i++) { + if (expected[i] === AST_ERROR) continue; + if (expected[i] === QUERY_PARAMS) continue; + expect(actual[i]).toEqual(expected[i]); } }); } diff --git a/tests/clickhouse-reference/01292_create_user.sql.expected.formatted.sql b/tests/clickhouse-reference/01292_create_user.sql.expected.formatted.sql index 3fc53840b..992308495 100644 --- a/tests/clickhouse-reference/01292_create_user.sql.expected.formatted.sql +++ b/tests/clickhouse-reference/01292_create_user.sql.expected.formatted.sql @@ -33,21 +33,21 @@ DROP USER u1_01292, u2_01292_renamed, u3_01292; CREATE USER u1_01292 NOT IDENTIFIED; -CREATE USER u2_01292 IDENTIFIED BY 'qwe123'; +CREATE USER u2_01292 IDENTIFIED WITH plaintext_password BY 'qwe123'; CREATE USER u3_01292 IDENTIFIED BY 'qwe123'; -CREATE USER u4_01292 IDENTIFIED BY 'qwe123'; +CREATE USER u4_01292 IDENTIFIED WITH sha256_password BY 'qwe123'; -CREATE USER u5_01292 IDENTIFIED BY '18138372FAD4B94533CD4881F03DC6C69296DD897234E0CEE83F727E2E6B1F63'; +CREATE USER u5_01292 IDENTIFIED WITH sha256_hash BY '18138372FAD4B94533CD4881F03DC6C69296DD897234E0CEE83F727E2E6B1F63'; -CREATE USER u6_01292 IDENTIFIED BY 'qwe123'; +CREATE USER u6_01292 IDENTIFIED WITH double_sha1_password BY 'qwe123'; -CREATE USER u7_01292 IDENTIFIED BY '8DCDD69CE7D121DE8013062AEAEB2A148910D50E'; +CREATE USER u7_01292 IDENTIFIED WITH double_sha1_hash BY '8DCDD69CE7D121DE8013062AEAEB2A148910D50E'; -CREATE USER u8_01292 IDENTIFIED BY 'qwe123'; +CREATE USER u8_01292 IDENTIFIED WITH bcrypt_password BY 'qwe123'; -CREATE USER u9_01292 IDENTIFIED BY '$2a$12$rz5iy2LhuwBezsM88ZzWiemOVUeJ94xHTzwAlLMDhTzwUxOHaY64q'; +CREATE USER u9_01292 IDENTIFIED WITH bcrypt_hash BY '$2a$12$rz5iy2LhuwBezsM88ZzWiemOVUeJ94xHTzwAlLMDhTzwUxOHaY64q'; SHOW CREATE USER u4_01292; @@ -67,7 +67,7 @@ ALTER USER u2_01292 IDENTIFIED BY '123qwe'; ALTER USER u3_01292 IDENTIFIED BY '123qwe'; -ALTER USER u4_01292 IDENTIFIED BY '123qwe'; +ALTER USER u4_01292 IDENTIFIED WITH plaintext_password BY '123qwe'; ALTER USER u5_01292 NOT IDENTIFIED; @@ -217,7 +217,7 @@ SET DEFAULT ROLE NONE TO u6_01292; DROP USER u1_01292, u2_01292, u3_01292, u4_01292, u5_01292, u6_01292; -CREATE USER u1_01292 IDENTIFIED BY 'qwe123' HOST LOCAL SETTINGS readonly=1; +CREATE USER u1_01292 IDENTIFIED WITH plaintext_password BY 'qwe123' HOST LOCAL SETTINGS readonly=1; ALTER USER u1_01292 NOT IDENTIFIED HOST LIKE '%.%.myhost.com' SETTINGS PROFILE default DEFAULT ROLE NONE; @@ -261,13 +261,13 @@ DROP USER u1_01292, u2_01292, u3_01292, u4_01292, `u5_01292@%.host.com`, `u6_012 DROP USER `u7_01292@%.host.com`, `u8_01292@%.otherhost.com`; -CREATE USER u1_01292 IDENTIFIED BY 'qwe123' HOST LOCAL; +CREATE USER u1_01292 IDENTIFIED WITH plaintext_password BY 'qwe123' HOST LOCAL; CREATE USER u2_01292 NOT IDENTIFIED HOST LIKE '%.%.myhost.com' DEFAULT ROLE NONE; CREATE USER u3_01292 IDENTIFIED BY 'qwe123' HOST LOCAL, IP '192.169.1.1', IP '192.168.0.0/16' DEFAULT ROLE r1_01292; -CREATE USER u4_01292 IDENTIFIED BY 'qwe123' HOST ANY DEFAULT ROLE ALL EXCEPT r1_01292; +CREATE USER u4_01292 IDENTIFIED WITH double_sha1_password BY 'qwe123' HOST ANY DEFAULT ROLE ALL EXCEPT r1_01292; SELECT name, @@ -306,7 +306,7 @@ DROP USER u1_01292, u2_01292, u3_01292, u4_01292, u5_01292; DROP ROLE r1_01292, r2_01292; -CREATE USER u1_01292 IDENTIFIED BY 'qwe123', BY 'qwerty10', BY '123qwe', BY 'abc'; +CREATE USER u1_01292 IDENTIFIED WITH plaintext_password BY 'qwe123', WITH kerberos REALM 'qwerty10', WITH bcrypt_password BY '123qwe', WITH ldap SERVER 'abc'; SELECT name, diff --git a/tests/clickhouse-reference/01702_system_query_log.sql.expected.formatted.sql b/tests/clickhouse-reference/01702_system_query_log.sql.expected.formatted.sql index b48e3ee93..c03a82b9e 100644 --- a/tests/clickhouse-reference/01702_system_query_log.sql.expected.formatted.sql +++ b/tests/clickhouse-reference/01702_system_query_log.sql.expected.formatted.sql @@ -44,7 +44,7 @@ SOURCE(clickhouse(DB 'sqllt' TABLE 'table' HOST 'localhost' PORT 9001)) LIFETIME(MIN 0 MAX 0) LAYOUT(FLAT()); -CREATE USER sqllt_user IDENTIFIED BY 'password'; +CREATE USER sqllt_user IDENTIFIED WITH plaintext_password BY 'password'; CREATE ROLE sqllt_role; diff --git a/tests/clickhouse-reference/03172_bcrypt_validation.sql.expected.formatted.sql b/tests/clickhouse-reference/03172_bcrypt_validation.sql.expected.formatted.sql index 428ee122f..7cd689b0b 100644 --- a/tests/clickhouse-reference/03172_bcrypt_validation.sql.expected.formatted.sql +++ b/tests/clickhouse-reference/03172_bcrypt_validation.sql.expected.formatted.sql @@ -1,4 +1,4 @@ -- Tags: no-fasttest DROP USER IF EXISTS `03172_user_invalid_bcrypt_hash`; -CREATE USER `03172_user_invalid_bcrypt_hash` IDENTIFIED BY '012345678901234567890123456789012345678901234567890123456789'; -- { serverError BAD_ARGUMENTS } \ No newline at end of file +CREATE USER `03172_user_invalid_bcrypt_hash` IDENTIFIED WITH bcrypt_hash BY '012345678901234567890123456789012345678901234567890123456789'; -- { serverError BAD_ARGUMENTS } \ No newline at end of file diff --git a/tests/clickhouse-reference/03174_multiple_authentication_methods_show_create.sql.expected.formatted.sql b/tests/clickhouse-reference/03174_multiple_authentication_methods_show_create.sql.expected.formatted.sql index b71bc4c73..f25dd70eb 100644 --- a/tests/clickhouse-reference/03174_multiple_authentication_methods_show_create.sql.expected.formatted.sql +++ b/tests/clickhouse-reference/03174_multiple_authentication_methods_show_create.sql.expected.formatted.sql @@ -1,10 +1,10 @@ -- Tags: no-fasttest, no-parallel -- Create user with mix both implicit and explicit auth type, starting with with -CREATE USER u_03174_multiple_auth_show_create IDENTIFIED BY '1', BY '2', BY '3', BY '4'; +CREATE USER u_03174_multiple_auth_show_create IDENTIFIED WITH plaintext_password BY '1', BY '2', WITH bcrypt_password BY '3', BY '4'; SHOW CREATE USER u_03174_multiple_auth_show_create; DROP USER IF EXISTS u_03174_multiple_auth_show_create; -- Create user with mix both implicit and explicit auth type, starting with by -CREATE USER u_03174_multiple_auth_show_create IDENTIFIED BY '1', BY '2', BY '3', BY '4'; \ No newline at end of file +CREATE USER u_03174_multiple_auth_show_create IDENTIFIED BY '1', WITH plaintext_password BY '2', WITH bcrypt_password BY '3', BY '4'; \ No newline at end of file diff --git a/tests/clickhouse-reference/03773_create_user_param_auth_methods.sql.expected.formatted.sql b/tests/clickhouse-reference/03773_create_user_param_auth_methods.sql.expected.formatted.sql index 82b465b87..5efb325c2 100644 --- a/tests/clickhouse-reference/03773_create_user_param_auth_methods.sql.expected.formatted.sql +++ b/tests/clickhouse-reference/03773_create_user_param_auth_methods.sql.expected.formatted.sql @@ -6,4 +6,4 @@ DROP USER IF EXISTS user_param_auth_03773; SET param_password = 'test_password_03773'; -- Before fix: This would fail with UNKNOWN_QUERY_PARAMETER on remote nodes -CREATE USER user_param_auth_03773 IDENTIFIED BY 'placeholder' ON CLUSTER test_shard_localhost; \ No newline at end of file +CREATE USER user_param_auth_03773 IDENTIFIED WITH plaintext_password BY 'placeholder' ON CLUSTER test_shard_localhost; \ No newline at end of file diff --git a/tests/helpers.ts b/tests/helpers.ts index 1867db8e3..526af6fa5 100644 --- a/tests/helpers.ts +++ b/tests/helpers.ts @@ -2,7 +2,8 @@ import * as fs from 'fs'; import * as path from 'path'; import { substituteQueryParameters } from '../src/query-parameters'; -export { stripAstMeta, stripVolatile } from '../src/meta'; +export { stripVolatile } from '../src/meta'; +export { formatExplainJson } from '../src/json-explain'; /** Directory holding the ClickHouse reference `.sql` cases and their expected outputs. */ export const CLICKHOUSE_DIR = path.join(__dirname, 'clickhouse-reference'); diff --git a/tests/truncate-target.test.ts b/tests/truncate-target.test.ts index 84db49e1a..6ebb788f3 100644 --- a/tests/truncate-target.test.ts +++ b/tests/truncate-target.test.ts @@ -1,5 +1,5 @@ import { parse } from '../src/index'; -import { stripAstMeta } from './helpers'; +import { formatExplainJson } from './helpers'; /** * Standalone coverage for the `TRUNCATE` target disambiguation, focused on the @@ -19,8 +19,8 @@ import { stripAstMeta } from './helpers'; * ./clickhouse -q "explain ast json=1 " * * (see scripts/gen output; the EXPLAIN string is unescaped and its `.ast` - * extracted). Comparisons use stripAstMeta so library-only/underscore fields - * are ignored — only ClickHouse-native fields are asserted. + * extracted). Comparisons use formatExplainJson so library-only fields are + * ignored — only ClickHouse-native fields are asserted. */ const cases: { sql: string; ast: unknown }[] = [ @@ -119,9 +119,9 @@ describe('TRUNCATE target disambiguation (has_all / has_tables)', () => { for (const { sql, ast } of cases) { it(sql, () => { const statements = parse(sql); - const actual = stripAstMeta(statements) as unknown[]; + const actual = formatExplainJson(statements, 2) as unknown[]; expect(actual).toHaveLength(1); - expect(actual[0]).toEqual(ast); + expect(actual[0]).toEqual({ version: 2, ast }); }); } });