fix(docs): repair Doxygen SQL extraction — coverage gaps + private split - #369
Merged
Conversation
The Markdown/JSON API extraction runs Doxygen's C++ parser over raw SQL, which silently dropped functions and misclassified visibility: - `::type` casts in `LANGUAGE sql` bodies read as C++ `::` scope resolution and dropped the whole enclosing CREATE FUNCTION memberdef — losing jsonb_path_query / jsonb_path_query_first and ~hundreds of generated eql_v3_internal extractor overloads. - Regular `-- …` SQL comments parsed as code, minting phantom functions (overloads, extractor, queries, accessor, inlinable). - `is_private` keyed on a leading `_` in the function NAME, but internal functions live in the `eql_v3_internal` schema, so none were flagged (984 functions, all reported public). - `@example` in version.sql detached its doc comment (dropping `version` entirely); and `*.template` sources were documented alongside their generated `.sql`, duplicating `version` with $RELEASE_VERSION placeholder text. Fixes: - doxygen-filter.sh: strip `$$…$$` bodies and neutralize `--` comments so Doxygen sees only clean declarations + `//!` doc comments. - xml-to-markdown.py: treat the `eql_v3_internal` schema as private. - Doxyfile: document only `*.sql` (drop `*.template`; version.sql is build-generated from the template before Doxygen runs). - version.template: replace `@example` with plain prose. Result: manifest 984 → 1680 functions (986 public, 694 private, 0 false-positive private); version + jsonb_path_query(_first) now present; phantom functions gone. Domains unchanged (catalog-sourced). All doc-gen tests + SQL coverage validation pass. Claude-Session: https://claude.ai/code/session_01CqDNqLSEEkCi7xAJFq7HJA
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
coderdan
added a commit
to cipherstash/docs
that referenced
this pull request
Jul 6, 2026
Wires the v2 EQL reference to the corrected manifest (public. domains, eql_v3./eql_v3_internal. functions — cipherstash/encrypt-query-language#369) and turns the drift-lint into a real gate. - generate-eql-api-docs.ts: schema-aware drift-lint — matches public.<domain>, eql_v3.<fn>, eql_v3_internal.<fn> by schema AND name, so right-name/wrong-schema refs are caught, not just fabricated ones. Group public overloads by name (the raw 1680-function surface -> 39 functions); omit internal functions from the page (kept in the lint's known set). Manifest source: EQL_MANIFEST_PATH -> release cache -> sample, auto-strict against any real manifest. - generate-eql-docs.ts: extract json/eql-manifest.json from the release tarball to a gitignored cache the reference generator reads (best-effort; falls back to the sample until a release ships the manifest). - Swept content/docs/reference/eql/*.mdx (165 refs): domains -> public., internal ctors -> eql_v3_internal., PG aliases -> canonical catalog names (int4->integer, float8->double, bool->boolean, ste_vec_*->jsonb_*). - Updated the illustrative sample fixture to the corrected schema. Drift-lint verified green against a locally-generated corrected manifest; report-only against the committed sample until a release ships the manifest. Claude-Session: https://claude.ai/code/session_01CqDNqLSEEkCi7xAJFq7HJA
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes the SQL→Doxygen→XML→Markdown/JSON extraction pipeline so the generated API docs and eql-manifest.json reflect the real SQL function surface, including correct public vs internal visibility.
Changes:
- Update XML extraction to classify functions in the
eql_v3_internalschema as private (while keeping the legacy leading-underscore check). - Replace the Doxygen input filter with an
awk-based preprocessor that strips$$...$$bodies and neutralizes--comments to prevent Doxygen’s C++ parser mis-parsing SQL. - Limit Doxygen inputs to
*.sql(exclude*.template) and adjustversion.templatedocs to avoid@exampledetachment/duplication effects.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| tasks/docs/generate/xml-to-markdown.py | Adjusts visibility detection to treat eql_v3_internal schema functions as private. |
| tasks/docs/doxygen-filter.sh | Reworks the Doxygen filter to strip dollar-quoted bodies and neutralize SQL line comments for safer parsing. |
| src/v3/version.template | Updates version function docs to avoid @example-related extraction issues. |
| Doxyfile | Restricts Doxygen file patterns to *.sql to avoid documenting templates alongside generated SQL. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Address Copilot review on #369: - test_xml_to_markdown.py: add test_internal_schema_is_private, exercising process_function directly — asserts eql_v3_internal.* -> private and eql_v3.* -> public, so the schema-based visibility can't silently regress to the old leading-underscore heuristic. - doxygen-filter.sh: set -euo pipefail + explicit one-arg check, so a stray invocation fails fast (exit 2) instead of awk blocking on stdin. Claude-Session: https://claude.ai/code/session_01CqDNqLSEEkCi7xAJFq7HJA
coderdan
added a commit
to cipherstash/docs
that referenced
this pull request
Jul 6, 2026
Wires the v2 EQL reference to the corrected manifest (public. domains, eql_v3./eql_v3_internal. functions — cipherstash/encrypt-query-language#369) and turns the drift-lint into a real gate. - generate-eql-api-docs.ts: schema-aware drift-lint — matches public.<domain>, eql_v3.<fn>, eql_v3_internal.<fn> by schema AND name, so right-name/wrong-schema refs are caught, not just fabricated ones. Group public overloads by name (the raw 1680-function surface -> 39 functions); omit internal functions from the page (kept in the lint's known set). Manifest source: EQL_MANIFEST_PATH -> release cache -> sample, auto-strict against any real manifest. - generate-eql-docs.ts: extract json/eql-manifest.json from the release tarball to a gitignored cache the reference generator reads (best-effort; falls back to the sample until a release ships the manifest). - Swept content/docs/reference/eql/*.mdx (165 refs): domains -> public., internal ctors -> eql_v3_internal., PG aliases -> canonical catalog names (int4->integer, float8->double, bool->boolean, ste_vec_*->jsonb_*). - Updated the illustrative sample fixture to the corrected schema. Drift-lint verified green against a locally-generated corrected manifest; report-only against the committed sample until a release ships the manifest. Claude-Session: https://claude.ai/code/session_01CqDNqLSEEkCi7xAJFq7HJA
coderdan
added a commit
to cipherstash/docs
that referenced
this pull request
Jul 8, 2026
Wires the v2 EQL reference to the corrected manifest (public. domains, eql_v3./eql_v3_internal. functions — cipherstash/encrypt-query-language#369) and turns the drift-lint into a real gate. - generate-eql-api-docs.ts: schema-aware drift-lint — matches public.<domain>, eql_v3.<fn>, eql_v3_internal.<fn> by schema AND name, so right-name/wrong-schema refs are caught, not just fabricated ones. Group public overloads by name (the raw 1680-function surface -> 39 functions); omit internal functions from the page (kept in the lint's known set). Manifest source: EQL_MANIFEST_PATH -> release cache -> sample, auto-strict against any real manifest. - generate-eql-docs.ts: extract json/eql-manifest.json from the release tarball to a gitignored cache the reference generator reads (best-effort; falls back to the sample until a release ships the manifest). - Swept content/docs/reference/eql/*.mdx (165 refs): domains -> public., internal ctors -> eql_v3_internal., PG aliases -> canonical catalog names (int4->integer, float8->double, bool->boolean, ste_vec_*->jsonb_*). - Updated the illustrative sample fixture to the corrected schema. Drift-lint verified green against a locally-generated corrected manifest; report-only against the committed sample until a release ships the manifest. Claude-Session: https://claude.ai/code/session_01CqDNqLSEEkCi7xAJFq7HJA
coderdan
added a commit
to cipherstash/docs
that referenced
this pull request
Jul 8, 2026
Wires the v2 EQL reference to the corrected manifest (public. domains, eql_v3./eql_v3_internal. functions — cipherstash/encrypt-query-language#369) and turns the drift-lint into a real gate. - generate-eql-api-docs.ts: schema-aware drift-lint — matches public.<domain>, eql_v3.<fn>, eql_v3_internal.<fn> by schema AND name, so right-name/wrong-schema refs are caught, not just fabricated ones. Group public overloads by name (the raw 1680-function surface -> 39 functions); omit internal functions from the page (kept in the lint's known set). Manifest source: EQL_MANIFEST_PATH -> release cache -> sample, auto-strict against any real manifest. - generate-eql-docs.ts: extract json/eql-manifest.json from the release tarball to a gitignored cache the reference generator reads (best-effort; falls back to the sample until a release ships the manifest). - Swept content/docs/reference/eql/*.mdx (165 refs): domains -> public., internal ctors -> eql_v3_internal., PG aliases -> canonical catalog names (int4->integer, float8->double, bool->boolean, ste_vec_*->jsonb_*). - Updated the illustrative sample fixture to the corrected schema. Drift-lint verified green against a locally-generated corrected manifest; report-only against the committed sample until a release ships the manifest. Claude-Session: https://claude.ai/code/session_01CqDNqLSEEkCi7xAJFq7HJA
coderdan
added a commit
to cipherstash/docs
that referenced
this pull request
Jul 8, 2026
Wires the v2 EQL reference to the corrected manifest (public. domains, eql_v3./eql_v3_internal. functions — cipherstash/encrypt-query-language#369) and turns the drift-lint into a real gate. - generate-eql-api-docs.ts: schema-aware drift-lint — matches public.<domain>, eql_v3.<fn>, eql_v3_internal.<fn> by schema AND name, so right-name/wrong-schema refs are caught, not just fabricated ones. Group public overloads by name (the raw 1680-function surface -> 39 functions); omit internal functions from the page (kept in the lint's known set). Manifest source: EQL_MANIFEST_PATH -> release cache -> sample, auto-strict against any real manifest. - generate-eql-docs.ts: extract json/eql-manifest.json from the release tarball to a gitignored cache the reference generator reads (best-effort; falls back to the sample until a release ships the manifest). - Swept content/docs/reference/eql/*.mdx (165 refs): domains -> public., internal ctors -> eql_v3_internal., PG aliases -> canonical catalog names (int4->integer, float8->double, bool->boolean, ste_vec_*->jsonb_*). - Updated the illustrative sample fixture to the corrected schema. Drift-lint verified green against a locally-generated corrected manifest; report-only against the committed sample until a release ships the manifest. Claude-Session: https://claude.ai/code/session_01CqDNqLSEEkCi7xAJFq7HJA
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Discovered while wiring the docs-repo consumer of
eql-manifest.json: the manifest was missing documented functions and reporting every function as public.Root causes
The Markdown/JSON extraction pipeline runs Doxygen's C++ parser over raw SQL, which mis-parses several constructs:
jsonb_path_query,jsonb_path_query_first+ ~hundreds ofeql_v3_internalextractor overloads silently missing::typecasts inLANGUAGE sqlbodies read as C++::scope-resolution, dropping the whole enclosingCREATE FUNCTIONmemberdefoverloads,extractor,queries,accessor,inlinable-- …SQL comments parsed as C++ codeis_privatechecked for a leading_in the function name; internal functions live in theeql_v3_internalschemaversionmissing; duplicateversionw/$RELEASE_VERSIONtext@exampledetached its doc comment;*.templatesources documented alongside their generated.sqlFixes
doxygen-filter.sh— strip$$…$$bodies and neutralize--comments, so Doxygen sees only clean declarations +//!doc comments (no body tokens or comment prose to mis-parse).xml-to-markdown.py— treat theeql_v3_internalschema as private (keeps the legacy leading-underscore check too).Doxyfile— document only*.sql.version.sqlis build-generated fromversion.template(viadocs:generate→depends=["build"]) before Doxygen runs, so nothing is lost.version.template— replace@examplewith plain prose.Result
eql_v3_internal.*in source).version,jsonb_path_query,jsonb_path_query_firstnow present; phantom functions gone.test_xml_to_json,test_xml_to_markdown) +docs:validate(159 SQL files, 0 errors) pass.Review notes
API.mdnow includes a large Internal functions section (the 694eql_v3_internaloverloads that were previously dropped). This is the intended public/internal split; flag if you'd rather keep internals out of the released Markdown.->and a bare>with empty args — from the pre-existing operator-name remapping inprocess_function. Out of scope here; can follow up.Enables the docs-repo manifest consumer (cipherstash/docs#46) to run a green drift-lint against the real function surface.
https://claude.ai/code/session_01CqDNqLSEEkCi7xAJFq7HJA