External plugin host support gaps found while building a DM / Dameng driver

[haos666]

Context

Hi! We have been building a third-party DM / Dameng database driver as a Tabularis external plugin: https://github.com/haos666/tabularis-dameng-plugin

The plugin has reached a plugin-side v1.0.0 baseline: https://github.com/haos666/tabularis-dameng-plugin/releases/tag/v1.0.0

It is implemented as a standalone Java process using Tabularis' JSON-RPC-over-stdin/stdout plugin contract. The Dameng JDBC driver is provided locally by the user and is not redistributed by the plugin.

The plugin side currently implements and validates:

• connect / ping / schemas / tables / columns
• indexes / foreign keys / schema snapshot / batch metadata
• views and view definitions
• routines and routine parameters
• triggers and trigger definitions, plus plugin-side create/drop trigger RPCs
• explain plans
• SQL execution for SELECT, DML, and DDL
• row editing CRUD
• DDL preview helpers
• execute_query_batch
• BLOB/CLOB handling, including a Tabularis-compatible BLOB wire format

We have validated it with:

• a local Docker DM instance
• direct JSON-RPC protocol smoke tests
• the Tabularis UI
• the Tabularis MCP connection path

While doing this, we found a few host-side gaps where the external plugin contract or host forwarding does not yet expose capabilities that the plugin can already provide. We would like to help with PRs, but wanted to open an issue first to align on scope and sequencing.

Gaps found

1. Manifest schema and plugin guide are slightly behind runtime capabilities

The TypeScript runtime model and UI already use capabilities.triggers, but plugins/manifest.schema.json does not declare it yet.

Also, plugins/PLUGIN_GUIDE.md still shows older column metadata field names in at least one example:

• column_default
• is_primary_key

The current runtime model appears to consume:

• default_value
• is_pk

The guide also does not currently document the trigger RPCs used by the UI.

2. SQL dump/import does not support external plugin drivers

When using the SQL/database import flow with an external plugin connection, Tabularis currently returns Unsupported driver. In src-tauri/src/dump_commands.rs, import/export paths appear to dispatch only built-in drivers such as MySQL, PostgreSQL, and SQLite.

For the DM plugin, this means a valid schema script can be executed through database-native tooling or plugin SQL execution paths, but not through Tabularis' import UI.

3. External plugin batch execution is implemented plugin-side but not fully host-forwarded

The DM plugin implements execute_query_batch and verifies it directly through JSON-RPC. This is useful for multi-statement scripts because the plugin can execute all statements on the same JDBC connection and return per-statement results.

The host external plugin driver would need to forward batch execution to the plugin when available, with a graceful fallback for plugins that do not implement it.

4. Oracle-like SQL block splitting

DM trigger/function/procedure DDL commonly uses Oracle-like blocks such as:

CREATE OR REPLACE TRIGGER ...
BEGIN
  ...
END;
/

If the host SQL splitter splits inside these blocks, the plugin receives incomplete statements. This affects DM and would likely also affect Oracle-like external plugins.

5. BLOB preview/export hooks for external plugins

Tabularis has host-side BLOB preview/export hooks for built-in drivers. For external plugins, forwarding equivalent RPCs such as save_blob_to_file and fetch_blob_as_data_url would allow full UI parity.

Suggested PR sequencing

To keep reviews manageable, we are thinking of a small number of focused PRs rather than one large DM-specific change:

PR 1: External plugin contract/docs alignment

• Add capabilities.triggers to plugins/manifest.schema.json
• Update PLUGIN_GUIDE.md column metadata examples to current runtime field names
• Document trigger RPCs used by the UI

This is documentation/schema alignment only and should not change runtime behavior.

PR 2: External plugin execution/import support

• Forward execute_query_batch to external plugins when implemented
• Use external plugin execution for SQL import instead of returning Unsupported driver
• Improve Oracle-like block splitting for CREATE FUNCTION, CREATE PROCEDURE, and CREATE TRIGGER blocks

This would let external plugins run real initialization scripts through Tabularis.

PR 3: External plugin BLOB UI hooks

• Forward BLOB preview/export RPCs to external plugins
• Preserve existing built-in driver behavior

Goal

We are not asking Tabularis to add a built-in Dameng driver. The DM driver is intentionally maintained as an external plugin. The goal is to close a few generic host-side gaps so external plugins that already implement richer metadata, execution, triggers, batch, and BLOB behavior can expose those capabilities through the Tabularis UI.

Would this sequencing be acceptable? If preferred, we can start with the contract/docs alignment PR first and keep the runtime forwarding changes separate.

[debba]

Hi @haos666 — thanks for the unusually thorough writeup, and for building this as a proper external plugin rather than asking us to vendor a DM driver. I went through each gap against the current main and can confirm all five. The proposed sequencing works well for us. A few notes per point:

1. Manifest schema + guide alignment — Confirmed.

• capabilities.triggers is consumed by the TS runtime (src/types/plugins.ts, used in ExplorerSidebar, SidebarSchemaItem, SidebarDatabaseItem) but the schema doesn't declare it. Note capabilities has additionalProperties: false, so this isn't just missing documentation — a manifest setting triggers would currently fail validation. Adding it is required for plugin parity.

• PLUGIN_GUIDE.md uses column_default / is_primary_key; the runtime model uses is_pk and default_value. Worth fixing.

  One thing to flag for scoping: PR1 as described (schema + docs) won't actually make plugin triggers work on its own. The host's external plugin driver doesn't forward the trigger RPCs yet — get_triggers / get_trigger_definition / create_trigger / drop_trigger fall through to the trait default that returns "Triggers not supported by this driver", and the frontend swallows that into an empty list. So declaring the capability is correct for PR1, but the actual get_triggers/create_trigger/drop_trigger forwarding belongs in the runtime PR (PR2) alongside batch/import. Happy to take it either way — just want the capability flag and the forwarding to land together so the UI section isn't shown-but-empty.

2. SQL import/export for external plugins — Confirmed. dump_commands.rs hardcodes mysql/postgres/sqlite and returns Unsupported driver for everything else, on both the export and import paths. Routing these through the plugin's SQL execution (and execute_query_batch where available) is the right fix. One thing to keep in mind: the built-in import paths lean on driver-specific session tuning (FK checks off, autocommit, PRAGMAs). For plugins we'd want a plain transactional/best-effort path without assuming those exist.

3. execute_query_batch host forwarding — Confirmed. The plugin driver doesn't override execute_batch, so it uses the trait default that loops one statement per execute_query RPC — no shared connection/transaction. Forwarding to the plugin's execute_query_batch when implemented, with a graceful fallback to the per-statement loop for plugins that don't, is exactly the right shape and matches how other optional capabilities degrade.

4. Oracle-like block splitting — Confirmed, and partly scaffolded already. The splitter has an oracle dialect and a plugin's sql_dialect manifest value already flows into it via normalizeDialect. The catch is the ORACLE preset is currently identical to GENERIC — no / terminator or BEGIN…END block awareness (the code comment calls this out explicitly). The existing dollarQuoting (Postgres) and goDelimiter (MSSQL) handling are good templates for adding a slash-terminator / PL/SQL block mode. Please gate it behind the dialect flag so MySQL/Postgres splitting is untouched.

cc @ymadd, who did the splitter rewrite — he'll have the best read on where the slash-terminator / PL/SQL block mode should hook in and how to keep the existing dialects untouched.

5. BLOB preview/export hooks — Confirmed. save_blob_to_file / fetch_blob_as_data_url are trait methods the plugin driver doesn't override, so they hit the default "not supported" error. Forwarding both as RPCs (with the default error preserved for plugins that don't implement them) gives full parity.

On sequencing: yes, let's do PR1 (schema + docs) first since it's low-risk and unblocks manifest validation. For PR2 please keep it reviewable — splitting "batch forwarding", "import support", and "Oracle block splitting" into separate commits (or even separate PRs) would help us review each independently, since they touch different subsystems. PR3 (BLOB) stands alone fine.

A few house rules so review goes smoothly:

• Every new util/helper needs unit tests in the same change (splitter changes especially — please add cases for BEGIN…END blocks, nested blocks, and the / terminator).
• New i18n keys go in all 8 locales.
• Keep built-in driver behavior byte-for-byte unchanged; all new paths should be additive and capability-gated.

Once these land — want to list it in our registry? We're in the middle of rolling out a hosted plugin registry (Tabularium, registry.tabularis.dev) that surfaces drivers directly in the New Connection catalogue — see #299. It's a hybrid model: the registry indexes plugins, GitHub keeps hosting the releases, so your DM plugin stays where it is and just becomes discoverable + installable in-app (with tracked downloads and tabularis://install deep links). Once the host-side gaps above are closed and DM has parity, we'd be happy to have it listed — no built-in driver required, exactly as you intended. cc @NewtTheWolf, who's driving the registry work — he can point you at the submission flow once it's ready.

Thanks again — this is a model example of how to propose host-side plugin contract work. Looking forward to the PRs.

[NewtTheWolf]

Hey @haos666, love the work!

Regarding the new Registry: It’s still a work in progress, but already quite stable. You can actually go ahead and submit your plugin at https://registry.tabularis.dev/.

Just a heads-up that there are some breaking changes in the Core Schema. As a workaround, you can simply keep your existing manifest.json for the old definition and add a .tabularium file specifically for the registry.

You can check out how I set it up here for reference: https://codeberg.org/NewtTheWolf/firestore-tabularis/src/branch/main/.tabularium

[haos666]

Thanks @debba and @NewtTheWolf, this is very helpful.

The sequencing and scoping notes make sense to me. I will start with the low-risk contract/docs alignment PR first: capabilities.triggers in the manifest schema, the current column metadata field names in PLUGIN_GUIDE.md, and trigger RPC documentation. I will keep that PR runtime-free and reference this issue.

For the runtime work, I will keep the changes separated and additive as suggested. In particular, trigger RPC forwarding and execute_query_batch forwarding can be reviewed independently from SQL import/export and Oracle-like block splitting, and I will add tests with each helper/change. Built-in driver behavior will be left unchanged.

For the registry, thanks for the pointer. I will keep the existing plugin manifest.json as-is for current Tabularis installs and add a separate .tabularium file in the DM plugin repo before submitting it to the registry.

[ymadd]

Thanks for the cc @debba — I had a close look at how the slash-terminator / PL/SQL block mode should land in the splitter. Good news first: the plumbing is already in place. A plugin's sql_dialect manifest value already flows into splitStatements → normalizeDialect, and the ORACLE preset exists as a separate constant precisely so this divergence stays a one-line edit. @haos666 — note that normalizeDialect silently falls back to generic for unknown values, so the DM plugin should declare sql_dialect: "oracle". That's worth a line in PLUGIN_GUIDE.md, and it's runtime-free, so it can ride along in PR1.

Recommended semantics: SQL*Plus-style termination rules

I'd model the splitting on how SQL*Plus decides where a unit ends, without trying to parse PL/SQL:

1. A line-leading lone / (only whitespace around it on the line) is a statement terminator for the whole dialect — not just after blocks. Oracle-style dumps routinely terminate plain SQL with / too.
2. When a statement starts with a PL/SQL-mode opener (list below), ; stops terminating for that statement: we wait for the next lone /. No BEGIN/END depth tracking — lexically matching END vs END IF; / END LOOP; / END CASE; is the classic bug farm, and SQL*Plus itself doesn't do it either: everything until / is one buffer.
3. EOF with no / → the whole remainder is one statement. This matters for the editor: selecting a complete CREATE PROCEDURE … END; (no trailing / — valid text to send through a driver API) and hitting run must yield one statement, not fragments split at the internal semicolons.

Deliberately not replicated from SQLPlus (these are termination rules, not full buffer semantics): / re-executing the previous buffer, blank-line/. buffer handling, END; / on one line, and client directives (PROMPT, REM, SET, WHENEVER). One accepted incompatibility, pinned by a test: a lone / line that is actually a continuation of a division expression (SELECT 10\n/\n2) will split — SQLPlus chokes on that form too, and no formatter emits it.

Where it hooks in

Not in the tokenizer's ; path. The splitter is currently two passes (collect raw segments by delimiter → fold comment-only segments); I'd add a third pass between them, gated so it doesn't even run for other dialects:

• Tokenizer: a slashDelimiter token modeled directly on readGoDelimiter (same line-leading gate and regex shape — /* can't collide because the regex requires whitespace/EOL after the slash), plus q-quote scanning (below). Everything behind new DialectOptions flags (slashTerminator, plsqlBlocks, qQuoting) enabled only in the ORACLE preset — so the byte-for-byte guarantee for existing dialects is structural, not by-inspection.
• collectSegments: each raw segment records what terminated it (; / / / EOF).
• New foldBlocks pass: if a segment's leading keyword (comment-tolerant — CREATE /* x */ OR REPLACE is legal SQL) is a block opener, merge segments until the first slash-terminated one, meaningful or not: in END;\n/ the slash actually terminates the whitespace-only segment after END;, which is exactly the kind of bookkeeping that needs a dedicated test. The merged statement re-slices the source as one contiguous span, so internal ; and comments inside the body reach the server byte-identical.

Opener list

The SQL*Plus PL/SQL-mode list, kept as a const array so dialect-specific additions are one line: DECLARE, BEGIN (optional <<label>> prefixes skipped), CREATE [OR REPLACE] [EDITIONABLE|NONEDITIONABLE] + FUNCTION | PROCEDURE | TRIGGER | PACKAGE [BODY] | TYPE [BODY] | LIBRARY | [AND COMPILE|AND RESOLVE] JAVA SOURCE, and WITH FUNCTION/PROCEDURE (12c+ inline declarations). Two traps worth calling out explicitly:

• Plain CREATE TYPE specs (no BODY) are PL/SQL source units — SQL*Plus waits for / on them too, so they stay in the list.
• FUNCTION is not a reserved word, so WITH function AS (SELECT …) is a legal CTE. Disambiguate on the third token: identifier → inline-function opener (wait for /); AS → ordinary CTE, normal ; splitting. A ( also means CTE — the column-list form WITH function (col) AS (…) is legal too, since an inline function's name can't start with a paren.

@haos666: two questions — does DM have block-opening forms outside this list (e.g. CREATE CLASS, or a Java-source equivalent)? And do DM's dump tools emit DISQL client directives (PROMPT-style lines) in scripts? If so, a follow-up flag treating those lines as comments would be cheap; v1 would not handle them.

Q-quoting belongs in the same change

Without it the block work has a hole: a body containing q'[it's; tricky]' (or the national variant nq'[…]') mis-splits. Scanner modeled on scanDollarQuoted, with two details: an identifier-boundary guard like the existing E-string check (so colq'x' doesn't false-positive), and delimiter identification by code point while keeping Token.length / StatementRange in UTF-16 code units per the existing contract (an astral-plane delimiter must not land offsets inside a surrogate pair).

Tests

All in tests/utils/sqlSplitter/: the DM trigger example from this issue verbatim; multiple /-separated units; nested BEGIN…END with END IF/LOOP; PACKAGE BODY with several procedures; ; and / inside regular, q-quoted and nq-quoted strings; mixed ;// termination; division continuation lines (/2) not splitting; line-leading /* not a terminator; comment-between-keywords openers; labeled blocks; WITH FUNCTION vs a CTE named function (both the AS and column-list forms); missing trailing / at EOF; END;\n/ terminator ownership; statement ranges of merged blocks (editor integration); and the existing five dialect suites as unchanged canaries.

Side note on scope: this host-side work is what the editor needs (run-statement-at-cursor is synchronous TS). For the SQL import path specifically, once execute_query_batch forwarding lands, the host could additionally hand the raw script to the plugin and let it split natively — the two changes compose rather than overlap, which also supports the PR2 split you suggested.

Happy to take the splitter piece as its own PR, or to review if @haos666 prefers to drive it — either way works for me.

[haos666]

Thanks @ymadd, this is extremely useful.

I’ll add a short sql_dialect: "oracle" note to the guide in PR #317, since that is runtime-free and fits the contract/docs scope.

For the two DM-specific questions, I’ll verify against the local DM instance and available DM docs before answering so I don’t guess:

• whether DM has block-opening forms outside the SQL*Plus-style list
• whether DISQL dump output includes client directives such as PROMPT / SET / SPOOL

I’ll report back here before starting the splitter PR.

[haos666]

I rechecked the two DM-specific questions against the official DM docs and a local DM8 Docker instance.

Block openers

Yes, DM has additional block-opening forms outside the SQL*Plus-style list above:

• CREATE [OR REPLACE] CLASS ... END
• CREATE [OR REPLACE] CLASS BODY ... END
• CREATE [OR REPLACE] JAVA CLASS ... { ... }

References:

• DM8 class type docs: https://eco.dameng.com/document/dm/zh-cn/pm/class-type.html

I also verified these locally with disql against DM8. The following all execute successfully when terminated by a line-leading /:

• CREATE CLASS CLS_SPLITTER_VFY ... END; /
• CREATE CLASS BODY CLS_SPLITTER_VFY ... END; /
• CREATE JAVA CLASS JCLS_SPLITTER_VFY { ... } /

One related nuance: DM also exposes CREATE TYPE ... AS OBJECT / CREATE TYPE BODY objects as CLASS / CLASS BODY in USER_OBJECTS / USER_SOURCE, but their source text remains CREATE TYPE / CREATE TYPE BODY. So the splitter should keep the existing TYPE [BODY] opener handling and add the DM-specific CLASS, CLASS BODY, and JAVA CLASS forms.

DISQL client directives / dumps

DISQL itself does support client commands such as SET, SPOOL, WHENEVER, START / @, and PROMPT-style output commands.

References:

• DISQL common commands: https://eco.dameng.com/document/dm/zh-cn/pm/common-commands.html
• DISQL scripts: https://eco.dameng.com/document/dm/zh-cn/pm/disql-script.html

For DM's standard dexp / dimp flow, the official docs describe .dmp output files plus logs rather than SQL text dumps, and my local dexp test produced only dev_schema.dmp and dev_schema.log. I did not find generated SQL scripts containing PROMPT / SET / SPOOL lines in that path.

Reference:

• dexp logical export: https://eco.dameng.com/document/dm/zh-cn/pm/dexp-logical-export.html

So my read is:

• for the first splitter PR, treating DISQL client directives as out of scope still seems fine;
• for DM compatibility, the Oracle-mode block opener list should add CREATE CLASS, CREATE CLASS BODY, and CREATE JAVA CLASS;
• the q-quote tests are still relevant for DM. I verified q'[...]' locally with ; and / inside the string.

[ymadd]

Thanks @haos666 — verifying against a live DM8 instance instead of guessing is much appreciated.

All three additions (CLASS, CLASS BODY, JAVA CLASS) slot straight into the opener const array, and your TYPE vs CLASS read is exactly right: the splitter only ever sees source text, so TYPE [BODY] stays and the CLASS forms are additive. JAVA CLASS needs no structural change either — the wait-for-slash model doesn't interpret the body, so a { … } Java body with its own semicolons is covered; I'd just add one test with Java code inside. And good to have DISQL directives confirmed out of scope for v1.

No open design questions left on my side — looking forward to the splitter PR, happy to review.