PSGO-268: TypeScript rewrite — Phase 0 scaffold - #607
Draft
Matovidlo wants to merge 69 commits into
Draft
Conversation
Stands up the TypeScript MCP server skeleton alongside the existing Python tree (Python keeps running until parity). Phase 0 of the Python -> TypeScript rewrite. - toolchain: package.json (@keboola/mcp-server, bin for npx, type module), tsup (ESM + dts + shebang), vitest, tsconfig (@keboola/tsconfig), oxlint + oxfmt (@keboola/oxlint-config) matching the keboola/ui monorepo - src/config.ts: faithful port of config.py (CLI/KBC_*/X-* resolution, aliases, URL amend, branch normalization, secret redaction) - src/server.ts + transports/stdio.ts + index.ts (bin): MCP SDK server over stdio with one scaffold tool; HTTP transport deferred to Phase 1 - __tests__: config unit tests + in-memory tools/list smoke test (16 tests) - format command scoped to src/__tests__ so it never touches the Python tree - plan + CI/CD runbook under feature_spec/mcp-typescript-rewrite/ Verified: type-check, oxlint, oxfmt --check, vitest (16 passed), tsup build, and `node dist/index.js` serves tools/list over stdio. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the HTTP transport so the server runs in container/server mode, mirroring the Python streamable-http setup. - transports/http.ts: Hono app + MCP SDK StreamableHTTPServerTransport in stateless mode (`sessionIdGenerator: undefined`); routes /mcp (POST), /, /health-check, and 405 for GET/DELETE /mcp - per-request config: X-* headers + Authorization: Bearer layered over the base Config on every request (parity with the Python per-request config) - logger.ts: pino to stderr (keeps stdout clean for stdio JSON-RPC) - index.ts: wire streamable-http / http-compat to startHttp - __tests__/http.test.ts: boots on an ephemeral port and drives the server with the SDK streamable-HTTP client (health, tools/list, base + per-request config) 20 tests pass; type-check, oxlint, oxfmt --check, and tsup build all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…Phase 2 foundation) Lays the client layer the tool handlers build on, reusing @keboola/api-client (v5) service clients instead of bespoke HTTP code. - clients/urls.ts: deriveServiceUrls() — faithful port of the Python KeboolaClient URL derivation (every service at https://<service>.<suffix> from the storage hostname); rejects non-`connection.` Storage API URLs - clients/keboola.ts: createKeboolaClients(config) builds storage/queue/metastore clients per request from the resolved Config (more added as tools are ported) - depend on @keboola/api-client ^5.0.0 (matches current ui source; scheduler + AI-docs arrive once keboola/ui#6862 publishes) - __tests__/clients.test.ts: URL derivation + factory guards (27 tests total) type-check, oxlint, oxfmt --check, vitest all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cross-cutting infra every tool needs (Plan §4/§5): TOON output encoding and a declarative tool registration wrapper, ported from the Python serializer= + tool_errors() machinery. - serialize.ts: toonSerialize / toonSerializeCompact via @toon-format/toon (the canonical TS impl of the format the Python server used). filterToonNulls is a faithful port of _filter_toon_nulls: drops null object keys but keeps list-of-objects column alignment. - mcp/tool.ts: registerTool() — typed ToolDefinition (zod inputSchema, annotations, optional serializer + recovery hint), serializes handler output to TOON text, and maps thrown errors to an MCP isError result with an optional recovery hint. - server.ts: get_server_info now goes through registerTool (dogfoods the helper). - tests: serializer null-filtering + TOON tables; smoke tests assert TOON output. 33 tests pass; type-check, oxlint, oxfmt --check all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First shared tool subsystem (used by project, storage, flow, jobs, data-app tools). Faithful 1:1 port of the Python links.ProjectLinksManager — pure UI/docs URL building from base URL + project id + optional dev-branch id. - constants.ts: ORCHESTRATOR/CONDITIONAL_FLOW/DATA_APP component ids, FlowType, FLOW_TYPES, MetadataField (subset) - links.ts: ProjectLinksManager with project/flow/scheduler/component/data-app/ transformation/job/bucket/table link builders + getLinks() dispatcher - __tests__/links.test.ts: branch segment insertion, flow-path routing, component type routing, FQ table-id split, getLinks dispatch 41 tests pass; type-check, oxlint, oxfmt --check all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Proves the full tool path end to end: zod input -> handler -> @keboola/api-client -> Keboola API, with TOON output + error wrapping. Ported from tools/project.py. - tools/project.ts: registerProjectTools(server, config); update_project_description posts the description to the current branch's metadata (storage.branches.saveDevBranchMetadata) - clients/keboola.ts: expose effective branchId (config.branchId ?? 'default' — 'default' is the SAPI production-branch alias, matching Python; no lookup needed) - server.ts: register project tools - msw-based test: drives the tool via the in-memory MCP client and asserts the request hits branch/default/metadata (prod) and branch/<id>/metadata (dev branch) - add msw devDependency 43 tests pass; type-check, oxlint, oxfmt --check all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ported from tools/jobs.py. Both modes plus log retrieval.
- tools/jobs.ts: get_jobs — MODE 2 lists summaries via queue.searchJobs with
branch/component/config/status filters + pagination + sort; MODE 1 fetches full
details per id via queue.getJob, optionally enriching with execution logs from
storage events (filtered by type, reversed to chronological). Maps the API's
component/config keys to componentId/configId; result/configData coerced to {}.
- clients/keboola.ts: createLinksManager() resolves the project id from the verified
token (port of ProjectLinksManager.from_client) for job/dashboard links.
- server.ts: register job tools.
- msw tests: list mapping + dashboard link, detail + job link, and filtered
chronological logs — all verified against mocked queue/storage endpoints.
run_job deferred: needs a queue createJob endpoint not yet in @keboola/api-client
(same gap pattern as scheduler; to be added alongside keboola/ui#6862).
46 tests pass; type-check, oxlint, oxfmt --check all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adopts the hybrid client strategy: a thin raw HTTP client (faithful port of the Python RawKeboolaClient) for endpoints where @keboola/api-client's typed methods diverge from the exact SAPI calls; api-client stays the default for clean cases. - clients/raw.ts: createRawClient — get/getText/post/put/patch/delete, SAPI/Bearer auth header, detailed error formatting (exception/error/exceptionId), retry with backoff on retryable statuses (incl. 409). Ported from clients/base.py. - clients/keboola.ts: expose rawStorage (rooted at <storage>/v2/storage; prefers the OAuth bearer token like Python's bearer_or_sapi_token). - tools/storage.ts: update_descriptions — parses bucket/table/column item_ids, groups by type, updates via the raw metadata endpoints (table+column metadata can't be done through api-client v5), aggregates per-item results. - msw tests: bucket metadata endpoint + body, column metadata via table endpoint, invalid item_id reported without any API call. 49 tests pass; type-check, oxlint, oxfmt --check all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes the jobs module. run_job uses the raw queue client (createJob isn't in
@keboola/api-client v5) — the hybrid strategy in action.
- tools/jobs.ts: run_job posts {component, config, mode:'run', branchId?, configRowIds?}
to the queue `jobs` endpoint and returns the created job's details + links.
- clients/keboola.ts: add rawQueue (raw client rooted at the queue base URL).
- fix: get_jobs MODE 2 now sends the raw branch id (omitted on production), not the
storage `default` alias — the queue search endpoint expects the actual branch id,
matching the Python client.
- extend jobs tests: run_job production (no branchId) + dev branch (branchId +
configRowIds in payload).
51 tests pass; type-check, oxlint, oxfmt --check all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ugh verbatim Ported from tools/oauth.py. - tools/oauth.ts: create_oauth_url mints a short-lived (1h) component-scoped Storage token via the raw tokens endpoint, then builds the external OAuth authorization URL (https://external.keboola.com/oauth/index.html?token=..&sapiUrl=..#/component/config). - mcp/tool.ts: string handler results are returned as text verbatim (parity with FastMCP), only objects are TOON-encoded — needed for URL/markdown-returning tools. - msw test: asserts the scoped token request body and the exact OAuth URL. 52 tests pass; type-check, oxlint, oxfmt --check all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- clients/keboola.ts: add rawAi (raw client rooted at the AI service base).
- tools/components.ts: get_config_examples fetches docs/components/{id} from the AI
service (api-client gap) and renders root/row examples as markdown; returns '' on
lookup failure (parity with Python).
- msw tests: markdown rendering + empty-string-on-404.
54 tests pass; type-check, oxlint, oxfmt --check all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ported from tools/doc.py. Posts the question to the AI service docs/question endpoint (via rawAi) and returns the answer text + source URLs. 55 tests pass; type-check, oxlint, oxfmt --check all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ported from tools/search.py. Posts the query to the AI service suggest/component endpoint (via rawAi) and returns matching component ids with scores + dashboard links. 56 tests pass; type-check, oxlint, oxfmt --check all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tpError Ported from tools/components/tools.py + utils.fetch_component + model.Component. - tools/components.ts: fetchComponent (AI catalog preferred, merges Storage `data` for sync actions, falls back to Storage API on 404), capabilitiesFromFlags + toComponent mapper (alias-tolerant), and get_components (concurrent fetch + links). - clients/raw.ts: RawHttpError carries the HTTP status so callers can branch (404). - extend components tests: AI+Storage merge with derived capabilities, and the 404 -> Storage fallback path. 58 tests pass; type-check, oxlint, oxfmt --check all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ported from tools/components/tools.py. - clients/keboola.ts: add rawSyncActions (raw client rooted at the sync-actions base). - tools/components.ts: run_sync_action fetches the root config, shallow-merges an optional row's parameters/storage on top, carries runtime+authorization from the root (docker-runner contract), and POSTs to the sync-actions `actions` endpoint. - test: row-over-root merge + authorization carry-through + endpoint. 59 tests pass; type-check, oxlint, oxfmt --check all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… redaction Ported from tools/components/tools.py + model.py configuration models. - clients/encryption.ts: redactSecrets — plaintext '#'-prefixed secrets -> [REDACTED], KBC:: ciphers kept (security-critical; secrets must not reach model context). - constants.ts: ALL_COMPONENT_TYPES + CONFIGURATION_FOLDER_NAME metadata key. - tools/components.ts: configuration model mappers (toConfigSummary, toConfiguration, toComponentSummary) + get_configs with all 3 modes — specific configs (full detail, redacted), by component_ids, and by component_types (expand empty -> all). - tests: list-by-id grouping + full-detail with secret redaction. NOTE: transformation parameter simplification (Snowflake/BigQuery display reshape) is deferred to create_sql_transformation (needs the inverse); transformations return raw params for now. 61 tests pass; type-check, oxlint, oxfmt --check all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds ajv (JSON-schema validation for flow/components), jose (OAuth JWS), and dd-trace (APM, loaded via NODE_OPTIONS in the image). Adds the gen:tools-docs / check:tools-docs npm scripts and syncs the lockfile. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the queryService base URL (query.<suffix>) to deriveServiceUrls, makes raw DELETE tolerate 204/empty bodies, and adds a bundling-aware resource-path helper so runtime resource files resolve under both src (vitest/tsx) and dist (bundled). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ports get_buckets/get_tables and query_data, plus WorkspaceManager (workspace resolution, Snowflake via Query Service API / BigQuery via Storage API, dialect-specific identifier quoting, paging + truncation). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ports create_config, add_config_row, update_config, update_config_row, create_sql_transformation, update_sql_transformation (diff-based updates, secret encryption, MCP metadata) plus the tolerant draft-07 JSON-schema validator. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ports get_flows, get_flow_schema, get_flow_examples, create_flow, create_conditional_flow, update_flow, modify_flow — preserving the conditional vs legacy distinction, with a local scheduler client and ajv-based schema validation. Includes the copied flow resource files. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ports get_project_info (+ system prompt resource), global search, the four semantic tools (via the Metastore service), and the six data-app tools (local DataScience client + code-template resources). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ports tool filtering (project/role/branch gating) and header authorization, the OAuth provider (JWS), /preview/configuration (reusing the same gating — AI-3438), the one-click prompts, and the validation-error formatter. Registers them and the remaining tools in createServer; adds the X-Allowed/Disallowed-Tools + X-Read-Only-Mode config fields. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds scripts/gen-tools-docs.ts (builds the server with gating skipped, lists tools via the in-memory client, emits TOOLS.md) wired to the check:tools-docs CI gate, and regenerates TOOLS.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the Python image with a multi-stage Node Dockerfile (ARG build vs ENV run, SKIP_ENV_VALIDATION at build, dd-trace via NODE_OPTIONS). Adds src/env.ts (validated process-level env, segregated from per-request Config) with HOSTNAME_SUFFIX/APP_ENV/APP_VERSION derivation ported from server.py, and copies runtime resources into dist via tsup. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the Python/tox CI matrix with Node (type-check, lint, vitest, check:tools-docs, build) and swaps the PyPI publish for npm publish on semantic v* tags. Points the Anthropic registry server.json at the npm package. release.yml/kaibench.yml are unchanged — both build the Node Dockerfile and drive it via supported CLI flags. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adopts the keboola/go-utils testproject model for TS integtests: projects.json pool, redis lease per (host, projectId), per-test-case acquisition, infinite retry on pool exhaustion, fs-lock fallback for local runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…back) Ports go-utils pkg/testproject to TS: projects.json loader, redis locker (ioredis SET NX PX + Lua refresh/release, auto-extend at TTL/4), host-local fs-locker fallback, pool getTestProject with infinite-retry-on-exhaustion + backend selector, cleanProject reset with dedicated-project guard, and the per-test getTestProjectForTest fixture. Adds ioredis; type-checks integtests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the .github/ci/projects.json template (build/ is gitignored), the export-kbc-projects composite action (envsubst TEST_KBC_PROJECT_* secrets), the separate vitest.integ config + test:integ scripts, and rewrites integtests/README.md for the redis-leased per-case flow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…id to typed ai
Splits the 1042-line search.ts into src/tools/search/{index,tools,globalSearch,
model,jsonpath}.ts. Migrates find_component_id to the typed ai.suggestComponent.
Keeps the global search tool on the raw client (documented): the typed
storage.search.globalSearch serializes array params in a form SAPI rejects.
Behavior unchanged (5 tests green).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…position Deletes the now-superseded data_apps.model.ts, semantic.ts, semantic.model.ts (their content moved into the respective tools/<module>/ directories). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tComponent) Ports the find_component_id case from integtests/tools/test_search.py. Needs no seeded data; verifies the raw→typed ai.suggestComponent migration against the real AI service (returns ex-generic-v2). Verified passing via the fs-lock. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds an `integration_tests` job that runs in parallel with the unit `build` job (keeping the fast lane fast). It unwraps the project pool via the export-kbc-projects action (substitutes $TEST_KBC_PROJECT_*_TOKEN secrets into .github/ci/projects.json — same mechanism as keboola/go-monorepo), then runs `npm run test:integ` with the shared redis lease (TEST_MCP_PROJECTS_LOCK_HOST/ PASSWORD) so leases coordinate cross-tool. Push-only (fork PRs lack secrets). Commits the 10-project pool template with token placeholders (no secrets). Required repo config: vars.TEST_MCP_PROJECTS_LOCK_HOST, secrets TEST_MCP_PROJECTS_LOCK_PASSWORD + TEST_KBC_PROJECT_<id>_TOKEN per project. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| name: Integration test results (${{ matrix.python-version }}) | ||
| path: ./integtest-results.xml | ||
| reporter: 'java-junit' | ||
| secrets: ${{ toJSON(secrets) }} |
…OJECTS_FILE Aligns with the keboola/go-monorepo convention: TEST_KBC_PROJECTS_FILE is read from the centrally-maintained repo/org variable (filename), not hardcoded from the export-action output. The export step still writes the pool file at the workspace root from the committed template + TEST_KBC_PROJECT_* secrets. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…729/2731/2732/2908) Replaces the placeholder pool with the actual Keboola integration-test projects documented in integtests/README.md — 2728/2729 (Snowflake) + 2731/2732 (BigQuery) pool projects and 2908 (storage-branches), all on the europe-west3.gcp stack (gcs staging). Tokens stay $TEST_KBC_PROJECT_<id>_TOKEN placeholders filled from GitHub secrets. Re-documents the pool table in the README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New pool on europe-west3.gcp: Snowflake 3053/3054, BigQuery 3056/3057, and 3055 (Snowflake, storage-branches). Updates the committed template + README table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds integtests/helpers/seed.ts (creates the standard fixtures — 2 buckets, 1 CSV table via sync create-from-string, 2 configs — over the raw Storage API, matching the Python conftest) and ports the storage get_buckets/get_tables integ tests. Verified live against the project pool via the fs-lock. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Storage API rejects synchronous force-drop ("Synchronous drop is not
supported, use async call"), which broke every seeded integ test at the clean
step. Switch bucket deletion to the async call (force=true&async=true) and poll
the returned storage job to completion before proceeding. Verified live.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ython) Ports the remaining Python integ cases onto the project-pool harness + seed helper, across components, flow, jobs, sql, search, storage, storage_branches, data_apps, semantic, and the server-level suites (mcp_server, errors, validate, workspace). Each test leases a project, seeds fixtures where needed, and drives the real MCP server over the in-memory transport, asserting on TOON output. Feature/variant-gated cases self-skip when the leased project lacks the capability (semantic-tooling, storage-branches, conditional-vs-legacy flow) — they run in CI against the dedicated projects. Also tolerate a 403 on branch-metadata delete during cleanProject. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- cleanProject: tolerate 403 on config purge-from-trash (some pool tokens lack the permission); leftover trashed configs are harmless. - jobs: extractJobId accepts TOON's quoted id (id: "123"). - storage/sql: pin FQN + warehouse-native-type tests to Snowflake (BigQuery leases expose no fully_qualified_name); decode the TOON-quoted Snowflake FQN (JSON-escaped) instead of truncating it at the first inner quote. Live: the five previously-failing files now pass (70/70 run + the seeded COUNT). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Splits the 848-line validation.ts into src/tools/validation/{index,types,sanitize,
validate,model}.ts (all <~400 LOC). Public API at @/tools/validation unchanged
(re-exported via index.ts); storage-schema.json import rewired. Pure move — 30
components tests green, tsc/lint clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…Query fix)
Splits the 962-line storage.ts into src/tools/storage/{index,model,tools,usage}.ts.
Makes get_tables fully_qualified_name dialect-aware: BigQuery now gets a
backtick-quoted `dataset`.`table` FQN (+ STRING native-type default) resolved from
the project's defaultBackend, instead of no FQN — so query_data works on BigQuery
tables. Snowflake path unchanged. Integ FQN + seeded-COUNT tests parametrized over
both backends; verified live (Snowflake + BigQuery). registerStorageTools export
preserved; 8 unit tests green, tsc/lint clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The TypeScript rewrite reaches full parity — all 39 tools, 394 unit tests, and the integration suite (91 pass / 13 feature-gated skips) green — so the Python sources are removed: src/keboola_mcp_server/, the Python test suite (tests/) and integ .py files, pyproject.toml, uv.lock, mypy.ini, and logging-json.conf. The TS integ tests, their data fixtures/helpers, and the data-app code templates (src/resources/data_app/*.py) are retained. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Switch dependabot from the uv (pyproject) ecosystem to npm; remove the now-gone src/keboola_mcp_server ignore from oxlint + oxfmt configs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…g test modify_python_js_data_app injects the query_data code (needs the workspace dialect), so the prod+draft lifecycle test now provisions a read-only workspace and connects with its schema (as the Streamlit tests do). Full integ suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…AI service)
Adds feature_spec/docs-search-pgvector/{RFC.md,architecture.md} designing the
move of docs_query / find_component_id / component-docs off the AI service onto
@keboola/docs-search (keboola/ui#6672) — a pgvector index built out-of-band by a
cron job and only read by the MCP.
RFC covers: SDK retrieval mapping, full AI-service removal, Postgres in
docker-compose, the availability gate, and the access-token open question
(recommendation: index is infra via server-side DATABASE_URL, tools follow
read-only visibility). Architecture doc covers: Postgres+pgvector provisioning
(Terraform + extension enablement), the aside cron index-builder (incremental +
transactional + gated), index lifecycle & the MCP read contract, failure modes,
and a staged rollout — the MCP never builds the index, only reads the last good one.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…cs-search Implements the docs-search RFC (feature_spec/docs-search-pgvector/): migrate the two semantic-retrieval docs tools off the legacy AI service onto a prebuilt pgvector index that the MCP only ever reads. - New src/clients/docsSearch.ts: DocsSearch provider over a process-scoped pg.Pool. Retrieval tier (search/answerQuestion/recommendComponents/isReady) vendored from @keboola/docs-search (keboola/ui#6672) behind its exact interface, since that package is a private, unpublished workspace pkg; documented swap-to-published-package path. - docs_query -> answerQuestion, find_component_id -> recommendComponents. Output shapes unchanged. Component id recovered from the doc source_key (component:<id>). - Availability gate (RFC pt5): DOCS_INDEX_TOOL_NAMES filtered from tools/list and denied on call when no index is configured/reachable (getDocsSearch() === null). The rest of the server is unaffected. - env.ts: optional DATABASE_URL + DOCS_EMBEDDER_* + DOCS_LLM_* (deployment infra; server boots without them). docker-compose.yml adds a pgvector/pgvector dev service. - Removed the now-dead typed ai client (suggestComponent); rawAi retained only for the component catalog docs/components/{id} (schemas/examples the index does not hold). - Tests: fake DocsSearch injected via setDocsSearchForTests; mapping + gate coverage. Deviations from the RFC (recorded in RFC.md "Implementation status"): component-catalog paths (get_config_examples, fetchComponent) stay on the AI catalog — the index holds markdown docs, not config schemas; and the SDK needs to expose sourceKey on recommendComponents. Index building stays out of scope (architecture doc). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The docs-search SDK gap (recommendComponents not exposing sourceKey) is fixed on the SDK PR (keboola/ui#6672, aa8a779). Update the vendored client comment + RFC to reflect that the vendored SELECT now matches the published shape, so the swap will be a clean drop-in. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ment gate choice The docs_query integration test still assumed the legacy AI path; docs_query is now gated off unless a pgvector index is configured, so the test would fail in projects without one. Skip the suite (parity with storage_branches) when DATABASE_URL/DOCS_EMBEDDER_* are unset, and assert real retrieval only where an index is reachable. Also document (ponytail comment) that the availability gate is intentionally config-level, not a per-request reachability probe — isReady() remains available for strict gating. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…stcontainers integ Make the docs-search tools work end-to-end on localhost and in CI without any live service or API key, closing the two RFC gaps that were previously deferred. - StubEmbedder (deterministic, offline) added to the provider; selected via DOCS_EMBEDDER_MODEL=stub (no endpoint/key needed). createEmbedderFromEnv is shared so the index is built and queried with identical vectors. - npm run docs:build (scripts/docs-build.ts + scripts/docsIndex.ts): migrate the pgvector schema + seed a small fixture corpus into DATABASE_URL. Dev mirror of the production out-of-band build (fixtures, not real docs — connectors stay on the build side per #6672). - integtests/tools/doc.test.ts rebuilt self-contained: provisions pgvector via testcontainers, seeds with StubEmbedder, drives docs_query + find_component_id through the MCP and asserts real retrieval + component-id recovery. Skips with a warning if Docker is unavailable; the existing integration_tests CI job runs it as-is. - Verified locally: docker compose up + docs:build + retrieval returns the seeded doc and recovers keboola.ex-db-mysql (score 1.000), isReady() true. Quickstart: docker compose up -d pgvector DATABASE_URL=postgres://mcp:mcp@localhost:5432/docs DOCS_EMBEDDER_MODEL=stub npm run docs:build Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… the other tests The docs-search gating broke integ tests that assumed the old AI-service docs path. Wire a shared Postgres for the integ run (like the go-monorepo metastore pattern) and fix the affected tests. - CI integration_tests job: `docker compose up -d --wait pgvector`, and set DATABASE_URL + DOCS_EMBEDDER_MODEL=stub (+ DIM) on the test:integ step so docs_query / find_component_id are enabled during integ. The rest of the suite is unaffected. - integtests/tools/doc.test.ts: env-driven (skips unless DATABASE_URL + stub embedder set), seeds the fixture corpus into the shared pgvector in beforeAll with the deterministic StubEmbedder — the same embedder the server uses at query time — and drives docs_query + find_component_id through the MCP. Dropped @testcontainers/postgresql (no longer needed). - mcp_server.test.ts: exclude docs_query / find_component_id from the strict tool-set comparison (feature-gated on the index, like search/semantic). - errors.test.ts: drop the AI-service 422 empty-query case (docs-search returns no results, not a 422 — covered by doc.test.ts). - search.test.ts: drop the find_component_id case (moved to doc.test.ts; it is a docs-index tool now, not a global-search one). Verified locally: docker compose up + seed + the env-driven provider (getDocsSearch, exactly as the server resolves it) returns the seeded doc + recovers keboola.ex-db-mysql, isReady(). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…r-compose stack Two production-image bugs surfaced while wiring the local run stack: 1. The image never booted: @keboola/api-client ships ESM with extensionless subpath imports (`import 'dayjs/plugin/utc'`) that Node 22's strict ESM resolver rejects (ERR_MODULE_NOT_FOUND). It was marked external, so the broken import shipped as-is into dist/index.js. Unit/integ tests hid it (vitest inlines api-client). Fix: force-bundle @keboola/api-client + dayjs via tsup `noExternal` so esbuild resolves the imports at build time. Verified: `node dist/index.js --transport streamable-http` now serves /health-check, with and without the dd-trace import hook. 2. The resources copy was non-idempotent (`cp -R src/resources dist/resources` nests into dist/resources/resources on a re-run, leaving the runtime path empty). Made it `rm -rf ... && cp` so repeated builds are safe. Also make the local run stack real: - tsup emits `dist/docs-build.js` (migrate + seed CLI) so it runs in the prod image. - docker-compose gains `docs-seed` (one-shot migrate+seed) and `mcp` (streamable-HTTP) services alongside pgvector. `docker compose up --build` brings up the full stack; `docker compose up -d --wait pgvector` still starts only Postgres (used by test:integ). Verified end-to-end: full stack up → docs-seed exits 0 (5 docs/chunks) → mcp healthy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…per-request init logs
run_sync_action (and any typed-client tool) returned an opaque "Bad Request": the useful
reason ("Invalid access token") + support exception id live on api-client's `ApiError.data`,
but the tool layer used only `error.message` (the bare HTTP status text). Add
describeToolError() at the tool choke point to compose "<status>: <data.error> (exception
ID: …)". Our raw client already builds this into its message, so it passes through.
Also demote the two "… tools initialized." logs from info to debug: createServer() runs per
HTTP request, so at info they spam one pair of lines per request with no signal.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e vector dimension Adds a third embedder so the docs index can run with no external service and no API key, and makes the vector size configurable across the embedder and the pgvector schema. - LocalEmbedder (DOCS_EMBEDDER_MODEL=local): runs a HuggingFace model as ONNX on CPU via transformers.js, mean-pooled + L2-normalized. Model id via DOCS_EMBEDDER_LOCAL_MODEL (default Xenova/all-MiniLM-L6-v2, 384). @huggingface/transformers is an OPTIONAL dep, dynamically imported — stub/remote users don't need it; a clear error is thrown if a `local` run is missing it. - createEmbedderFromEnv now selects stub | local | remote, each with its own default dim (3072 stub/remote, 384 local), all overridable via DOCS_EMBEDDER_DIM. - Configurable dimension end-to-end: migrationSql(dim) parametrizes halfvec(N); the seeder drops+recreates the tables when the dim changes (a dim switch is a full reindex anyway). - docker-compose: a shared `x-docs-embedder` anchor drives docs-seed + mcp together so the build-time and query-time embedders can never drift; flip it to local/remote in one place. - Tests: embedder selection + configurable-dim unit tests (410 total green). Verified end-to-end against pgvector with the local model: it does real semantic matching where the stub cannot — "MySQL database extractor" -> keboola.ex-db-mysql (0.635), "mysql" -> keboola.ex-db-mysql (0.452), "load data from a snowflake warehouse" -> keboola.wr-db-snowflake (0.557); halfvec column built at halfvec(384). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Parity with the Python server, which logged its resolved config at boot. On startup the
server now emits one INFO line ("Keboola MCP server starting") with the transport/host/port,
version, docs-index status (e.g. "configured (model=local, dim=384)" vs "not configured"),
the redacted Config, and the full deployment env.
Redaction (redactedEnv in env.ts): secret-named keys (token/secret/password/api_key/jwt)
are masked to ***, DATABASE_URL credentials are stripped (host/db kept), and every schema
key is shown (unset → null) so the dump is complete. Config.toString() already redacts its
own secret fields. No raw token/key/password is ever logged.
This is the first thing to check when a tool misbehaves — e.g. it immediately shows whether
DATABASE_URL / DOCS_EMBEDDER_MODEL are set, which was exactly the recent confusion.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ublic docs
Adds `npm run docs:crawl`: a simplified local stand-in for the production out-of-band index
builder. It crawls the public help.keboola.com + developers.keboola.com sitemaps, extracts
each page's main content (cheerio), chunks it, embeds with the configured embedder, and
writes the pgvector index the MCP reads. No Keboola stack/token — only public HTTP + local
Postgres. Flags: --limit N, --source help|dev|all.
- seedDocsIndex now chunks each source into multiple doc_chunk rows (chunkText: ~1000-char
windows, 100 overlap, word-boundary), so long real pages retrieve well; short fixtures
stay one chunk (existing behavior/tests unchanged).
- cheerio added as an optional dep, dynamically imported (like @huggingface/transformers).
Usage:
docker compose up -d --wait pgvector
DATABASE_URL=postgres://mcp:mcp@localhost:5432/docs DOCS_EMBEDDER_MODEL=local DOCS_EMBEDDER_DIM=384 \
npm run docs:crawl -- --limit 50
Then point the MCP at the same DATABASE_URL + embedder.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ull pages) Without an LLM, answerQuestion concatenated the full content of the top-K (15) parent docs. Fine for the tiny fixtures, but against the real crawled index that was 94K–314K chars per query — it blew past client context limits. Cap the LLM-less extractive answer: top 5 docs, each truncated to 800 chars, titled. Also fetch only 5 parents (not 15) on the no-LLM path. Real-index docs_query now returns ~0.8–4 KB instead of hundreds of KB; full-page context remains the LLM path's job. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds scripts/docs-crawl.ts to the tsup entries so the production image ships `dist/docs-crawl.js` (alongside docs-build.js). This lets an in-cluster index build run `node dist/docs-crawl.js` — crawl the public help+dev docs, embed with the configured (Azure OpenAI) embedder, and write the pgvector index the server reads. cheerio is an optionalDependency, so `npm ci --omit=dev` keeps it in the runtime image. Also type-guards the --source filter (`.filter((s): s is Source => …)`) so the file passes the dts build now that it's a compiled entry, and bumps the version. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
First step of rewriting the Keboola MCP server from Python (FastMCP) to TypeScript at 1:1 functional parity — see PSGO-268. Full plan + CI/CD runbook are in
feature_spec/mcp-typescript-rewrite/.This PR stands up the TS skeleton alongside the Python tree (Python keeps running until parity), so it's safe to land incrementally. Draft — the rewrite proceeds module-by-module in follow-up commits.
What's in Phase 0
keboola/uimonorepo:package.json(@keboola/mcp-server,binfornpx, ESM), tsup (ESM + dts + shebang), vitest,@keboola/tsconfig, oxlint + oxfmt via@keboola/oxlint-config.src/config.ts— faithful port ofconfig.py: CLI /KBC_*env /X-*header resolution with key normalization + aliases, URL amendment, branch-id normalization, secret redaction.src/server.ts+transports/stdio.ts+src/index.ts— MCP SDK server over stdio with one scaffold tool (get_server_info). HTTP/streamable-HTTP (Hono) is Phase 1.__tests__/— config unit tests + an in-memorytools/listsmoke test.src/__tests__so it never reformats the Python sources/docs.Verification (local)
tsc --noEmit✅ ·oxlint✅ ·oxfmt --check✅vitest run→ 16 passed ✅tsupbuild ✅ ·node dist/index.jsservestools/listover stdio ✅Depends on / related
@keboola/api-clientgap support (scheduler + AI docs endpoints): keboola/ui#6862 — consumed once published.Not yet (next phases)
HTTP transport, client adapters (
@keboola/api-client), zod models, the 39 tools, tool-filtering/auth/preview/oauth, TOOLS.md generator, CI/Docker/npm/KaiBench, then integtests + Python removal.🤖 Generated with Claude Code