AI-2922: local-backend mode — Phase 2/3/4 (Docker execution, data apps, security hardening)#488
Draft
Matovidlo wants to merge 34 commits into
Draft
AI-2922: local-backend mode — Phase 2/3/4 (Docker execution, data apps, security hardening)#488Matovidlo wants to merge 34 commits into
Matovidlo wants to merge 34 commits into
Conversation
Add --local-backend / --data-dir CLI flags and a complete local-mode server factory (create_local_server) backed by the filesystem and DuckDB. New tools registered in local mode (same MCP names as platform mode): get_tables — scans <data-dir>/tables/*.csv, returns name/columns/rows/size get_buckets — returns single virtual "local" bucket query_data — executes DuckDB SQL against CSV files, returns markdown table search — matches filenames and CSV column headers (case-insensitive) get_project_info — returns local metadata (data_dir, table count, engine) Platform tools (run_job, create_flow, etc.) are not registered in local mode. create_server is kept as a backward-compatible alias for create_platform_server. duckdb >= 0.10.0 added as optional [local] dep and included in tox test extras. 42 new tests, all 1288 tests pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…hema discovery Add docker.py with ComponentSetupResult/ComponentRunResult models and all execution logic (setup_component, run_image_component, run_source_component) plus pure helpers for subprocess orchestration. Add schema.py with Developer Portal API calls (get_component_schema, find_component_id) for component metadata discovery without authentication. Wire both into tools.py as four new MCP tools: setup_component, run_component, get_component_schema, find_component_id. Extend LocalBackend with delegation methods. All 97 local-mode tests pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…e_to_keboola Add config.py: ComponentConfig CRUD (save/load/list/delete) storing component configurations as JSON files in <data-dir>/configs/. Add migrate.py: migrate_to_keboola uploads local CSV tables and saved configs to Keboola Storage via the public API (no kbc CLI dependency required). Creates the target bucket automatically; handles 422 already-exists gracefully. Add to backend.py: write_csv_table, delete_csv_table, configs_dir property, and config/migrate delegation methods. Add to tools.py: write_table, delete_table, save_config, list_configs, delete_config, run_saved_config, migrate_to_keboola — completing the end-to-end local workflow from data ingestion through component execution to platform migration. All 146 local-mode tests pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…_to_keboola_local tests - Add config_count field to LocalProjectInfo and populate it in get_project_info_local - Extend _LOCAL_PROJECT_INSTRUCTION to mention new Phase 3 tools - Add migrate_to_keboola_local tests in test_tools.py (tool-layer wrapper coverage) - Add test_integration.py with end-to-end workflows: data, config, and migration - Add test_local_server_registers_all_tools asserting all 16 tools registered - Bump version 1.56.0 → 1.57.0 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sync docs/local-backend.md with what was built across Phases 1, 2, and 4: - Status: RFC → Implemented (Phase 3 deferred) - Tool surface: add 9 new local-only tools (write_table, delete_table, save_config, list_configs, delete_config, run_saved_config, get_component_schema, find_component_id); move modify_data_app to platform-only; add docs_query + semantic tools with⚠️ callout - Server factory: remove register_local_prompts (not implemented) - run_component / setup_component: rename component_git_url → git_url; update return types to Pydantic models; note auto-setup in run_component - LocalBackend: remove generate_ts_app (Phase 3); add actual method list including write_csv_table, delete_csv_table, config CRUD, migrate_to_keboola - Filesystem layout: remove apps/ and catalog.json; add runs/ and configs/ - Migration: replace kbc-cli delegation with direct httpx Storage API approach - Schema Tier 3: note not implemented - Phases: add status badges (✅/⏸/🔶) and done/not-done breakdown for Phase 4 - Add Known Gaps table - Add End-to-End Smoke Tests section (Python transformation + Generic Extractor) - Dependency summary: add httpx entry Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ke tests
- Fix schema.py: use apps-api.keboola.com/apps/{id} (was components.keboola.com
which returns HTML). The correct public API is the Keboola Developer Portal
REST API — no auth required.
- Fix find_component_id: the API has no server-side text search; fetch all 366
apps (limit=500) and filter client-side by id/name/description.
- Fix docker.py run_source_component: memory_limit was accepted but never passed
to `docker compose run` — add --memory flag.
- Add scripts/test-local-backend.sh: smoke test script for local-backend mode
covering all non-Docker functionality (13 tests) with optional --no-docker
and --no-portal flags.
- Update test_schema.py and test_tools.py to match the real API response field
names (configurationSchema, imageTag, shortDescription) and client-side
filtering behavior.
- Bump version 1.57.0 → 1.58.0.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix Developer Portal API endpoint: apps-api.keboola.com/apps/{id}, not
components.keboola.com/components/{id} (which returns an HTML SPA)
- Document find_component_id client-side filtering (API has no server-side search)
- Add "Running the Server" section with CLI invocation and Claude Desktop config
- Add "Automated Smoke Tests" section documenting scripts/test-local-backend.sh
- Update Known Gaps: remove resolved Developer Portal issue, add note about
find_component_id fetching the full catalog per search call
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Local-mode tool functions have no Context parameter (they use LocalBackend instead). _trigger_event was asserting its presence for telemetry, producing a WARNING log on every local tool call. Return early instead of asserting — telemetry is skipped for local mode, which has no platform event backend. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…twork Exposes --docker-network CLI flag (env KBC_DOCKER_NETWORK, default bridge) that flows through all Docker calls in setup_component, run_image_component, and run_source_component. Users on machines where bridge DNS fails can pass --docker-network host to restore internet access inside build containers. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… of unsupported --network flag docker compose build does not accept --network as a CLI flag (only docker run / docker compose run do). Inject the build network via a temporary docker-compose.override.yml instead, which docker compose auto-discovers. The override file is written only when network != 'bridge' and is cleaned up after build regardless of outcome. Existing override files are left unchanged. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… of unsupported --network flag docker compose build and docker compose run do not accept --network as a CLI flag in this Docker Compose version. Remove --network from all docker compose commands and inject the network via a temporary docker-compose.override.yml instead (both build.network and network_mode for the dev service). The override file is written before build+dep-installs and cleaned up in a finally block. Only docker run (image mode) keeps --network since it is a valid docker CLI flag. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ride file docker compose run does not accept --memory or --network as CLI flags in this Docker Compose version. Inject both via docker-compose.override.yml (mem_limit and network_mode) so the dev service picks them up automatically. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…oot permission errors Docker creates files as root inside bind-mounted ./data directories. Replace clone_dir/data with a fresh run dir under data_dir/runs/ (created by Python, user-owned) and override the compose volume mount to point Docker at it. This eliminates PermissionError when prepare_data_dir tries to write into a previously Docker-created root-owned data/in directory. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
File-based extractors (e.g. keboola.ex-http) write CSV output to /data/out/files/ rather than /data/out/tables/. Extend collect_output_tables to also glob *.csv from the sibling out/files/ directory and copy them into the local catalog. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…oola Pass --api-url and --storage-token through create_local_server → LocalBackend so migrate_to_keboola can be called without repeating credentials in the tool call. Both parameters are now optional (str | None = None); the tool falls back to whatever was provided at server startup. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Five new local-mode MCP tools: - create_data_app: generates a self-contained HTML dashboard from a list of SQL-driven chart configs (bar/line/scatter/pie/table/heatmap); chart data is pre-computed by DuckDB server-side - run_data_app: starts a background Python HTTP server, returns browser URL - list_data_apps: lists all apps with running status - stop_data_app: kills the HTTP server for an app - delete_data_app: removes the app directory Dashboard uses Apache ECharts + Pico CSS served from CDN. All dynamic content is embedded as JSON in <script type="application/json"> blocks; no innerHTML with untrusted input. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Phase 3 marked done (HTML dashboards, not Vite/React) - Add all 5 data app tools to tool surface table - Document --docker-network flag and override-file approach - Document optional --api-url / --storage-token for migrate_to_keboola - Add apps/ to filesystem layout - Update dependency section (ECharts/Pico from CDN, no npm) - Correct tool count 16 → 21 in smoke test description - Refresh Known Gaps: remove Phase 3, add "data is static snapshot" Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…e job
Feature 1 – Progress reporting for Docker runs:
- Add _run_subprocess_logging() helper that streams stdout+stderr to a
per-run log file in real-time via Popen; callers can `tail -f` the path
while the MCP tool call blocks.
- Replace subprocess.run() with _run_subprocess_logging() in both
run_image_component() and run_source_component().
- Add log_file field to ComponentRunResult so the LLM can surface the path.
- Update docker tests to mock _run_subprocess_logging instead of subprocess.run;
add test_run_image_component_log_file_path.
Feature 2 – Stub hints for platform-only tools:
- Expand _LOCAL_PROJECT_INSTRUCTION to explicitly name unavailable platform
tools (run_job, create_flow, create_sql_transformation, deploy_data_app,
docs_query, create_config) and their local alternatives, so the LLM never
confusedly tries to call them in local mode.
Feature 3 – CI smoke test job:
- Add local_backend_smoke CI job that runs scripts/test-local-backend.sh
--no-docker --no-portal on ubuntu-latest on every push/PR.
- Fix smoke script expected tool count (16 → 21) to include the 5 data app
tools added in Phase 3.
Fix: regenerate iris-dashboard index.html from fixed template (literal \n
in pie chart formatter replaced with \\n so JS escape is correct).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Five diagrams covering: - Tech stack (Python, FastMCP, DuckDB, Docker, ECharts/Pico CSS) - Tool surface comparison: shared / local-only / platform-only + local alternatives - Data app end-to-end sequence (create → serve → browser rendering) - Three guided E2E workflows for new users (A: explore data, B: extract, C: migrate) - Docker component execution flow (Common Interface contract) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… prompts (v1.60.0)
RFC-1: Rename local tool list_configs→get_configs; add delete_config to platform mode
RFC-2: Replace pre-baked HTML data with Query Service emulator (appserver.py + DuckDB);
JS template now uses live 3-step HTTP API identical to production query.keboola.com
RFC-3: Add onboarding instructions and 3 workflow prompts to create_local_server()
RFC-4: Add docs/rfc/ design records for RFC-1 and RFC-2
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Tests all RFC-1/2/3 behaviors through FastMCP Client (full JSON-RPC layer): tool discovery, onboarding instructions/prompts, get_configs, delete_config, create_data_app (live HTML assertions), appserver HTTP API (3-step QS contract), and real DuckDB query results via the running appserver. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Relative data_dir (e.g. ./keboola_data) produced a relative volume path like keboola_data/runs/run-xxx which Docker rejects. Use .resolve() to get an absolute path, matching the fix already applied in run_git_component. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ng DOM ECharts v5 throws "chart instance already initialized on this dom" when echarts.init() is called on a container that already has a registered instance. setLoading/clearBox/setError were clearing the DOM without calling ec.dispose() first, so Refresh always crashed the dashboard. Fix: extract disposeBox(box) helper that calls echarts.getInstanceByDom()+dispose before clearing children; all three functions now call it. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ECR registry images require AWS credentials not available in local mode. Updated _LOCAL_ONBOARDING, run_component/save_config tool descriptions, and ComponentConfig field descriptions to recommend git_url (public GitHub) as the preferred execution path for open-source Keboola components. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…nfigs Components like keboola.ex-daktela default to 7 days, silently missing older records. Added guidance in _LOCAL_ONBOARDING and save_config docstring to always call get_component_schema first, then ask the user whether they want exploratory (7-30 days) or full-history (explicit start date) data before setting date_from in parameters. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds optional authorization: dict param throughout the local pipeline (prepare_data_dir → run_image_component / run_source_component → backend → run_component_local / save_config_local / run_saved_config_local → MCP tools run_component + save_config). When present, authorization is written to config.json under "authorization" alongside parameters, enabling OAuth-based components (Google Analytics, Drive, Sheets, etc.) to run locally with manually-obtained credentials. ComponentConfig gains an authorization field so credentials are persisted with the saved config and automatically forwarded on run_saved_config. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ents with relative path Some components (e.g. component-shopify-v2) set KBC_DATADIR=./data in their docker-compose.yml. With our volume override mounting run_dir at /data, the component never found config.json because it resolved ./data relative to the /code workdir inside the container. Force -e KBC_DATADIR=/data/ on the docker compose run command so all source- based components always read config from the bind-mounted run_dir regardless of what their docker-compose.yml sets. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Security fixes (all local-backend only): - config.py: reject config_id values containing path separators (CWE-22) - docker.py: validate memory_limit and network against allowlists to prevent YAML injection into docker-compose.override.yml (CWE-74) - appserver.py: bind to 127.0.0.1 instead of 0.0.0.0; disable DuckDB external file access (enable_external_access=false); html.escape() on app.json values rendered into the index page (CWE-79) New feature: /apps index page - Intercepts GET /apps, /apps/ and /apps/index.html in the appserver - Dynamically renders a card grid from app.json + .running.json with running status badges and clickable links Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace _tool_manager.get_tools() with list_tools(run_middleware=False) (FastMCP 3.x dropped private _tool_manager API) - Bump version to 1.61.0 (minor: local-backend mode, security hardening, /apps index page) - Regenerate uv.lock with fastmcp 3.2.0, duckdb 1.5.2 - Apply black formatting to project.py, validation.py, test_server.py - Regenerate TOOLS.md (additionalProperties: false in create_config schema) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Rename src/keboola_mcp_server/tools/local/ → src/keboola_mcp_server/local_backend/ and tests/tools/local/ → tests/local_backend/. This clearly separates the experimental local-backend from the main tools package. The only change outside the local_backend directory is the import in server.py (one line). All internal imports updated accordingly. The local-backend is intentionally maintained at a lower bar than the main codebase: it is experimental, Docker/local-only, and excluded from the platform Keboola service. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
… rename - backend.py: bind to 127.0.0.1 instead of '' (all interfaces) in _find_free_port() - scripts/test-local-backend.sh: update all heredoc Python imports from tools.local.* to local_backend.* (missed by the earlier sed over *.py only) - scripts/test-local-backend.sh: replace FastMCP 2.x private API server._tool_manager.get_tools() with server.list_tools(run_middleware=False) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The delete_config tool was added to the standard MCP server (components/tools.py) but the integration test expected-tools set was not updated, causing CI failure. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
DuckDB's read_csv_auto() re-infers column types on every connection, causing
ConversionException when the AI writes SQL against BOOLEAN or TIMESTAMP columns
that look like strings (e.g. sla_overdue = 'true', closed != '').
Adds a .schema.json sidecar alongside each CSV that records DuckDB-detected
column types. On write (write_csv_table) and on first connection load, types are
auto-detected via DESCRIBE and persisted. Subsequent connections use explicit
read_csv(?, columns={...}) so types are stable across sessions.
- LocalBackend: _schema_path, _load_schema, detect_and_save_schema,
_save_schema_from_describe; _duckdb_connection uses explicit types when
sidecar present, falls back + refreshes on stale schema; delete_csv_table
removes sidecar alongside CSV
- LocalColumnInfo model + LocalTableInfo.column_types in get_tables / write_table
so the AI client sees types before writing SQL
- _LOCAL_PROJECT_INSTRUCTION updated with BOOLEAN/TIMESTAMP SQL guidance
- 14 new tests covering creation, loading, boolean query, first-load generation,
delete cleanup, and stale-schema fallback
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…(v1.62.0) _ensure_bucket 400 fix: some Keboola stacks return HTTP 400 (not 422) when creating a bucket that already exists. migrate.py now checks the response body for 'already exist' on 400 before raising, so migrate_to_keboola no longer fails on every second call to the same bucket. RFC-002 amendment: adds 'Migrating Local Data Apps to the Platform' section documenting how to extract SQL queries from app.json and recreate as Streamlit via modify_data_app. Proposes migrate_data_app Phase 4 tool. RFC-004 spec: full design for import_from_keboola (platform → local), covering table export/poll/download flow, config import, output model, name collision handling, and the Phase 4 data app import stretch goal. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Linear: AI-2922
Change Type
Summary
Full implementation of the local-backend mode — a self-contained, experimental execution environment that lets users run Keboola components and analyse data locally without a platform account.
What's included:
Core execution engine (
src/keboola_mcp_server/local_backend/)docker.py— Common Interface runner:run_image_component(pre-built ECR/registry image) andrun_source_component(clone +docker compose buildfrom GitHub)backend.py—LocalBackendorchestrator: manages run directories, catalog, config CRUD, data app lifecycleconfig.py— Persisted component configs (ComponentConfig) with full parameter + OAuth authorization supportappserver.py— Embedded HTTP server: Query Service API (DuckDB) + static file serving +/appsindex pagedataapp.py— ECharts/Pico CSS dashboard generator (bar, line, pie, scatter, heatmap, table)schema.py— Keboola Developer Portal schema discoverymigrate.py— Push local tables/configs to a Keboola platform projectMCP tools (
tools.py, registered only when--local-backendis passed)write_table,query_data,get_tables,delete_tablefind_component_id,get_component_schema,setup_component,run_component,run_saved_configsave_config,get_configs,delete_configcreate_data_app,run_data_app,stop_data_app,list_data_apps,delete_data_appget_project_info,search,migrate_to_keboolaOAuth authorization support
authorizationfield onComponentConfigand all run tools{"oauth_api": {"credentials": {...}}}intoconfig.json— enables Google Analytics, Sheets, Drive, etc.Security hardening
config_id(rejects/,\, leading.)memory_limitandnetwork(YAML injection prevention)appserver.py: binds to127.0.0.1; DuckDBenable_external_access=false;html.escape()on index page valuesStructural separation
tools/local/→local_backend/— clearly isolated from the main tools packageserver.pyCompatibility
list_tools(run_middleware=False))uv.lockregenerated with duckdb 1.5.2Testing
Streamable-HTTPtransports)tests/local_backend/— all passtest_json_loggingexcluded — pre-existing failure onmainunrelated to this PR)Optional testing
canary-orionMCP (Streamable-HTTP)canary-orioncanary-orionChecklist
🤖 Generated with Claude Code