diff --git a/README.md b/README.md index bd1dff423..b61b62e50 100644 --- a/README.md +++ b/README.md @@ -50,12 +50,7 @@ Then, in your AI assistant: That's it. You get **three files**: -``` -graphify-out/ -├── graph.html open in any browser — click nodes, filter, search -├── GRAPH_REPORT.md the highlights: key concepts, surprising connections, suggested questions -└── graph.json the full graph — query it anytime without re-reading your files -``` +This is the MaluDb fork of [Graphify-Labs/graphify](https://github.com/Graphify-Labs/graphify). Upstream graphify maps a project — code, docs, PDFs — into a queryable knowledge graph (`graphify-out/graph.json`) using tree-sitter AST extraction, with zero LLM cost for code. This fork adds a MaluDb export path on top of that: the graph is persisted into MaluDb as subjects and statements, and application code is linked to your database data model, so agents using MaluDb memory can answer questions like *"which files write to `orders`?"* directly from memory. **Works in** Claude Code, Cursor, Codex, Gemini CLI, GitHub Copilot, and 15+ more — [pick your platform](#install). @@ -108,7 +103,7 @@ What you get out of the box: --- -## Benchmarks +## How the graph maps to MaluDb | Benchmark | Metric | graphify | Field | |---|---|---|---| @@ -117,87 +112,56 @@ What you get out of the box: | LongMemEval-S (n=50) | QA accuracy | **76%** | tied with dense RAG | | Graph build | LLM credits | **0** | per-token for most systems | -Every system ran on the same harness with the same model and budgets, scored by a judge blind-validated against a second judge (90.6% agreement, Cohen's kappa 0.81). Full per-system tables, the code-intelligence result, and reproduction commands: **[BENCHMARKS.md](./BENCHMARKS.md)**. +| graphify | MaluDb | +|---|---| +| Node | Subject with canonical name `/`; the node label becomes an alias | +| Edge | Statement (subject–verb–object); the edge relation is the verb | +| Confidence `EXTRACTED` / `INFERRED` / `AMBIGUOUS` | Statement confidence `1.0` / `0.7` / `0.4` | +| Community assignment | Carried on the node payload | ---- +The server upserts idempotently — re-pushing the same graph updates rather than duplicates, so it is safe to re-export after every rebuild. ## Prerequisites -| Requirement | Minimum | Check | Install | -|---|---|---|---| -| Python | 3.10+ | `python --version` | [python.org](https://www.python.org/downloads/) | -| uv *(recommended)* | any | `uv --version` | `curl -LsSf https://astral.sh/uv/install.sh \| sh` | -| pipx *(alternative)* | any | `pipx --version` | `pip install pipx` | +- **Python 3.10+** and [uv](https://docs.astral.sh/uv/) (recommended) or pipx/pip +- **A running MaluDb API server** and a bearer token (mint one via the server's `/v1/tokens` endpoint) +- For `--postgres` introspection: network access to the target PostgreSQL database -**macOS quick install (Homebrew):** -```bash -brew install python@3.12 uv -``` +## Installation -**Windows quick install:** -```powershell -winget install astral-sh.uv -``` +If `uv` is not installed yet (`uv --version` fails), install it first and open a new shell: -**Ubuntu/Debian:** ```bash -sudo apt install python3.12 python3-pip pipx -# or install uv: curl -LsSf https://astral.sh/uv/install.sh | sh ``` ---- - -## Install - -> **Official package:** The PyPI package is `graphifyy` (double-y). Other `graphify*` packages on PyPI are not affiliated. The CLI command is still `graphify`. - -**Step 1 — install the package:** - -```bash -# Recommended (isolated env; if 'graphify' isn't found after, run: uv tool update-shell): -uv tool install graphifyy - -# Alternatives: -pipx install graphifyy -pip install graphifyy # may need PATH setup — see note below -``` - -**Step 2 — register the skill with your AI assistant:** +Install from this fork (the PyPI package `graphifyy` is upstream and does not include the MaluDb exporter): ```bash -graphify install -``` - -That's it. Open your AI assistant and type `/graphify .` +# From a clone (recommended for development): +git clone https://github.com/maludb/graphify.git +cd graphify +uv tool install . -To install the assistant skill into the current repository instead of your user -profile, add `--project`: +# With the PostgreSQL introspection and SQL-parsing extras: +uv tool install ".[postgres,sql]" -```bash -graphify install --project -graphify install --project --platform codex +# Or directly from GitHub without cloning: +uv tool install "graphifyy[postgres,sql] @ git+https://github.com/maludb/graphify.git" ``` -Project-scoped installs write under the current directory, for example -`.claude/skills/graphify/SKILL.md` or `.agents/skills/graphify/SKILL.md` (plus a -`references/` sidecar the skill loads on demand), and -print a `git add` hint for files that can be committed. -Per-platform commands that support project-scoped installs accept the same flag, -for example `graphify claude install --project` or `graphify codex install --project`. - -> **PowerShell note:** Use `graphify .` not `/graphify .` — the leading slash is a path separator in PowerShell. +The CLI command is `graphify`. If the command is not found after install, run `uv tool update-shell` and open a new terminal. -> **`graphify: command not found`?** `uv tool install` / `pipx install` put the `graphify` command in their tool bin dir (`~/.local/bin`). If your shell can't find it right after install — common on a fresh macOS + zsh setup — that dir isn't on your `PATH` yet: run `uv tool update-shell` (or `pipx ensurepath`), then open a new terminal. With plain `pip`, add `~/.local/bin` (Linux) or `~/Library/Python/3.x/bin` (Mac) to your PATH, or run `python -m graphify`. +The base install needs **no API keys and no extras** for code-only extraction — AST parsing runs fully offline. The two extras used by this fork: -> **Running with `uvx` / `uv tool run` instead of installing?** Name the package, not the command: `uvx --from graphifyy graphify install`. Plain `uvx graphify …` fails (`No solution found … no versions of graphify`) because `uv tool run` reads the first word as a *package*, and the package is `graphifyy` — the `graphify` command lives inside it. - -> **Avoid `pip install` on Mac/Windows** if possible. The skill resolves Python at runtime from `graphify-out/.graphify_python`; if that points to a different environment than where `pip` installed the package, you'll get `ModuleNotFoundError: No module named 'graphify'`. `uv tool install` and `pipx install` isolate the package in their own env and avoid this entirely. +| Extra | What it adds | +|---|---| +| `postgres` | `graphify extract --postgres ` live schema introspection (`psycopg`) | +| `sql` | tree-sitter-sql parsing for `.sql` files and SQL-statement analysis | -> **Git hooks and uv tool / pipx:** `graphify hook install` embeds the current interpreter path directly into the hook scripts at install time, so the post-commit hook fires correctly even in GUI git clients and CI runners where `~/.local/bin` is not on PATH. If you reinstall or upgrade graphify, re-run `graphify hook install` to refresh the embedded path. +## Configuration -
-Pick your platform (20+ assistants, click to expand) +The MaluDb exporter reads two environment variables so credentials stay off the command line (and out of `ps` output / shell history): | Platform | Install command | |----------|----------------| @@ -361,129 +325,54 @@ You can also set `GRAPHIFY_GOOGLE_WORKSPACE=1`. Graphify exports shortcuts into ## Common commands ```bash -/graphify . # build graph for current folder -/graphify ./docs --update # re-extract only changed files -/graphify . --cluster-only # rerun clustering without re-extracting -/graphify . --cluster-only --resolution 1.5 # more granular communities -/graphify . --cluster-only --exclude-hubs 99 # suppress utility super-hubs from god-node rankings -/graphify . --no-viz # skip the HTML, just the report + JSON -/graphify . --wiki # build a markdown wiki from the graph -graphify export callflow-html # Mermaid architecture/call-flow HTML (auto-regenerates on every git commit if hook is installed) - -/graphify query "what connects auth to the database?" -/graphify path "UserService" "DatabasePool" -/graphify explain "RateLimiter" - -/graphify add https://arxiv.org/abs/1706.03762 # fetch a paper and add it -/graphify add # transcribe and add a video - -graphify hook install # auto-rebuild on git commit -graphify merge-graphs a.json b.json # combine two graphs - -graphify prs # PR dashboard: CI state, review status, worktree mapping -graphify prs 42 # deep dive on PR #42 with graph impact -graphify prs --triage # AI ranks your review queue (uses whatever backend is configured) -graphify prs --conflicts # PRs sharing graph communities — merge-order risk +export MALUDB_URL="http://localhost:8000" +export MALUDB_TOKEN="" ``` -See the [full command reference](#full-command-reference) below. - ---- - -## Ignoring files - -Create a `.graphifyignore` in your project root — same syntax as `.gitignore`, including `!` negation. - -**`.gitignore` is respected automatically.** graphify reads the `.gitignore` in each directory. If a `.graphifyignore` is also present, the two are **merged** — `.graphifyignore` patterns are evaluated last, so they win on conflicts (including `!` negations). Adding a `.graphifyignore` only ever excludes more; it never re-includes a file your `.gitignore` already excluded. Subdirectory scoping works the same way as git — an ignore file only affects its own subtree. - -``` -# .graphifyignore -node_modules/ -dist/ -*.generated.py - -# only index src/, ignore everything else -* -!src/ -!src/** -``` +### `graphify export maludb` flags ---- +| Flag | Default | Purpose | +|---|---|---| +| `--push URL` | `$MALUDB_URL` | MaluDb API server base URL (http/https) | +| `--token T` | `$MALUDB_TOKEN` | Bearer token | +| `--namespace NS` | project directory name | Namespace prefix for all subjects created by this import | +| `--graph PATH` | `graphify-out/graph.json` | Graph file to push | +| `--sql-usage` | off | Mine SQL table references from source files and push them as extra cross-namespace links | +| `--source-root DIR` | the graph's project directory | Root the graph's `source_file` paths are resolved against for SQL scanning | +| `--db-schema S` | `public` | Schema assumed for unqualified table names found in SQL | +| `--datamodel-ns NS` | `datamodel` | Namespace of the data-model graph the mined links point at | -## Team setup +## Quick start -`graphify-out/` is meant to be committed to git so everyone on the team starts with a map. +### 1. Build a code graph -**Recommended `.gitignore` additions:** -``` -graphify-out/cost.json # local only -# graphify-out/cache/ # optional: commit for speed, skip to keep repo small +```bash +cd ~/my-app +graphify extract . # AST-only, offline, no API key needed ``` -> `manifest.json` is now portable — keys are stored as relative paths and re-anchored on load, so committing it is safe and avoids a full rebuild on first checkout. +This writes `graphify-out/graph.json` (plus `graph.html` and `GRAPH_REPORT.md`). Inside Claude Code you can run `/graphify .` instead — same output. -**Workflow:** -1. One person runs `/graphify .` and commits `graphify-out/`. -2. Everyone pulls — their assistant reads the graph immediately. -3. Run `graphify hook install` to auto-rebuild after each commit (AST only, no API cost). This also sets up a git merge driver so `graph.json` is never left with conflict markers — two devs committing in parallel get their graphs union-merged automatically. -4. When docs or papers change, run `/graphify --update` to refresh those nodes. - ---- - -## Using the graph directly +### 2. Push it to MaluDb ```bash -# query the graph from the terminal -graphify query "show the auth flow" -graphify query "what connects DigestAuth to Response?" --graph graphify-out/graph.json - -# expose the graph as an MCP server (for repeated tool-call access) -python -m graphify.serve graphify-out/graph.json -python -m graphify.serve --graph graphify-out/graph.json # --graph flag also accepted - -# register with Kimi Code: -kimi mcp add --transport stdio graphify -- python -m graphify.serve graphify-out/graph.json - -# or serve over HTTP so a whole team points at one URL (no local graphify needed): -python -m graphify.serve graphify-out/graph.json --transport http --port 8080 -python -m graphify.serve graphify-out/graph.json --transport http --host 0.0.0.0 --api-key "$SECRET" +graphify export maludb +# Pushed to MaluDb (http://localhost:8000, namespace 'my-app'): +# 1842 nodes (1842 new, 0 updated), 5210 edges (5210 new) ``` -The MCP server gives your assistant structured access: `query_graph`, `get_node`, `get_neighbors`, `shortest_path`, `list_prs`, `get_pr_impact`, `triage_prs`. - -### Shared HTTP server - -`--transport stdio` (the default) spawns one local server per developer. `--transport http` serves the same tools over the MCP Streamable HTTP transport, so a single shared process can serve the graph for the whole team — clients point their IDE MCP config at `http://:8080/mcp` instead of running graphify locally. +The namespace defaults to the project directory name; pass `--namespace` to control it. -| Flag | Default | Purpose | -|---|---|---| -| `--transport {stdio,http}` | `stdio` | Transport to serve on | -| `--host` | `127.0.0.1` | HTTP bind host (use `0.0.0.0` to expose beyond localhost) | -| `--port` | `8080` | HTTP bind port | -| `--api-key` | env `GRAPHIFY_API_KEY` | Require `Authorization: Bearer ` (or `X-API-Key`) | -| `--path` | `/mcp` | HTTP mount path | -| `--json-response` | off | Return plain JSON instead of SSE streams | -| `--stateless` | off | No per-session state (for load-balanced / CI deployments) | -| `--session-timeout` | `3600` | Reap idle stateful sessions after N seconds (`0` disables) | +### 3. Push your data model -The default `127.0.0.1` bind is loopback-only. Set `--host 0.0.0.0` **and** `--api-key` together when exposing on a shared host. Run it in a container: +Build a graph of the live database schema and push it under the `datamodel` namespace: ```bash -docker build -t graphify . -docker run -p 8080:8080 -v "$(pwd)/graphify-out:/data" graphify \ - /data/graph.json --transport http --host 0.0.0.0 --api-key "$SECRET" +graphify extract --postgres "postgresql://user:pass@host/db" +graphify export maludb --namespace datamodel ``` -> **WSL / Linux note:** Ubuntu ships `python3`, not `python`. Use a venv to avoid conflicts: -> ```bash -> python3 -m venv .venv && .venv/bin/pip install "graphifyy[mcp]" -> ``` - ---- - -## Environment variables - -These are only needed for **headless / CI extraction** (`graphify extract`). When running via the `/graphify` skill inside your IDE, the model API is provided by your IDE session — no extra keys needed. +Every table becomes a typed `table` node carrying real catalog metadata — columns with types and nullability, primary-key columns, and unique constraints (including unique indexes not declared as constraints). Foreign keys become edges between tables. If you keep DDL in the repo instead, plain `graphify extract` on a directory containing `.sql` files produces the same typed nodes from the `CREATE TABLE` statements. | Variable | Used for | When required | |---|---|---| @@ -546,222 +435,38 @@ The PyPI package is `graphifyy`; `graphify` is only the command it provides. `uv **`python -m graphify` works but `graphify` command doesn't** Your shell's `PATH` doesn't include the bin directory the command was installed to. Prefer `uv tool install` / `pipx install` over plain `pip`, then run `uv tool update-shell` / `pipx ensurepath` and open a new terminal (see the install notes above). -**`/graphify .` causes "path not recognized" in PowerShell** -PowerShell treats a leading `/` as a path separator. Use `graphify .` (no slash) on Windows. +Re-push the code graph with SQL-usage mining enabled: -**Graph has fewer nodes after `--update` or rebuild** -If a refactor deleted files, the old nodes linger. Pass `--force` (or set `GRAPHIFY_FORCE=1`) to overwrite even when the rebuild has fewer nodes. - -**Graph has duplicate nodes for the same entity (ghost duplicates)** -Ghost duplicates (same symbol appearing twice — once from AST extraction with a source location, once from semantic extraction without) are now automatically merged at build time. If you see this in a graph built before v0.8.33, run a full re-extract to clean up: ```bash -graphify extract . --force +cd ~/my-app +graphify export maludb --sql-usage +# sql-usage: 317 table-reference links mined from /home/me/my-app +# Pushed to MaluDb (...): 1842 nodes (0 new, 1842 updated), 5527 edges (317 new) ``` -**Ollama runs out of VRAM / context window exceeded** -The KV-cache window is auto-sized but may be too large for your GPU. Reduce it: -```bash -GRAPHIFY_OLLAMA_NUM_CTX=8192 graphify extract ./docs --backend ollama --token-budget 4000 -``` - -**`LLM returned invalid JSON` / `Unterminated string` warnings** -The model's JSON response hit its output-token limit and was cut off mid-string. graphify auto-recovers (it splits the chunk and re-extracts the halves, and an oversized single document is first sliced at heading/paragraph boundaries so the whole file is still covered), so these warnings are noisy but not data loss. To reduce the churn, raise the output cap or shrink each chunk's output: -```bash -GRAPHIFY_MAX_OUTPUT_TOKENS=16384 graphify extract . --mode deep # lift the cap -graphify extract . --mode deep --token-budget 4000 # smaller input chunks -> smaller output -``` -With a cloud gateway like OpenRouter, prefer `--backend openai` (set `OPENAI_BASE_URL`) over the Ollama shim — it's a cleaner OpenAI-compatible path. If the model has its own max-output ceiling, lowering `--token-budget` is the reliable lever. - -**Graph HTML is too large to open in a browser (>5000 nodes)** -Skip HTML generation and use the JSON directly: -```bash -graphify cluster-only ./my-project --no-viz -graphify query "..." -``` +The miner scans each source file that appears in the graph (Python, PHP, TypeScript/JavaScript, Rust, Go, Ruby, Java, Kotlin, C#, C/C++, Perl, and `.sql`) for `INSERT INTO` / `UPDATE … SET` / `DELETE FROM` (→ `writes` links) and `FROM` / `JOIN` (→ `reads` links). Each hit becomes a link from the file's node to `datamodel/.`, pushed with the server's `resolve_external` option: targets that exist in the tenant's data-model graph resolve into real edges; unknown targets land in the import report's `skipped` list rather than failing the import. -**`graph.json` has conflict markers after two devs commit at once** -Run `graphify hook install` — it sets up a git merge driver that union-merges `graph.json` automatically so conflicts never happen. +The mining is deliberately regex-based — the goal is `INFERRED`-confidence "this file touches that table" edges, not full query understanding. System catalogs (`pg_catalog`, `information_schema`, `pg_*`), SQL set-returning functions, and interpolated/placeholder names are filtered out. -**Extraction returns empty nodes/edges for docs or PDFs** -Docs, PDFs, and images require an LLM call — code-only corpora need no key. Check that your API key is set and the backend is correct: -```bash -ANTHROPIC_API_KEY=sk-... graphify extract ./docs --backend claude -``` +## Typical workflow -**Skill version mismatch warning in your IDE** -Your installed graphify version is different from the skill file. Update: ```bash -uv tool upgrade graphifyy -graphify install # overwrites the skill file -``` +# one-time +export MALUDB_URL=... MALUDB_TOKEN=... +graphify extract --postgres "$DATABASE_URL" # data-model graph +graphify export maludb --namespace datamodel -**Claude Code prompt cache invalidated after every `graphify extract`** -Graphify writes output files (`graph.json`, `graphify-out/`) into the workspace. If those paths aren't ignored, every write invalidates Claude Code's prompt cache, forcing a full re-upload at cache-write rates on the next turn. Add them to `.claudeignore`: -```text -# .claudeignore -graph.json -graphify-out/ -``` - ---- - -## Full command reference +# per repository, repeat after significant changes +cd ~/repos/my-app +graphify extract . +graphify export maludb --sql-usage -``` -/graphify # run on current directory -/graphify ./raw # run on a specific folder -/graphify ./raw --mode deep # more aggressive relationship extraction -/graphify ./raw --update # re-extract only changed files -/graphify ./raw --directed # preserve edge direction -/graphify ./raw --cluster-only # rerun clustering on existing graph -/graphify ./raw --no-viz # skip HTML visualization -/graphify ./raw --obsidian # generate Obsidian vault -/graphify ./raw --obsidian --obsidian-dir ~/vault # write into an existing vault (never overwrites your own notes or .obsidian config) -/graphify ./raw --wiki # build agent-crawlable markdown wiki -/graphify ./raw --svg # export graph.svg -/graphify ./raw --graphml # export for Gephi / yEd -/graphify ./raw --neo4j # generate cypher.txt for Neo4j -/graphify ./raw --neo4j-push bolt://localhost:7687 -/graphify ./raw --falkordb # generate cypher.txt for FalkorDB -/graphify ./raw --falkordb-push falkordb://localhost:6379 -/graphify ./raw --watch # auto-sync as files change -/graphify ./raw --mcp # start MCP stdio server - -/graphify add https://arxiv.org/abs/1706.03762 -/graphify add -/graphify add https://... --author "Name" --contributor "Name" - -/graphify query "what connects attention to the optimizer?" -/graphify query "..." --dfs --budget 1500 -/graphify path "DigestAuth" "Response" -/graphify explain "SwinTransformer" - -graphify save-result --question "Q" --answer "A" --nodes Foo Bar --outcome useful # record how a Q&A turned out (work memory; outcome ∈ useful|dead_end|corrected) -graphify reflect # aggregate graphify-out/memory/ outcomes into reflections/LESSONS.md -graphify reflect --if-stale # no-op when LESSONS.md is already newer than every input (cheap to run each session) -graphify reflect --out docs/LESSONS.md # write the lessons doc somewhere else -graphify reflect --graph graphify-out/graph.json # group lessons by community + write the work-memory overlay (.graphify_learning.json) - # the overlay tags nodes preferred/tentative/contested (recency-weighted, with provenance); - # graphify explain / query then show a "Lesson:" hint, flagged "code changed — re-verify" when the source moved on - -graphify uninstall # remove from all platforms in one shot -graphify uninstall --purge # also delete graphify-out/ -graphify uninstall --project --platform codex # remove project-scoped install files only - -graphify hook install # post-commit + post-checkout hooks -graphify hook uninstall -graphify hook status - -# always-on assistant instructions - platform-specific -graphify claude install # CLAUDE.md + PreToolUse hook (Claude Code) -graphify claude uninstall -graphify codebuddy install # CODEBUDDY.md + PreToolUse hook (CodeBuddy) -graphify codebuddy uninstall -graphify codex install # AGENTS.md + PreToolUse hook in .codex/hooks.json (Codex) -graphify opencode install # AGENTS.md + tool.execute.before plugin (OpenCode) -graphify kilo install # native Kilo skill + /graphify command + AGENTS.md + .kilo plugin -graphify kilo uninstall -graphify cursor install # .cursor/rules/graphify.mdc (Cursor) -graphify cursor uninstall -graphify gemini install # GEMINI.md + BeforeTool hook (Gemini CLI) -graphify gemini uninstall -graphify copilot install # skill file (GitHub Copilot CLI) -graphify copilot uninstall -graphify aider install # AGENTS.md (Aider) -graphify aider uninstall -graphify claw install # AGENTS.md (OpenClaw) -graphify claw uninstall -graphify droid install # AGENTS.md (Factory Droid) -graphify droid uninstall -graphify trae install # AGENTS.md (Trae) -graphify trae uninstall -graphify trae-cn install # AGENTS.md (Trae CN) -graphify trae-cn uninstall -graphify hermes install # AGENTS.md + ~/.hermes/skills/ (Hermes) -graphify hermes uninstall -graphify amp install # skill file (Amp) -graphify amp uninstall -graphify agents install # ~/.agents/skills/ + AGENTS.md (cross-framework; alias: graphify skills) -graphify agents uninstall -graphify kiro install # .kiro/skills/ + .kiro/steering/graphify.md (Kiro IDE/CLI) -graphify kiro uninstall -graphify pi install # skill file (Pi coding agent) -graphify pi uninstall -graphify devin install # skill file + .windsurf/rules/graphify.md (Devin CLI) -graphify devin uninstall -graphify antigravity install # .agents/rules + .agents/workflows (Google Antigravity) -graphify antigravity uninstall - -graphify extract ./docs # headless LLM extraction for CI (no IDE needed) -graphify extract ./docs --backend gemini # explicit backend: gemini, kimi, claude, openai, deepseek, ollama, bedrock, or claude-cli -graphify extract ./docs --backend gemini --model gemini-3.1-pro-preview -graphify extract ./docs --backend ollama # local Ollama (set OLLAMA_BASE_URL / OLLAMA_MODEL) - no API key needed for loopback -OPENAI_BASE_URL=http://localhost:8080/v1 OPENAI_MODEL=my-model graphify extract ./docs --backend openai # any OpenAI-compatible server (llama.cpp, vLLM, LM Studio) -ANTHROPIC_BASE_URL=http://localhost:4000 ANTHROPIC_MODEL=my-model graphify extract ./docs --backend claude # any Anthropic-compatible endpoint (LiteLLM proxy, gateways) -GRAPHIFY_OLLAMA_NUM_CTX=32768 graphify extract ./docs --backend ollama # override KV-cache window (auto-sized by default) -GRAPHIFY_OLLAMA_KEEP_ALIVE=0 graphify extract ./docs --backend ollama # unload model after each chunk (saves VRAM on small GPUs) -graphify extract ./docs --backend bedrock # AWS Bedrock via IAM - no API key, uses AWS credential chain -graphify extract ./docs --backend claude-cli # route through Claude Code CLI - no API key, uses your Claude subscription -graphify extract ./docs --backend azure # Azure OpenAI (set AZURE_OPENAI_API_KEY + AZURE_OPENAI_ENDPOINT) -graphify extract ./docs --max-workers 16 # AST parallelism (also GRAPHIFY_MAX_WORKERS) -graphify extract --postgres "postgresql://user:pass@host/db" # introspect live PostgreSQL schema directly -graphify extract ./my-workspace --cargo # introspect Rust Cargo workspace dependencies directly -graphify extract ./docs --token-budget 30000 # smaller semantic chunks for local/small models -graphify extract ./docs --max-concurrency 2 # fewer parallel LLM calls (useful for local inference) -graphify extract ./docs --api-timeout 900 # longer HTTP timeout for slow local models (default 600s) -graphify extract ./docs --google-workspace # export .gdoc/.gsheet/.gslides via gws before extraction -graphify extract ./docs --mode deep # richer semantic extraction via extended system prompt -graphify extract ./docs --no-cluster # raw extraction only, skip clustering -graphify extract ./docs --timing # print per-stage wall-clock timings to stderr (also works on cluster-only) -graphify extract ./docs --force # overwrite graph.json even if new graph has fewer nodes (use after refactors or to clear ghost duplicates) -graphify extract ./docs --dedup-llm # LLM tiebreaker for ambiguous entity pairs (uses same API key) -graphify extract ./docs --global --as myrepo # extract and register into the cross-project global graph -GRAPHIFY_MAX_OUTPUT_TOKENS=32768 graphify extract ./docs --backend claude # raise output cap for dense corpora - -graphify export callflow-html # graphify-out/-callflow.html -graphify export callflow-html --max-sections 8 # cap generated architecture sections -graphify export callflow-html --output docs/arch.html -graphify export callflow-html ./some-repo/graphify-out - -graphify global add graphify-out/graph.json --as myrepo # register a project graph into ~/.graphify/global-graph.json -graphify global remove myrepo # remove a project from the global graph -graphify global list # show all registered repos + node/edge counts -graphify global path # print path to the global graph file - -graphify prs # PR dashboard: CI, review, worktree, graph impact -graphify prs 42 # deep dive on PR #42 -graphify prs --triage # AI triage ranking (auto-detects backend from env) -graphify prs --worktrees # worktree → branch → PR mapping -graphify prs --conflicts # PRs sharing graph communities (merge-order risk) -graphify prs --base main # filter to PRs targeting a specific base branch -graphify prs --repo owner/repo # run against a different GitHub repo -GRAPHIFY_TRIAGE_BACKEND=kimi graphify prs --triage # use a specific backend for triage - -graphify clone https://github.com/karpathy/nanoGPT -graphify merge-graphs a.json b.json --out merged.json -graphify --version # print installed version -graphify watch ./src -graphify check-update ./src -graphify update ./src -graphify update ./src --no-cluster # skip reclustering, write raw AST graph only -graphify update ./src --force # overwrite even if new graph has fewer nodes -graphify cluster-only ./my-project -graphify cluster-only ./my-project --graph path/to/graph.json # custom graph location -graphify cluster-only ./my-project --max-concurrency 16 --batch-size 200 # parallel community labeling (large graphs) -graphify cluster-only ./my-project --resolution 1.5 # more, smaller communities -graphify cluster-only ./my-project --exclude-hubs 99 # exclude p99 degree nodes from partitioning -graphify cluster-only ./my-project --no-label # keep "Community N" placeholders -graphify cluster-only ./my-project --backend=gemini # backend for community naming -graphify cluster-only ./my-project --backend=gemini --model gemini-2.5-pro # specific model -graphify label ./my-project # (re)name communities with the configured backend -graphify label ./my-project --backend=openai --model gpt-4o # force a specific backend and model +# then query the graph locally... +graphify query "what writes to the orders table?" +# ...or let any MaluDb-connected agent answer from memory ``` -> **Community names:** inside an agent (Claude Code, Gemini CLI) the agent names communities itself. When you run the bare CLI, `cluster-only` auto-names them with the configured backend (built-in or custom OpenAI-compatible provider) — pass `--no-label` to keep `Community N`, or run `graphify label` to (re)generate names on demand. - ---- - -## Learn more +`graphify hook install` (upstream feature) rebuilds the graph on every git commit; re-run the export afterwards to keep MaluDb in sync. - [How it works](docs/how-it-works.md) — the extraction pipeline, community detection, confidence scoring, benchmarks - [ARCHITECTURE.md](ARCHITECTURE.md) — module breakdown, how to add a language @@ -772,66 +477,53 @@ graphify label ./my-project --backend=openai --model gpt-4o # force a specific ## Built on graphify: Penpax -[**Penpax**](https://graphifylabs.ai) is the always-on layer built on top of graphify — it applies the same graph approach to your entire working life: meetings, browser history, emails, files, and code, updating continuously in the background. +**`error: --push URL (or MALUDB_URL) required`** — set `MALUDB_URL` or pass `--push http://host:port`. Only `http`/`https` schemes are accepted. -Built for people whose work lives across hundreds of conversations and documents they can never fully reconstruct. No cloud, fully on-device. +**`error: --token (or MALUDB_TOKEN) required`** — mint a bearer token on the MaluDb server (`/v1/tokens`) and export it as `MALUDB_TOKEN`. -**Free trial launching soon.** [Join the waitlist →](https://graphifylabs.ai) +**`MaluDb import failed: HTTP 401/403`** — the token is invalid or lacks write access for the tenant. ---- +**`--sql-usage` mined 0 links** — the scanner resolves the graph's `source_file` paths against `--source-root` (default: the graph's project directory, i.e. the parent of `graphify-out/`). If you moved `graph.json` or run from elsewhere, pass `--source-root` explicitly. -
-Contributing +**Mined links all land in `skipped`** — the targets are `/.
`; make sure the data-model graph was pushed under the same namespace (`--datamodel-ns`, default `datamodel`) and that unqualified names match your schema (`--db-schema`, default `public`). -### Development setup +**`--postgres` fails to import psycopg** — install the extra: `uv tool install ".[postgres]"`. -The project uses [uv](https://docs.astral.sh/uv/) for dev workflow. Install it once, then: +## Development ```bash -git clone https://github.com/safishamsi/graphify.git +git clone https://github.com/maludb/graphify.git cd graphify -git checkout v8 # active development branch +uv sync --all-extras # venv + all extras + dev group (pytest, ruff, pyright) -# Create the project venv and install graphify + all extras + the dev group -# (pytest). uv installs the dev dependency group by default; pass --no-dev to -# skip it. -uv sync --all-extras +uv run pytest tests/ -q # full suite +uv run pytest tests/test_export.py -q # MaluDb exporter tests +uv run pytest tests/test_pg_introspect.py -q # PostgreSQL introspection tests ``` -Verify the editable install: -```bash -uv run graphify --version -uv run python -c "import graphify; print(graphify.__file__)" -``` +Fork-specific code lives in: + +- `graphify/export.py` — `push_to_maludb()` (stdlib `urllib`, no extra dependency) +- `graphify/sql_usage.py` — SQL-usage mining (code file → table links) +- `graphify/pg_introspect.py` — live PostgreSQL schema introspection with catalog metadata +- `graphify/__main__.py` — the `graphify export maludb` CLI wiring -### Running tests +To pull upstream improvements: ```bash -uv run pytest tests/ -q # run the full suite -uv run pytest tests/test_extract.py -q # one module -uv run pytest tests/ -q -k "python" # filter by name +git remote add upstream https://github.com/Graphify-Labs/graphify +git fetch upstream && git merge upstream/v8 ``` -> macOS note: the test suite includes both `sample.f90` and `sample.F90` fixtures. These collide on case-insensitive HFS+ / APFS file systems. Run on Linux or in a Docker container if you need to test both Fortran variants simultaneously. - -### Git workflow - -- Active development happens on the `v8` branch. -- Commit style: `fix: ` / `feat: ` / `docs: ` -- Before opening a PR, run `uv run pytest tests/ -q` and confirm it passes. -- Add a fixture file to `tests/fixtures/` and tests to `tests/test_languages.py` for any new language extractor. +## Related MaluDb repositories -### What to contribute +- [maludb-core](https://github.com/maludb/maludb-core) — the memory database (PostgreSQL 17 extension) +- API servers: [maludb-lamp-api-server](https://github.com/maludb/maludb-lamp-api-server) (reference), [maludb-python-api-server](https://github.com/maludb/maludb-python-api-server), [maludb-fastify-api-server](https://github.com/maludb/maludb-fastify-api-server) — any of them accepts `graphify export maludb` +- [maludb-terminal](https://github.com/maludb/maludb-terminal) — Rust CLI + MCP server for querying the memory the graph lands in -**Worked examples** are the most useful contribution. Run `/graphify` on a real corpus, save the output to `worked/{slug}/`, write an honest `review.md` covering what the graph got right and wrong, and open a PR. +MaluDb is an open-source memory database supported by [Kinetic Seas Incorporated](https://kineticseas.com). -**Extraction bugs** — open an issue with the input file, the cache entry (`graphify-out/cache/`), and what was missed or wrong. - -See [ARCHITECTURE.md](ARCHITECTURE.md) for module responsibilities and how to add a language. - - - ---- +## License

diff --git a/graphify/__main__.py b/graphify/__main__.py index 285d19c1b..bd4061454 100644 --- a/graphify/__main__.py +++ b/graphify/__main__.py @@ -4065,7 +4065,7 @@ def _to_simple(g: "_nx.Graph") -> "_nx.Graph": elif cmd == "export": subcmd = sys.argv[2] if len(sys.argv) > 2 else "" - if subcmd not in ("html", "callflow-html", "obsidian", "wiki", "svg", "graphml", "neo4j", "falkordb"): + if subcmd not in ("html", "callflow-html", "obsidian", "wiki", "svg", "graphml", "neo4j", "falkordb", "maludb"): print("Usage: graphify export ", file=sys.stderr) print(" html [--graph PATH] [--labels PATH] [--node-limit N] [--no-viz]", file=sys.stderr) print(" callflow-html [GRAPH|DIR] [--graph PATH] [--labels PATH] [--report PATH] [--sections PATH] [--output HTML]", file=sys.stderr) @@ -4078,6 +4078,10 @@ def _to_simple(g: "_nx.Graph") -> "_nx.Graph": print(" (or set NEO4J_PASSWORD instead of --password to keep it off argv)", file=sys.stderr) print(" falkordb [--graph PATH] [--push URI] [--user U] [--password P]", file=sys.stderr) print(" (or set FALKORDB_PASSWORD instead of --password to keep it off argv)", file=sys.stderr) + print(" maludb [--graph PATH] [--push URL] [--namespace NS] [--token T]", file=sys.stderr) + print(" [--sql-usage --source-root DIR [--db-schema S] [--datamodel-ns NS]]", file=sys.stderr) + print(" (URL defaults to MALUDB_URL; set MALUDB_TOKEN instead of --token to keep it off argv;", file=sys.stderr) + print(" namespace defaults to the graph's project directory name)", file=sys.stderr) sys.exit(1) # Parse shared args @@ -4109,8 +4113,19 @@ def _to_simple(g: "_nx.Graph") -> "_nx.Graph": # NEO4J_PASSWORD otherwise. push_password: str | None = ( os.environ.get("FALKORDB_PASSWORD") if subcmd == "falkordb" + else os.environ.get("MALUDB_TOKEN") if subcmd == "maludb" else os.environ.get("NEO4J_PASSWORD") ) or None + # maludb-only settings: --push falls back to MALUDB_URL, --token is an + # alias for --password (same F-031 keep-it-off-argv rationale), and the + # namespace defaults to the graph's project directory name. + maludb_namespace: str | None = None + maludb_sql_usage = False + maludb_source_root: str | None = None + maludb_db_schema = "public" + maludb_datamodel_ns = "datamodel" + if subcmd == "maludb" and not push_uri: + push_uri = os.environ.get("MALUDB_URL") or None i = 0 while i < len(args): a = args[i] @@ -4166,6 +4181,18 @@ def _to_simple(g: "_nx.Graph") -> "_nx.Graph": push_user = args[i + 1]; i += 2 elif a == "--password" and i + 1 < len(args): push_password = args[i + 1]; i += 2 + elif a == "--token" and i + 1 < len(args): + push_password = args[i + 1]; i += 2 + elif a == "--namespace" and i + 1 < len(args): + maludb_namespace = args[i + 1]; i += 2 + elif a == "--sql-usage": + maludb_sql_usage = True; i += 1 + elif a == "--source-root" and i + 1 < len(args): + maludb_source_root = args[i + 1]; i += 2 + elif a == "--db-schema" and i + 1 < len(args): + maludb_db_schema = args[i + 1]; i += 2 + elif a == "--datamodel-ns" and i + 1 < len(args): + maludb_datamodel_ns = args[i + 1]; i += 2 elif subcmd == "callflow-html" and not a.startswith("-") and not graph_path_explicit: candidate = Path(a) if candidate.name == "graph.json" or candidate.suffix.lower() == ".json": @@ -4370,6 +4397,43 @@ def _to_simple(g: "_nx.Graph") -> "_nx.Graph": f"import), so load a graph with: graphify export falkordb --push " f"falkordb://localhost:6379") + elif subcmd == "maludb": + from graphify.export import push_to_maludb as _push_maludb + if not push_uri: + print("error: --push URL (or MALUDB_URL) required, e.g. " + "graphify export maludb --push https://api.example.com", file=sys.stderr) + sys.exit(1) + if not push_password: + print("error: --token (or MALUDB_TOKEN) required — a MaluDb bearer token", file=sys.stderr) + sys.exit(1) + namespace = maludb_namespace + if not namespace: + # graph.json lives in /graphify-out/, so the project + # directory is the graph dir's parent (cwd when defaulted). + project_dir = graph_path.resolve().parent.parent + namespace = re.sub(r"[^A-Za-z0-9._-]", "-", project_dir.name).strip("-") or "graphify" + extra_links = None + push_options = None + if maludb_sql_usage: + from graphify.sql_usage import mine_sql_usage + source_root = maludb_source_root or str(graph_path.resolve().parent.parent) + extra_links = mine_sql_usage( + G, source_root, + datamodel_ns=maludb_datamodel_ns, + default_schema=maludb_db_schema) + push_options = {"resolve_external": True} + print(f"sql-usage: {len(extra_links)} table-reference links mined from {source_root}") + result = _push_maludb(G, base_url=push_uri, token=push_password, + namespace=namespace, communities=communities, + extra_links=extra_links, options=push_options) + n = result.get("nodes", {}) + e = result.get("edges", {}) + n_skipped = len(result.get("skipped", [])) + print(f"Pushed to MaluDb ({push_uri}, namespace '{result.get('namespace', namespace)}'): " + f"{n.get('imported', 0)} nodes ({n.get('created', 0)} new, {n.get('resolved', 0)} updated), " + f"{e.get('imported', 0)} edges ({e.get('created', 0)} new)" + + (f", {n_skipped} skipped" if n_skipped else "")) + elif cmd == "benchmark": from graphify.benchmark import run_benchmark, print_benchmark diff --git a/graphify/export.py b/graphify/export.py index 36e4f9949..f58bf2528 100644 --- a/graphify/export.py +++ b/graphify/export.py @@ -1565,6 +1565,106 @@ def _safe_label(label: str) -> str: return {"nodes": nodes_pushed, "edges": edges_pushed} +def push_to_maludb( + G: nx.Graph, + base_url: str, + token: str, + namespace: str, + communities: dict[int, list[str]] | None = None, + timeout: int = 300, + extra_links: list[dict] | None = None, + options: dict | None = None, +) -> dict: + """Push graph to a MaluDb memory database via POST /v1/graph/import. + + MaluDb (https://github.com/maludb) persists the graph in PostgreSQL: + nodes become subjects (canonical name "/", label as + alias) and edges become subject-verb-object statements (the relation is + the verb; EXTRACTED/INFERRED/AMBIGUOUS confidence maps to 1.0/0.7/0.4). + The server upserts idempotently, so re-pushing the same graph updates + rather than duplicates — same contract as push_to_neo4j. + + No extra dependency: uses stdlib urllib against any MaluDb /v1 API + server. Auth is a bearer token (mint one via the server's /v1/tokens). + + Returns the server's import report: + {"namespace", "nodes": {...}, "edges": {...}, "verbs_created", + "chunks", "skipped"}. + """ + import urllib.error + import urllib.request + from urllib.parse import urlparse + + # Operator-supplied push target (like neo4j's bolt URI), not untrusted + # content — so only the scheme is enforced. security.validate_url would + # wrongly reject localhost / private-network MaluDb servers here. + if urlparse(base_url).scheme.lower() not in ("http", "https"): + raise ValueError(f"MaluDb base URL must be http(s), got {base_url!r}") + + node_community = _node_community_map(communities) if communities else {} + + nodes = [] + for node_id, data in G.nodes(data=True): + n: dict = {"id": node_id} + for key in ("label", "source_file", "source_location", "file_type"): + value = data.get(key) + if isinstance(value, (str, int, float, bool)): + n[key] = value + cid = node_community.get(node_id, data.get("community")) + if cid is not None: + n["community"] = cid + nodes.append(n) + + links = [] + for u, v, data in G.edges(data=True): + e: dict = { + "source": u, + "target": v, + "relation": data.get("relation") or "related_to", + } + confidence = data.get("confidence") + if confidence is not None: + e["confidence"] = confidence + links.append(e) + + try: + from importlib.metadata import version as _pkg_version + provenance = f"graphify-{_pkg_version('graphifyy')}" + except Exception: + provenance = "graphify" + + if extra_links: + links = links + list(extra_links) + + payload = { + "namespace": namespace, + "provenance": provenance, + "graph": {"nodes": nodes, "links": links}, + } + if options: + payload["options"] = dict(options) + + request = urllib.request.Request( + base_url.rstrip("/") + "/v1/graph/import", + data=json.dumps(payload).encode("utf-8"), + headers={ + "Authorization": f"Bearer {token}", + "Content-Type": "application/json", + }, + method="POST", + ) + try: + with urllib.request.urlopen(request, timeout=timeout) as response: # nosec B310 - validate_url enforces http(s) + return json.loads(response.read().decode("utf-8")) + except urllib.error.HTTPError as exc: + detail = exc.read().decode("utf-8", errors="replace")[:500] + raise RuntimeError( + f"MaluDb import failed: HTTP {exc.code} from {base_url}: {detail}" + ) from exc + except urllib.error.URLError as exc: + raise RuntimeError(f"MaluDb import failed: cannot reach {base_url}: {exc.reason}") from exc + + def to_graphml( G: nx.Graph, communities: dict[int, list[str]], diff --git a/graphify/extract.py b/graphify/extract.py index 652261ba6..8da1983ee 100644 --- a/graphify/extract.py +++ b/graphify/extract.py @@ -7597,11 +7597,17 @@ def _obj_name(n) -> str | None: return _read(c) return None - def _add_node(nid: str, label: str, line: int) -> None: + def _add_node(nid: str, label: str, line: int, *, node_type: str | None = None, + metadata: dict | None = None) -> None: if nid not in seen_ids: seen_ids.add(nid) - nodes.append({"id": nid, "label": label, "file_type": "code", - "source_file": str_path, "source_location": f"L{line}"}) + node = {"id": nid, "label": label, "file_type": "code", + "source_file": str_path, "source_location": f"L{line}"} + if node_type: + node["type"] = node_type + if metadata: + node["metadata"] = sanitize_metadata(metadata) + nodes.append(node) edges.append({"source": file_nid, "target": nid, "relation": "contains", "confidence": "EXTRACTED", "source_file": str_path, "source_location": f"L{line}", "weight": 1.0}) @@ -7611,6 +7617,80 @@ def _add_edge(src: str, tgt: str, relation: str, line: int) -> None: "confidence": "EXTRACTED", "source_file": str_path, "source_location": f"L{line}", "weight": 1.0}) + # Node types that follow a column name but are constraints, not the type. + _NON_TYPE_KINDS = { + "keyword_not", "keyword_null", "keyword_primary", "keyword_key", + "keyword_unique", "keyword_references", "keyword_default", + "keyword_check", "keyword_constraint", "keyword_auto_increment", + "object_reference", "ERROR", "comment", ",", + } + + def _constraint_columns(constraint) -> list[str]: + cols: list[str] = [] + for oc in constraint.children: + if oc.type == "ordered_columns": + for col in oc.children: + if col.type == "column": + for ci in col.children: + if ci.type == "identifier": + cols.append(_read(ci)) + return cols + + def _parse_column_metadata(col_defs) -> dict: + """Columns/pk/unique from a column_definitions subtree → table node metadata.""" + columns: list[dict] = [] + pk: list[str] = [] + unique: list[list[str]] = [] + for cd in col_defs.children: + if cd.type == "column_definition": + name: str | None = None + col_type: str | None = None + not_null = False + inline_pk = False + inline_unique = False + for cc in cd.children: + if name is None: + if cc.type == "identifier": + name = _read(cc) + elif col_type is None and cc.type not in _NON_TYPE_KINDS: + col_type = _read(cc) + elif cc.type == "keyword_not": + not_null = True + elif cc.type == "keyword_primary": + inline_pk = True + elif cc.type == "keyword_unique": + inline_unique = True + if name: + columns.append({"name": name, "type": (col_type or "").upper(), + "nullable": not (not_null or inline_pk)}) + if inline_pk: + pk.append(name) + if inline_unique: + unique.append([name]) + elif cd.type == "constraints": + for constraint in cd.children: + if constraint.type != "constraint": + continue + is_pk = any(c.type == "keyword_primary" for c in constraint.children) + is_unique = any(c.type == "keyword_unique" for c in constraint.children) + if not (is_pk or is_unique): + continue + cols = _constraint_columns(constraint) + if not cols: + continue + if is_pk: + pk.extend(c for c in cols if c not in pk) + else: + unique.append(cols) + meta: dict = {} + if columns: + meta["columns"] = columns + if pk: + meta["pk"] = pk + if unique: + meta["unique"] = unique + return meta + def walk(node) -> None: t = node.type line = node.start_point[0] + 1 @@ -7619,7 +7699,10 @@ def walk(node) -> None: name = _obj_name(node) if name: nid = _make_id(stem, name) - _add_node(nid, name, line) + col_defs = next((c for c in node.children + if c.type == "column_definitions"), None) + meta = _parse_column_metadata(col_defs) if col_defs is not None else {} + _add_node(nid, name, line, node_type="table", metadata=meta or None) table_nids[name.lower()] = nid # Foreign key REFERENCES for col in node.children: @@ -7674,7 +7757,7 @@ def walk(node) -> None: name = _obj_name(node) if name: nid = _make_id(stem, name) - _add_node(nid, name, line) + _add_node(nid, name, line, node_type="view") table_nids[name.lower()] = nid # FROM/JOIN table references inside view body _walk_from_refs(node, nid, line) @@ -7699,7 +7782,7 @@ def walk(node) -> None: src_nid = table_nids.get(name.lower()) if not src_nid: src_nid = _make_id(stem, name) - _add_node(src_nid, name, line) + _add_node(src_nid, name, line, node_type="table") table_nids[name.lower()] = src_nid for child in node.children: if child.type == "add_constraint": diff --git a/graphify/pg_introspect.py b/graphify/pg_introspect.py index 24567fa18..0868bb154 100644 --- a/graphify/pg_introspect.py +++ b/graphify/pg_introspect.py @@ -1,6 +1,7 @@ from __future__ import annotations from pathlib import Path, PurePosixPath from graphify.extract import extract_sql +from graphify.security import sanitize_metadata def _quote_ident(name: str) -> str: @@ -8,6 +9,34 @@ def _quote_ident(name: str) -> str: return '"' + name.replace('"', '""') + '"' +def _annotate_table_metadata(result: dict, columns, pks, uniques) -> None: + """Attach real catalog columns/pk/unique onto table nodes. + + The synthetic DDL renders every table as ``(id INT)``, so whatever column + metadata extract_sql parsed from it is a stub — drop it, then attach the + catalog truth keyed by the quoted '"schema"."name"' label the DDL produced. + """ + by_table: dict[tuple[str, str], dict] = {} + for schema, table, col, dtype, nullable in columns: + meta = by_table.setdefault((schema, table), {}) + meta.setdefault("columns", []).append( + {"name": col, "type": dtype, "nullable": nullable == "YES"}) + for schema, table, pk_cols in pks: + by_table.setdefault((schema, table), {})["pk"] = list(pk_cols) + for schema, table, cols in uniques: + by_table.setdefault((schema, table), {}).setdefault("unique", []).append(list(cols)) + + label_meta = {f"{_quote_ident(s)}.{_quote_ident(t)}": m + for (s, t), m in by_table.items()} + for node in result.get("nodes", []): + if node.get("type") != "table": + continue + node.pop("metadata", None) # stub from the '(id INT)' DDL + meta = label_meta.get(node.get("label", "")) + if meta: + node["metadata"] = sanitize_metadata(meta) + + def introspect_postgres(dsn: str | None = None) -> dict: """Connect to PostgreSQL, reconstruct DDL, and extract via extract_sql().""" try: @@ -87,6 +116,47 @@ def introspect_postgres(dsn: str | None = None) -> dict: ORDER BY kcu1.table_schema, kcu1.table_name; """) fks = cur.fetchall() + + # 5. Columns per table + cur.execute(""" + SELECT table_schema, table_name, column_name, data_type, is_nullable + FROM information_schema.columns + WHERE table_schema NOT IN ('pg_catalog', 'information_schema') + ORDER BY table_schema, table_name, ordinal_position; + """) + columns = cur.fetchall() + + # 6. Primary keys + cur.execute(""" + SELECT tc.table_schema, tc.table_name, + ARRAY_AGG(kcu.column_name ORDER BY kcu.ordinal_position) AS pk_columns + FROM information_schema.table_constraints AS tc + JOIN information_schema.key_column_usage AS kcu + ON tc.constraint_name = kcu.constraint_name + AND tc.table_schema = kcu.table_schema + WHERE tc.constraint_type = 'PRIMARY KEY' + AND tc.table_schema NOT IN ('pg_catalog', 'information_schema') + GROUP BY tc.table_schema, tc.table_name, tc.constraint_name; + """) + pks = cur.fetchall() + + # 7. Unique indexes via pg_catalog — catches unique indexes that are + # not declared as constraints. Partial/expression indexes are skipped + # (they cannot be FK targets). + cur.execute(""" + SELECT n.nspname, t.relname, + ARRAY_AGG(a.attname ORDER BY x.ordinality) AS cols + FROM pg_index i + JOIN pg_class t ON t.oid = i.indrelid + JOIN pg_namespace n ON n.oid = t.relnamespace + JOIN LATERAL unnest(i.indkey) WITH ORDINALITY AS x(attnum, ordinality) ON TRUE + JOIN pg_attribute a ON a.attrelid = t.oid AND a.attnum = x.attnum + WHERE i.indisunique AND NOT i.indisprimary + AND i.indpred IS NULL AND i.indexprs IS NULL + AND n.nspname NOT IN ('pg_catalog', 'information_schema') + GROUP BY n.nspname, t.relname, i.indexrelid; + """) + uniques = cur.fetchall() finally: conn.close() @@ -139,4 +209,5 @@ def introspect_postgres(dsn: str | None = None) -> dict: # Pass virtual path and in-memory DDL content to extract_sql result = extract_sql(virtual_path, content=ddl_string) + _annotate_table_metadata(result, columns, pks, uniques) return result \ No newline at end of file diff --git a/graphify/sql_evidence.py b/graphify/sql_evidence.py new file mode 100644 index 000000000..34b558d95 --- /dev/null +++ b/graphify/sql_evidence.py @@ -0,0 +1,267 @@ +"""Detect and parse SQL statements embedded in application-code string literals. + +Pure helpers with no graphify.extract dependency (import direction is one-way: +extract.py -> sql_evidence.py). Host-language extractors flatten a string +expression (replacing interpolations with :data:`PARAM_PLACEHOLDER`), then: + + looks_like_sql(text) -> cheap gate, no parsing + sanitize_sql(text) -> placeholder/backtick normalization + parse_sql_statement() -> {"op", "tables_read", "tables_written", "joins"} + +``joins`` are equality predicates between qualified columns of two different +tables (JOIN ON or WHERE), with aliases resolved — the raw evidence from which +a later corpus-wide pass infers undeclared foreign keys. Precision beats +recall throughout: a parse tree containing ERROR nodes is discarded rather +than guessed at, and any name touched by an interpolation placeholder is +dropped. + +Known v1 limitation: quoted mixed-case identifiers (``"Users"`` vs ``users``) +are conflated by the casefolding in :func:`normalize_table_key`. +""" +from __future__ import annotations + +import re + +# Placeholder spliced in for interpolated/concatenated fragments and bind +# parameters. Parses as a plain identifier in both expression and relation +# position, so surrounding structure survives. +PARAM_PLACEHOLDER = "__gfx_param__" + +_SQL_HEAD = re.compile(r"\s*(SELECT|INSERT|UPDATE|DELETE|REPLACE|WITH)\b", re.IGNORECASE) +_SQL_BODY = re.compile(r"\b(FROM|INTO|SET|JOIN|WHERE)\b", re.IGNORECASE) +# Punctuation that essentially never appears in prose that merely starts with +# "Select ... from ...": placeholders, comparisons, wildcards, quoting, calls. +_SQL_SIGNAL = re.compile(r"[*=?;()'\"`%]|:\w") + +_NAMED_PARAM = re.compile(r"(? bool: + """Cheap gate: does this string plausibly contain a SQL statement? + + Requires a leading DML keyword plus a second structural keyword after it. + Because prose like "Select an option from the menu" satisfies both (and + even parses as valid SQL), additionally require either SQL-ish punctuation + or upper-cased keywords as written. + """ + if len(text) < _MIN_SQL_LEN: + return False + head = _SQL_HEAD.match(text) + if not head: + return False + body = _SQL_BODY.search(text, head.end()) + if not body: + return False + if _SQL_SIGNAL.search(text): + return True + return head.group(1).isupper() and body.group(1).isupper() + + +def sanitize_sql(text: str) -> str: + """Normalize app-code SQL so tree-sitter-sql can parse it. + + Strips MySQL backticks and replaces bind-parameter syntax (``?``, ``:name``, + ``%s``) with :data:`PARAM_PLACEHOLDER`. A placeholder glued to the front of + an identifier (WordPress-style ``{$wpdb->prefix}posts``) collapses onto the + identifier; an identifier glued to the front of a placeholder stays tainted + (the real name is unknowable). + """ + text = text.replace("`", "") + text = text.replace("?", PARAM_PLACEHOLDER) + text = _NAMED_PARAM.sub(PARAM_PLACEHOLDER, text) + text = _FORMAT_PARAM.sub(PARAM_PLACEHOLDER, text) + text = _GLUED_PREFIX.sub("", text) + return text + + +def normalize_table_key(name: str) -> str: + """Canonical matching key for a table name across evidence sources. + + Strips quoting, drops a leading ``public.`` schema qualifier, casefolds. + """ + parts = [p.strip('"').strip("`") for p in name.split(".") if p] + if len(parts) > 1 and parts[0].casefold() == "public": + parts = parts[1:] + return ".".join(parts).casefold() + + +_parser = None +_parser_failed = False + + +def _get_parser(): + global _parser, _parser_failed + if _parser is None and not _parser_failed: + try: + import tree_sitter_sql as tssql + from tree_sitter import Language, Parser + _parser = Parser(Language(tssql.language())) + except Exception: + _parser_failed = True + return _parser + + +def _tainted(name: str) -> bool: + return PARAM_PLACEHOLDER in name + + +def parse_sql_statement(text: str) -> dict | None: + """Parse one sanitized SQL string into table/join facts. + + Returns ``{"op", "tables_read", "tables_written", "joins"}`` or ``None`` + when the text does not parse cleanly (any ERROR node) or yields no table + references. ``joins`` entries are ``{"left": [table, column], "right": + [table, column]}`` with the two sides ordered lexically so that + ``a.x = b.y`` and ``b.y = a.x`` produce identical records. + """ + parser = _get_parser() + if parser is None: + return None + try: + tree = parser.parse(text.encode("utf-8")) + except Exception: + return None + root = tree.root_node + if root.has_error: + return None + + src = text.encode("utf-8") + + def _read(n) -> str: + return src[n.start_byte:n.end_byte].decode("utf-8", errors="replace") + + op: str | None = None + reads: dict[str, str] = {} # key -> display name + writes: dict[str, str] = {} + alias_map: dict[str, str] = {} # casefolded alias/name -> display name + joins: list[dict] = [] + seen_joins: set[tuple] = set() + + def _relation_parts(rel) -> tuple[str | None, str | None]: + """(table, alias) from a ``relation`` node.""" + table = alias = None + for c in rel.children: + if c.type == "object_reference" and table is None: + table = _read(c) + elif c.type == "identifier": + alias = _read(c) + return table, alias + + def _register(table: str | None, alias: str | None, bucket: dict[str, str]) -> None: + if not table or _tainted(table): + return + key = normalize_table_key(table) + if not key: + return + bucket.setdefault(key, table) + alias_map.setdefault(table.casefold(), table) + alias_map.setdefault(key, table) + if alias and not _tainted(alias): + alias_map.setdefault(alias.casefold(), table) + + def _qualified_field(n) -> tuple[str, str] | None: + """(table-or-alias, column) from a ``field`` node, or None if unqualified.""" + if n.type != "field": + return None + ref = col = None + for c in n.children: + if c.type == "object_reference": + ref = _read(c) + elif c.type == "identifier": + col = _read(c) + if ref and col: + return ref, col + return None + + def _collect_joins(n) -> None: + if n.type == "binary_expression" and len(n.children) == 3 and n.children[1].type == "=": + left = _qualified_field(n.children[0]) + right = _qualified_field(n.children[2]) + if left and right: + lt = alias_map.get(left[0].casefold(), left[0]) + rt = alias_map.get(right[0].casefold(), right[0]) + lc, rc = left[1], right[1] + if (not any(_tainted(x) for x in (lt, rt, lc, rc)) + and normalize_table_key(lt) != normalize_table_key(rt)): + a, b = sorted([(lt, lc), (rt, rc)]) + dedup = (normalize_table_key(a[0]), a[1].casefold(), + normalize_table_key(b[0]), b[1].casefold()) + if dedup not in seen_joins: + seen_joins.add(dedup) + joins.append({"left": [a[0], a[1]], "right": [b[0], b[1]]}) + for c in n.children: + _collect_joins(c) + + def _walk_statement(stmt) -> None: + nonlocal op + stmt_op: str | None = None + is_delete = False + for child in stmt.children: + if child.type in ("select", "insert", "update", "delete"): + stmt_op = child.type + is_delete = is_delete or child.type == "delete" + if op is None: + op = stmt_op + + def _walk(n, in_write_target: bool = False) -> None: + t = n.type + if t == "insert": + for c in n.children: + if c.type == "object_reference": + _register(_read(c), None, writes) + else: + _walk(c) + return + if t == "update": + # The relation directly under UPDATE is the write target; + # anything nested deeper (FROM/JOIN) is a read. + for c in n.children: + if c.type == "relation": + table, alias = _relation_parts(c) + _register(table, alias, writes) + else: + _walk(c) + return + if t == "from" and is_delete: + # DELETE FROM x: the from node holds the target directly. + for c in n.children: + if c.type == "object_reference": + _register(_read(c), None, writes) + elif c.type == "relation": + table, alias = _relation_parts(c) + _register(table, alias, writes) + else: + _walk(c) + return + if t == "relation": + table, alias = _relation_parts(n) + _register(table, alias, reads) + for c in n.children: + _walk(c) # subquery relations + return + for c in n.children: + _walk(c) + + _walk(stmt) + + for stmt in root.children: + if stmt.type == "statement": + _walk_statement(stmt) + + if op is None or (not reads and not writes): + return None + + for stmt in root.children: + if stmt.type == "statement": + _collect_joins(stmt) + + return { + "op": op, + "tables_read": sorted(reads.values(), key=str.casefold), + "tables_written": sorted(writes.values(), key=str.casefold), + "joins": joins, + } diff --git a/graphify/sql_usage.py b/graphify/sql_usage.py new file mode 100644 index 000000000..eb7f0846d --- /dev/null +++ b/graphify/sql_usage.py @@ -0,0 +1,131 @@ +"""SQL-usage mining: how application code touches database objects. + +Scans the source files behind a code graph for SQL statements and emits +cross-namespace links from the graph's FILE nodes to MaluDb data-model +nodes ("/.

"). Pushed with the server's +resolve_external option, targets that exist in the tenant's data-model +graph resolve into real edges; unknown targets are reported in the +import's skipped list, not fatal. + +Deliberately regex-based (no SQL parse): the goal is INFERRED-confidence +"this file reads/writes that table" edges, not query understanding. +""" +from __future__ import annotations +import re +from pathlib import Path + +# Table reference sites. Group 1..3: write forms; group 4: read forms. +_WRITE_RE = re.compile( + r"\bINSERT\s+INTO\s+([A-Za-z_\"][\w$.\"]*)" + r"|\bUPDATE\s+([A-Za-z_\"][\w$.\"]*)\s+SET\b" + r"|\bDELETE\s+FROM\s+([A-Za-z_\"][\w$.\"]*)", + re.IGNORECASE, +) +_READ_RE = re.compile(r"\b(?:FROM|JOIN)\s+([A-Za-z_\"][\w$.\"]*)", re.IGNORECASE) + +# Identifiers that follow FROM/JOIN in SQL or SQL-ish prose but are not tables. +_NOISE = { + "select", "the", "a", "an", "this", "that", "these", "each", "one", + "unnest", "generate_series", "jsonb_array_elements", "jsonb_array_elements_text", + "jsonb_each", "json_array_elements", "regexp_split_to_table", "lateral", + "dual", "information_schema", "pg_catalog", "now", "coalesce", "string_to_table", +} + +_MAX_FILE_BYTES = 2_000_000 +_MAX_REFS_PER_FILE = 200 + +_CODE_SUFFIXES = { + ".py", ".php", ".ts", ".tsx", ".js", ".jsx", ".rs", ".go", ".rb", + ".java", ".kt", ".cs", ".sql", ".c", ".h", ".cpp", ".pl", +} + + +def _clean_ident(raw: str) -> str | None: + """Normalize a captured identifier; None if it is noise.""" + name = raw.strip().strip('"').rstrip("(,;") + if not name or name.lower() in _NOISE: + return None + # placeholders / interpolations / pseudo-tables + if any(ch in name for ch in "%{}()"): + return None + bare = name.split(".")[-1] + if len(bare) < 3 or bare.lower() in _NOISE: + return None + # information_schema.x / pg_catalog.x / pg_* system relations + head = name.split(".")[0].lower() + if head in ("information_schema", "pg_catalog") or bare.lower().startswith("pg_"): + return None + return name + + +def _qualify(name: str, default_schema: str) -> str: + return name if "." in name else f"{default_schema}.{name}" + + +def scan_file(text: str) -> dict[str, set[str]]: + """Return {'reads': {names}, 'writes': {names}} found in one file's text.""" + writes: set[str] = set() + reads: set[str] = set() + for m in _WRITE_RE.finditer(text): + name = _clean_ident(next(g for g in m.groups() if g)) + if name: + writes.add(name) + if len(writes) >= _MAX_REFS_PER_FILE: + break + for m in _READ_RE.finditer(text): + name = _clean_ident(m.group(1)) + if name and name not in writes: + reads.add(name) + if len(reads) >= _MAX_REFS_PER_FILE: + break + return {"reads": reads, "writes": writes} + + +def file_nodes(G) -> dict[str, str]: + """Map source_file -> the file-level node id (shortest id per file).""" + best: dict[str, str] = {} + for node_id, data in G.nodes(data=True): + source = data.get("source_file") + if not source: + continue + if source not in best or len(node_id) < len(best[source]): + best[source] = node_id + return best + + +def mine_sql_usage( + G, + source_root: str, + datamodel_ns: str = "datamodel", + default_schema: str = "public", +) -> list[dict]: + """Build cross-namespace links: file node -> datamodel/.
. + + Only files that actually appear in the graph are scanned, so the link + sources always resolve; targets resolve server-side via the import's + resolve_external option (unknown tables are reported, not fatal). + """ + root = Path(source_root) + links: list[dict] = [] + for source_file, node_id in sorted(file_nodes(G).items()): + path = root / source_file + if not path.is_file() and Path(source_file).is_absolute(): + path = Path(source_file) + if not path.is_file() or path.suffix.lower() not in _CODE_SUFFIXES: + continue + try: + if path.stat().st_size > _MAX_FILE_BYTES: + continue + text = path.read_text(encoding="utf-8", errors="replace") + except OSError: + continue + found = scan_file(text) + for rel, names in (("writes", found["writes"]), ("reads", found["reads"])): + for name in sorted(names): + links.append({ + "source": node_id, + "target": f"{datamodel_ns}/{_qualify(name, default_schema)}", + "relation": rel, + "confidence": "INFERRED", + }) + return links diff --git a/tests/fixtures/schema_keys.sql b/tests/fixtures/schema_keys.sql new file mode 100644 index 000000000..ac49db21b --- /dev/null +++ b/tests/fixtures/schema_keys.sql @@ -0,0 +1,15 @@ +CREATE TABLE customers ( + id SERIAL PRIMARY KEY, + email VARCHAR(255) NOT NULL UNIQUE, + name TEXT +); + +CREATE TABLE orders ( + id INT, + customer_id INT NOT NULL, + org_id INT, + slug VARCHAR(64), + total NUMERIC(10,2), + PRIMARY KEY (id), + UNIQUE (org_id, slug) +); diff --git a/tests/test_export.py b/tests/test_export.py index be4743bc5..12087f185 100644 --- a/tests/test_export.py +++ b/tests/test_export.py @@ -603,3 +603,151 @@ def test_backup_env_disable(tmp_path, monkeypatch): (tmp_path / "graph.json").write_text('{"nodes":[],"links":[]}') (tmp_path / ".graphify_semantic_marker").write_text("{}") assert backup_if_protected(tmp_path) is None + + +def test_push_to_maludb_payload_and_report(monkeypatch): + """push_to_maludb POSTs a node-link payload and returns the server report.""" + import urllib.request + from graphify.export import push_to_maludb + + G = make_graph() + + captured = {} + + class FakeResponse: + def read(self): + return json.dumps( + {"namespace": "ns", "nodes": {"imported": G.number_of_nodes()}, + "edges": {"imported": G.number_of_edges()}, "skipped": []} + ).encode("utf-8") + + def __enter__(self): + return self + + def __exit__(self, *args): + return False + + def fake_urlopen(req, timeout=None): + captured["url"] = req.full_url + captured["auth"] = req.get_header("Authorization") + captured["body"] = json.loads(req.data.decode("utf-8")) + return FakeResponse() + + monkeypatch.setattr(urllib.request, "urlopen", fake_urlopen) + + report = push_to_maludb(G, "http://localhost:8000/", "tok123", "ns") + + assert captured["url"] == "http://localhost:8000/v1/graph/import" + assert captured["auth"] == "Bearer tok123" + body = captured["body"] + assert body["namespace"] == "ns" + assert body["provenance"].startswith("graphify") + assert {n["id"] for n in body["graph"]["nodes"]} == set(G.nodes()) + assert len(body["graph"]["links"]) == G.number_of_edges() + # relations and confidence tags pass through raw; the server maps them + for link in body["graph"]["links"]: + assert link["source"] in G and link["target"] in G + assert link["relation"] + assert report["nodes"]["imported"] == G.number_of_nodes() + + +def test_push_to_maludb_rejects_non_http_url(): + """file:// and other schemes are refused before any request is made.""" + import pytest + from graphify.export import push_to_maludb + + G = make_graph() + with pytest.raises(Exception): + push_to_maludb(G, "file:///etc/passwd", "tok", "ns") + + +def test_push_to_maludb_communities_attached(monkeypatch): + """Community ids ride along on the node payload when provided.""" + import urllib.request + from graphify.export import push_to_maludb + + G = make_graph() + communities = cluster(G) + + captured = {} + + class FakeResponse: + def read(self): + return b'{"namespace": "ns"}' + + def __enter__(self): + return self + + def __exit__(self, *args): + return False + + def fake_urlopen(req, timeout=None): + captured["body"] = json.loads(req.data.decode("utf-8")) + return FakeResponse() + + monkeypatch.setattr(urllib.request, "urlopen", fake_urlopen) + + push_to_maludb(G, "http://localhost:8000", "tok", "ns", communities=communities) + + tagged = [n for n in captured["body"]["graph"]["nodes"] if "community" in n] + assert tagged, "expected at least one node to carry a community id" + + +def test_sql_usage_scan_and_mine(tmp_path): + """sql_usage finds reads/writes in code and links from file nodes.""" + import networkx as nx + from graphify.sql_usage import scan_file, mine_sql_usage + + found = scan_file( + 'q = "SELECT a FROM orders o JOIN customers c ON 1=1"\n' + 'db.exec("INSERT INTO order_items VALUES (%s)")\n' + 'db.exec("UPDATE maludb_core.malu$community SET label = $1")\n' + 'x = "select * from unnest(arr)" # noise\n' + ) + assert found["reads"] == {"orders", "customers"} + assert found["writes"] == {"order_items", "maludb_core.malu$community"} + + src = tmp_path / "app.py" + src.write_text('c.execute("SELECT 1 FROM orders WHERE id=%s")\n') + G = nx.Graph() + G.add_node("app.py::main", label="main()", source_file="app.py") + G.add_node("app.py", label="app.py", source_file="app.py") # file node (shorter id) + links = mine_sql_usage(G, str(tmp_path), default_schema="shop") + assert links == [{ + "source": "app.py", + "target": "datamodel/shop.orders", + "relation": "reads", + "confidence": "INFERRED", + }] + + +def test_push_to_maludb_extra_links_and_options(monkeypatch): + """extra_links ride along and options land in the payload.""" + import urllib.request + from graphify.export import push_to_maludb + + G = make_graph() + captured = {} + + class FakeResponse: + def read(self): + return b'{"namespace": "ns"}' + + def __enter__(self): + return self + + def __exit__(self, *args): + return False + + def fake_urlopen(req, timeout=None): + captured["body"] = json.loads(req.data.decode("utf-8")) + return FakeResponse() + + monkeypatch.setattr(urllib.request, "urlopen", fake_urlopen) + + extra = [{"source": "x", "target": "datamodel/public.orders", + "relation": "writes", "confidence": "INFERRED"}] + push_to_maludb(G, "http://localhost:8000", "tok", "ns", + extra_links=extra, options={"resolve_external": True}) + assert captured["body"]["options"] == {"resolve_external": True} + assert captured["body"]["graph"]["links"][-1]["target"] == "datamodel/public.orders" diff --git a/tests/test_multilang.py b/tests/test_multilang.py index 5ad78f724..dc97eb9a8 100644 --- a/tests/test_multilang.py +++ b/tests/test_multilang.py @@ -504,6 +504,38 @@ def test_sql_schema_qualified_names(): assert any("Sales.Customer" in l for l in labels) assert any("Sales.SalesOrder" in l for l in labels) +def test_sql_table_nodes_typed(): + """CREATE TABLE nodes carry type=table, views type=view.""" + r = _extract_sql_or_skip() + by_label = {n["label"]: n for n in r["nodes"]} + assert by_label["users"].get("type") == "table" + assert by_label["active_users"].get("type") == "view" + + +def test_sql_column_metadata(): + """Column names/types/nullability and keys land in table node metadata.""" + r = _extract_sql_or_skip("schema_keys.sql") + by_label = {n["label"]: n for n in r["nodes"]} + + cust = by_label["customers"]["metadata"] + cols = {c["name"]: c for c in cust["columns"]} + assert cols["id"]["type"] == "SERIAL" + assert cols["id"]["nullable"] is False + assert cols["email"]["type"] == "VARCHAR(255)" + assert cols["email"]["nullable"] is False + assert cols["name"]["type"] == "TEXT" + assert cols["name"]["nullable"] is True + assert cust["pk"] == ["id"] + assert ["email"] in cust["unique"] + + orders = by_label["orders"]["metadata"] + assert orders["pk"] == ["id"] # table-level PRIMARY KEY (id) + assert ["org_id", "slug"] in orders["unique"] # composite UNIQUE + ocols = {c["name"]: c for c in orders["columns"]} + assert ocols["customer_id"]["nullable"] is False + assert ocols["total"]["type"] == "NUMERIC(10,2)" + + def test_sql_schema_qualified_alter_fk(): """ALTER TABLE with schema-qualified names produces correct edges.""" r = _extract_sql_or_skip("sample_schema_qualified.sql") diff --git a/tests/test_pg_introspect.py b/tests/test_pg_introspect.py index a133cdad3..6ef08c8f6 100644 --- a/tests/test_pg_introspect.py +++ b/tests/test_pg_introspect.py @@ -15,14 +15,21 @@ def _make_mock_psycopg(tables, views, routines, fks, host="myhost", dbname="mydb", - connect_raises=None): + connect_raises=None, + columns=None, pks=None, uniques=None): """Return a mock psycopg module wired to the provided catalog data. ``routines`` rows must be 5-tuples: (schema, name, rtype, body, ext_lang). ``fks`` rows must be 7-tuples: (constraint_name, t_schema, t_name, [cols], r_schema, r_name, [r_cols]) + ``columns`` rows are 5-tuples: (schema, table, column, data_type, is_nullable) + ``pks`` rows are 3-tuples: (schema, table, [pk_columns]) + ``uniques`` rows are 3-tuples: (schema, table, [index_columns]) ``connect_raises``, if set, is an exception *instance* raised by connect(). """ + columns = columns or [] + pks = pks or [] + uniques = uniques or [] class MockCursor: def __enter__(self): @@ -36,7 +43,13 @@ def execute(self, query, params=None): def fetchall(self): q = self.query.strip().lower() - if "information_schema.tables" in q: + if "information_schema.columns" in q: + return columns + elif "'primary key'" in q: + return pks + elif "pg_index" in q: + return uniques + elif "information_schema.tables" in q: return tables elif "information_schema.views" in q: return views @@ -244,6 +257,68 @@ def test_pg_introspect_composite_fk(): ) +def test_pg_introspect_column_metadata(): + """Catalog columns/PKs/unique indexes land as metadata on the right table + node, replacing the '(id INT)' stub the synthetic DDL produces.""" + mock_tables = [ + ("public", "users", "BASE TABLE"), + ("public", "orders", "BASE TABLE"), + ] + mock_columns = [ + ("public", "users", "id", "integer", "NO"), + ("public", "users", "email", "character varying", "NO"), + ("public", "users", "bio", "text", "YES"), + ("public", "orders", "id", "integer", "NO"), + ("public", "orders", "user_id", "integer", "NO"), + ] + mock_pks = [ + ("public", "users", ["id"]), + ("public", "orders", ["id"]), + ] + mock_uniques = [ + ("public", "users", ["email"]), + ] + + mock_psycopg = _make_mock_psycopg(mock_tables, [], [], [], + columns=mock_columns, pks=mock_pks, + uniques=mock_uniques) + + with patch.dict("sys.modules", {"psycopg": mock_psycopg}): + res = introspect_postgres("postgresql://myuser:secret@myhost/mydb") + + errors = validate_extraction(res) + assert errors == [], f"Validation errors: {errors}" + + by_label = {n["label"]: n for n in res["nodes"]} + users = by_label[_q("public", "users")] + assert users.get("type") == "table" + meta = users["metadata"] + cols = {c["name"]: c for c in meta["columns"]} + assert cols["email"]["type"] == "character varying" + assert cols["email"]["nullable"] is False + assert cols["bio"]["nullable"] is True + assert meta["pk"] == ["id"] + assert meta["unique"] == [["email"]] + # The stub 'id INT' column from the synthetic DDL must not leak through + assert len(meta["columns"]) == 3 + + orders_meta = by_label[_q("public", "orders")]["metadata"] + assert orders_meta["pk"] == ["id"] + assert "unique" not in orders_meta + + +def test_pg_introspect_empty_catalog_metadata(): + """No columns/pks/uniques rows → table nodes get no stub metadata.""" + mock_tables = [("public", "bare", "BASE TABLE")] + mock_psycopg = _make_mock_psycopg(mock_tables, [], [], []) + + with patch.dict("sys.modules", {"psycopg": mock_psycopg}): + res = introspect_postgres("postgresql://myuser:secret@myhost/mydb") + + bare = next(n for n in res["nodes"] if n["label"] == _q("public", "bare")) + assert "metadata" not in bare + + def test_pg_introspect_connection_error(): """A psycopg.OperationalError must be re-raised as ConnectionError with a sanitized message (no DSN/credentials) and no stack-trace noise.""" diff --git a/uv.lock b/uv.lock index 088ebbbdc..20cfe866e 100644 --- a/uv.lock +++ b/uv.lock @@ -97,7 +97,7 @@ name = "autograd" version = "1.8.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.13'" }, + { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" } }, ] sdist = { url = "https://files.pythonhosted.org/packages/67/1c/3c24ec03c8ba4decc742b1df5a10c52f98c84ca8797757f313e7bdcdf276/autograd-1.8.0.tar.gz", hash = "sha256:107374ded5b09fc8643ac925348c0369e7b0e73bbed9565ffd61b8fd04425683", size = 2562146, upload-time = "2025-05-05T12:49:02.502Z" } wheels = [ @@ -477,7 +477,7 @@ resolution-markers = [ "python_full_version < '3.11'", ] dependencies = [ - { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" } }, ] sdist = { url = "https://files.pythonhosted.org/packages/66/54/eb9bfc647b19f2009dd5c7f5ec51c4e6ca831725f1aea7a993034f483147/contourpy-1.3.2.tar.gz", hash = "sha256:b6945942715a034c671b7fc54f9588126b0b8bf23db2696e3ca8328f3ff0ab54", size = 13466130, upload-time = "2025-04-15T17:47:53.79Z" } wheels = [ @@ -555,7 +555,7 @@ resolution-markers = [ "python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ - { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, + { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.13'" }, { name = "numpy", version = "2.4.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.13'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/58/01/1253e6698a07380cd31a736d248a3f2a50a7c88779a1813da27503cadc2a/contourpy-1.3.3.tar.gz", hash = "sha256:083e12155b210502d0bca491432bb04d56dc3432f95a979b429f2848c3dbe880", size = 13466174, upload-time = "2025-07-26T12:03:12.549Z" } @@ -813,10 +813,10 @@ name = "ctranslate2" version = "4.7.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, + { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.13'" }, { name = "numpy", version = "2.4.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.13'" }, - { name = "pyyaml", marker = "python_full_version >= '3.11'" }, - { name = "setuptools", marker = "python_full_version >= '3.11'" }, + { name = "pyyaml" }, + { name = "setuptools" }, ] wheels = [ { url = "https://files.pythonhosted.org/packages/cb/e0/b69c40c3d739b213a78d327071240590792071b4f890e34088b03b95bb1e/ctranslate2-4.7.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:9017a355dd7c6d29dc3bca6e9fc74827306c61b702c66bb1f6b939655e7de3fa", size = 1255773, upload-time = "2026-02-04T06:11:04.769Z" }, @@ -926,7 +926,7 @@ name = "exceptiongroup" version = "1.3.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "typing-extensions", marker = "python_full_version < '3.11'" }, + { name = "typing-extensions" }, ] sdist = { url = "https://files.pythonhosted.org/packages/50/79/66800aadf48771f6b62f7eb014e352e5d06856655206165d775e675a02c9/exceptiongroup-1.3.1.tar.gz", hash = "sha256:8b412432c6055b0b7d14c310000ae93352ed6754f70fa8f7c34141f91c4e3219", size = 30371, upload-time = "2025-11-21T23:01:54.787Z" } wheels = [ @@ -951,12 +951,12 @@ name = "faster-whisper" version = "1.2.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "av", marker = "python_full_version >= '3.11'" }, - { name = "ctranslate2", marker = "python_full_version >= '3.11'" }, - { name = "huggingface-hub", marker = "python_full_version >= '3.11'" }, - { name = "onnxruntime", marker = "python_full_version >= '3.11'" }, - { name = "tokenizers", marker = "python_full_version >= '3.11'" }, - { name = "tqdm", marker = "python_full_version >= '3.11'" }, + { name = "av" }, + { name = "ctranslate2" }, + { name = "huggingface-hub" }, + { name = "onnxruntime" }, + { name = "tokenizers" }, + { name = "tqdm" }, ] wheels = [ { url = "https://files.pythonhosted.org/packages/05/99/49ee85903dee060d9f08297b4a342e5e0bcfca2f027a07b4ee0a38ab13f9/faster_whisper-1.2.1-py3-none-any.whl", hash = "sha256:79a66ad50688c0b794dd501dc340a736992a6342f7f95e5811be60b5224a26a7", size = 1118909, upload-time = "2025-10-31T11:35:47.794Z" }, @@ -1059,10 +1059,10 @@ name = "gensim" version = "4.4.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.13'" }, - { name = "scipy", version = "1.15.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" } }, + { name = "scipy", version = "1.15.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or python_full_version >= '3.13'" }, { name = "scipy", version = "1.17.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, - { name = "smart-open", marker = "python_full_version < '3.13'" }, + { name = "smart-open" }, ] sdist = { url = "https://files.pythonhosted.org/packages/1a/80/fe9d2e1ace968041814dbcfce4e8499a643a36c41267fa4b6c4f54cce420/gensim-4.4.0.tar.gz", hash = "sha256:a3f5b626da5518e79a479140361c663089fe7998df8ba52d56e1ded71ac5bdf5", size = 23260095, upload-time = "2025-10-18T02:06:45.962Z" } wheels = [ @@ -1090,7 +1090,7 @@ wheels = [ [[package]] name = "graphifyy" -version = "0.9.6" +version = "0.9.7" source = { editable = "." } dependencies = [ { name = "networkx", version = "3.4.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, @@ -1357,26 +1357,26 @@ name = "graspologic" version = "3.4.4" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "anytree", marker = "python_full_version < '3.13'" }, - { name = "beartype", marker = "python_full_version < '3.13'" }, - { name = "future", marker = "python_full_version < '3.13'" }, - { name = "gensim", marker = "python_full_version < '3.13'" }, - { name = "graspologic-native", marker = "python_full_version < '3.13'" }, - { name = "hyppo", marker = "python_full_version < '3.13'" }, - { name = "joblib", marker = "python_full_version < '3.13'" }, - { name = "matplotlib", marker = "python_full_version < '3.13'" }, - { name = "networkx", version = "3.4.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "anytree" }, + { name = "beartype" }, + { name = "future" }, + { name = "gensim" }, + { name = "graspologic-native" }, + { name = "hyppo" }, + { name = "joblib" }, + { name = "matplotlib" }, + { name = "networkx", version = "3.4.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or python_full_version >= '3.13'" }, { name = "networkx", version = "3.6.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, - { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.13'" }, - { name = "pot", marker = "python_full_version < '3.13'" }, - { name = "scikit-learn", version = "1.7.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" } }, + { name = "pot" }, + { name = "scikit-learn", version = "1.7.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or python_full_version >= '3.13'" }, { name = "scikit-learn", version = "1.8.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, - { name = "scipy", version = "1.15.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "scipy", version = "1.15.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or python_full_version >= '3.13'" }, { name = "scipy", version = "1.17.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, - { name = "seaborn", marker = "python_full_version < '3.13'" }, - { name = "statsmodels", marker = "python_full_version < '3.13'" }, - { name = "typing-extensions", marker = "python_full_version < '3.13'" }, - { name = "umap-learn", marker = "python_full_version < '3.13'" }, + { name = "seaborn" }, + { name = "statsmodels" }, + { name = "typing-extensions" }, + { name = "umap-learn" }, ] sdist = { url = "https://files.pythonhosted.org/packages/91/bb/0fe2ef85ea775e7b8416b2cf90097aa4b5e0c9c2271d7fe6789bab27d0ca/graspologic-3.4.4.tar.gz", hash = "sha256:79878caf367da3e89046a4ec94291c5b1a5da569f19fdd879d8b45c3563d7110", size = 5134258, upload-time = "2025-09-08T21:44:01.969Z" } wheels = [ @@ -1478,15 +1478,15 @@ name = "huggingface-hub" version = "1.15.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "filelock", marker = "python_full_version >= '3.11'" }, - { name = "fsspec", marker = "python_full_version >= '3.11'" }, - { name = "hf-xet", marker = "(python_full_version >= '3.11' and platform_machine == 'AMD64') or (python_full_version >= '3.11' and platform_machine == 'aarch64') or (python_full_version >= '3.11' and platform_machine == 'amd64') or (python_full_version >= '3.11' and platform_machine == 'arm64') or (python_full_version >= '3.11' and platform_machine == 'x86_64')" }, - { name = "httpx", marker = "python_full_version >= '3.11'" }, - { name = "packaging", marker = "python_full_version >= '3.11'" }, - { name = "pyyaml", marker = "python_full_version >= '3.11'" }, - { name = "tqdm", marker = "python_full_version >= '3.11'" }, - { name = "typer", marker = "python_full_version >= '3.11'" }, - { name = "typing-extensions", marker = "python_full_version >= '3.11'" }, + { name = "filelock" }, + { name = "fsspec" }, + { name = "hf-xet", marker = "platform_machine == 'AMD64' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'arm64' or platform_machine == 'x86_64'" }, + { name = "httpx" }, + { name = "packaging" }, + { name = "pyyaml" }, + { name = "tqdm" }, + { name = "typer" }, + { name = "typing-extensions" }, ] sdist = { url = "https://files.pythonhosted.org/packages/bb/b6/e22bd20a25299c34b8c5922c1545a6320825b13906eb0f7298edfd034a0b/huggingface_hub-1.15.0.tar.gz", hash = "sha256:28abfdddda3927fd4de6a63cf26ab012498a2c24dae52baf150c5c6edf98a1d5", size = 784100, upload-time = "2026-05-15T11:42:52.149Z" } wheels = [ @@ -1511,18 +1511,18 @@ name = "hyppo" version = "0.5.2" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "autograd", marker = "python_full_version < '3.13'" }, - { name = "future", marker = "python_full_version < '3.13'" }, - { name = "numba", marker = "python_full_version < '3.13'" }, - { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.13'" }, - { name = "pandas", version = "2.3.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "autograd" }, + { name = "future" }, + { name = "numba" }, + { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" } }, + { name = "pandas", version = "2.3.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or python_full_version >= '3.13'" }, { name = "pandas", version = "3.0.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, - { name = "patsy", marker = "python_full_version < '3.13'" }, - { name = "scikit-learn", version = "1.7.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "patsy" }, + { name = "scikit-learn", version = "1.7.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or python_full_version >= '3.13'" }, { name = "scikit-learn", version = "1.8.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, - { name = "scipy", version = "1.15.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "scipy", version = "1.15.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or python_full_version >= '3.13'" }, { name = "scipy", version = "1.17.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, - { name = "statsmodels", marker = "python_full_version < '3.13'" }, + { name = "statsmodels" }, ] sdist = { url = "https://files.pythonhosted.org/packages/dd/a6/0d84fe8486a1447da8bdb8ebb249d525fd8c1d0fe038bceb003c6e0513f9/hyppo-0.5.2.tar.gz", hash = "sha256:4634d15516248a43d25c241ed18beeb79bb3210360f7253693b3f154fe8c9879", size = 125115, upload-time = "2025-05-24T18:33:27.418Z" } wheels = [ @@ -1552,7 +1552,7 @@ name = "importlib-metadata" version = "9.0.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "zipp", marker = "python_full_version < '3.11'" }, + { name = "zipp" }, ] sdist = { url = "https://files.pythonhosted.org/packages/a9/01/15bb152d77b21318514a96f43af312635eb2500c96b55398d020c93d86ea/importlib_metadata-9.0.0.tar.gz", hash = "sha256:a4f57ab599e6a2e3016d7595cfd72eb4661a5106e787a95bcc90c7105b831efc", size = 56405, upload-time = "2026-03-20T06:42:56.999Z" } wheels = [ @@ -2279,8 +2279,8 @@ name = "numba" version = "0.65.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "llvmlite", marker = "python_full_version < '3.13'" }, - { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.13'" }, + { name = "llvmlite" }, + { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" } }, ] sdist = { url = "https://files.pythonhosted.org/packages/f6/c5/db2ac3685833d626c0dcae6bd2330cd68433e1fd248d15f70998160d3ad7/numba-0.65.1.tar.gz", hash = "sha256:19357146c32fe9ed25059ab915e8465fb13951cf6b0aace3826b76886373ab23", size = 2765600, upload-time = "2026-04-24T02:02:56.551Z" } wheels = [ @@ -2440,11 +2440,11 @@ name = "onnxruntime" version = "1.26.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "flatbuffers", marker = "python_full_version >= '3.11'" }, - { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, + { name = "flatbuffers" }, + { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.13'" }, { name = "numpy", version = "2.4.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.13'" }, - { name = "packaging", marker = "python_full_version >= '3.11'" }, - { name = "protobuf", marker = "python_full_version >= '3.11'" }, + { name = "packaging" }, + { name = "protobuf" }, ] wheels = [ { url = "https://files.pythonhosted.org/packages/d4/81/29a9eb470994a75eb7b3ccf32be314d7c66675a00ac7b50294816cc2db27/onnxruntime-1.26.0-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:ee1109ef4ef27cad90e823399e61e03b3c6c7bfe0fb820b4baf3678c15be8b3c", size = 18005108, upload-time = "2026-05-08T19:08:11.728Z" }, @@ -2530,10 +2530,10 @@ resolution-markers = [ "python_full_version < '3.11'", ] dependencies = [ - { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, - { name = "python-dateutil", marker = "python_full_version < '3.11'" }, - { name = "pytz", marker = "python_full_version < '3.11'" }, - { name = "tzdata", marker = "python_full_version < '3.11'" }, + { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" } }, + { name = "python-dateutil" }, + { name = "pytz" }, + { name = "tzdata" }, ] sdist = { url = "https://files.pythonhosted.org/packages/33/01/d40b85317f86cf08d853a4f495195c73815fdf205eef3993821720274518/pandas-2.3.3.tar.gz", hash = "sha256:e05e1af93b977f7eafa636d043f9f94c7ee3ac81af99c13508215942e64c993b", size = 4495223, upload-time = "2025-09-29T23:34:51.853Z" } wheels = [ @@ -2596,9 +2596,9 @@ resolution-markers = [ "python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ - { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, - { name = "python-dateutil", marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, - { name = "tzdata", marker = "(python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform == 'emscripten') or (python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform == 'win32')" }, + { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" } }, + { name = "python-dateutil" }, + { name = "tzdata", marker = "sys_platform == 'emscripten' or sys_platform == 'win32'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/f8/87/4341c6252d1c47b08768c3d25ac487362bf403f0313ddae4a2a26c9b1b4c/pandas-3.0.3.tar.gz", hash = "sha256:696a4a00a2a2a35d4e5deb3fc946641b96c944f02230e4f76137fe35d806c4fc", size = 4651414, upload-time = "2026-05-11T18:54:29.21Z" } wheels = [ @@ -2672,7 +2672,7 @@ name = "patsy" version = "1.0.2" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.13'" }, + { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" } }, ] sdist = { url = "https://files.pythonhosted.org/packages/be/44/ed13eccdd0519eff265f44b670d46fbb0ec813e2274932dc1c0e48520f7d/patsy-1.0.2.tar.gz", hash = "sha256:cdc995455f6233e90e22de72c37fcadb344e7586fb83f06696f54d92f8ce74c0", size = 399942, upload-time = "2025-10-20T16:17:37.535Z" } wheels = [ @@ -2855,8 +2855,8 @@ name = "pot" version = "0.9.6.post1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.13'" }, - { name = "scipy", version = "1.15.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" } }, + { name = "scipy", version = "1.15.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or python_full_version >= '3.13'" }, { name = "scipy", version = "1.17.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/42/8b/5f939eaf1fbeb7ff914fe540d659486951a056e5537b8f454362045b6c72/pot-0.9.6.post1.tar.gz", hash = "sha256:9b6cc14a8daecfe1268268168cf46548f9130976b22b24a9e8ec62a734be6c43", size = 604243, upload-time = "2025-09-22T12:51:14.894Z" } @@ -3199,12 +3199,12 @@ name = "pynndescent" version = "0.6.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "joblib", marker = "python_full_version < '3.13'" }, - { name = "llvmlite", marker = "python_full_version < '3.13'" }, - { name = "numba", marker = "python_full_version < '3.13'" }, - { name = "scikit-learn", version = "1.7.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "joblib" }, + { name = "llvmlite" }, + { name = "numba" }, + { name = "scikit-learn", version = "1.7.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or python_full_version >= '3.13'" }, { name = "scikit-learn", version = "1.8.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, - { name = "scipy", version = "1.15.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "scipy", version = "1.15.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or python_full_version >= '3.13'" }, { name = "scipy", version = "1.17.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/4a/fb/7f58c397fb31666756457ee2ac4c0289ef2daad57f4ae4be8dec12f80b03/pynndescent-0.6.0.tar.gz", hash = "sha256:7ffde0fb5b400741e055a9f7d377e3702e02250616834231f6c209e39aac24f5", size = 2992987, upload-time = "2026-01-08T21:29:58.943Z" } @@ -3870,10 +3870,10 @@ resolution-markers = [ "python_full_version < '3.11'", ] dependencies = [ - { name = "joblib", marker = "python_full_version < '3.11'" }, - { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, - { name = "scipy", version = "1.15.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, - { name = "threadpoolctl", marker = "python_full_version < '3.11'" }, + { name = "joblib" }, + { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" } }, + { name = "scipy", version = "1.15.3", source = { registry = "https://pypi.org/simple" } }, + { name = "threadpoolctl" }, ] sdist = { url = "https://files.pythonhosted.org/packages/98/c2/a7855e41c9d285dfe86dc50b250978105dce513d6e459ea66a6aeb0e1e0c/scikit_learn-1.7.2.tar.gz", hash = "sha256:20e9e49ecd130598f1ca38a1d85090e1a600147b9c02fa6f15d69cb53d968fda", size = 7193136, upload-time = "2025-09-09T08:21:29.075Z" } wheels = [ @@ -3919,10 +3919,10 @@ resolution-markers = [ "python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ - { name = "joblib", marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, - { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, - { name = "scipy", version = "1.17.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, - { name = "threadpoolctl", marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, + { name = "joblib" }, + { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" } }, + { name = "scipy", version = "1.17.1", source = { registry = "https://pypi.org/simple" } }, + { name = "threadpoolctl" }, ] sdist = { url = "https://files.pythonhosted.org/packages/0e/d4/40988bf3b8e34feec1d0e6a051446b1f66225f8529b9309becaeef62b6c4/scikit_learn-1.8.0.tar.gz", hash = "sha256:9bccbb3b40e3de10351f8f5068e105d0f4083b1a65fa07b6634fbc401a6287fd", size = 7335585, upload-time = "2025-12-10T07:08:53.618Z" } wheels = [ @@ -3972,7 +3972,7 @@ resolution-markers = [ "python_full_version < '3.11'", ] dependencies = [ - { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" } }, ] sdist = { url = "https://files.pythonhosted.org/packages/0f/37/6964b830433e654ec7485e45a00fc9a27cf868d622838f6b6d9c5ec0d532/scipy-1.15.3.tar.gz", hash = "sha256:eae3cf522bc7df64b42cad3925c876e1b0b6c35c1337c93e12c0f366f55b0eaf", size = 59419214, upload-time = "2025-05-08T16:13:05.955Z" } wheels = [ @@ -4033,7 +4033,7 @@ resolution-markers = [ "python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ - { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, + { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" } }, ] sdist = { url = "https://files.pythonhosted.org/packages/7a/97/5a3609c4f8d58b039179648e62dd220f89864f56f7357f5d4f45c29eb2cc/scipy-1.17.1.tar.gz", hash = "sha256:95d8e012d8cb8816c226aef832200b1d45109ed4464303e997c5b13122b297c0", size = 30573822, upload-time = "2026-02-23T00:26:24.851Z" } wheels = [ @@ -4104,9 +4104,9 @@ name = "seaborn" version = "0.13.2" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "matplotlib", marker = "python_full_version < '3.13'" }, - { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.13'" }, - { name = "pandas", version = "2.3.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "matplotlib" }, + { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" } }, + { name = "pandas", version = "2.3.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or python_full_version >= '3.13'" }, { name = "pandas", version = "3.0.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/86/59/a451d7420a77ab0b98f7affa3a1d78a313d2f7281a57afb1a34bae8ab412/seaborn-0.13.2.tar.gz", hash = "sha256:93e60a40988f4d65e9f4885df477e2fdaff6b73a9ded434c1ab356dd57eefff7", size = 1457696, upload-time = "2024-01-25T13:21:52.551Z" } @@ -4146,7 +4146,7 @@ name = "smart-open" version = "7.6.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "wrapt", marker = "python_full_version < '3.13'" }, + { name = "wrapt" }, ] sdist = { url = "https://files.pythonhosted.org/packages/c5/65/3ada667d32675399001bf022ad3d9f3989b57101351ebc71d6fbe2384634/smart_open-7.6.1.tar.gz", hash = "sha256:4347996e7ba21db7cd1e059632e0b30395407e4f6c660d2ddffc8f2a9ae5f990", size = 54754, upload-time = "2026-05-09T06:23:37.06Z" } wheels = [ @@ -4211,12 +4211,12 @@ name = "statsmodels" version = "0.14.6" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.13'" }, - { name = "packaging", marker = "python_full_version < '3.13'" }, - { name = "pandas", version = "2.3.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" } }, + { name = "packaging" }, + { name = "pandas", version = "2.3.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or python_full_version >= '3.13'" }, { name = "pandas", version = "3.0.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, - { name = "patsy", marker = "python_full_version < '3.13'" }, - { name = "scipy", version = "1.15.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "patsy" }, + { name = "scipy", version = "1.15.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or python_full_version >= '3.13'" }, { name = "scipy", version = "1.17.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/0d/81/e8d74b34f85285f7335d30c5e3c2d7c0346997af9f3debf9a0a9a63de184/statsmodels-0.14.6.tar.gz", hash = "sha256:4d17873d3e607d398b85126cd4ed7aad89e4e9d89fc744cdab1af3189a996c2a", size = 20689085, upload-time = "2025-12-05T23:08:39.522Z" } @@ -4337,7 +4337,7 @@ name = "tokenizers" version = "0.23.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "huggingface-hub", marker = "python_full_version >= '3.11'" }, + { name = "huggingface-hub" }, ] sdist = { url = "https://files.pythonhosted.org/packages/c1/60/21f715d9faba5f5407ff759472ade058ec4a507ad62bcea47cb847239a73/tokenizers-0.23.1.tar.gz", hash = "sha256:1feeeadf865a7915adc25445dea30e9933e593c31bb96c277cee36de227c8bfa", size = 365748, upload-time = "2026-04-27T14:43:25.606Z" } wheels = [ @@ -4928,10 +4928,10 @@ name = "typer" version = "0.25.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "annotated-doc", marker = "python_full_version >= '3.11'" }, - { name = "click", marker = "python_full_version >= '3.11'" }, - { name = "rich", marker = "python_full_version >= '3.11'" }, - { name = "shellingham", marker = "python_full_version >= '3.11'" }, + { name = "annotated-doc" }, + { name = "click" }, + { name = "rich" }, + { name = "shellingham" }, ] sdist = { url = "https://files.pythonhosted.org/packages/e4/51/9aed62104cea109b820bbd6c14245af756112017d309da813ef107d42e7e/typer-0.25.1.tar.gz", hash = "sha256:9616eb8853a09ffeabab1698952f33c6f29ffdbceb4eaeecf571880e8d7664cc", size = 122276, upload-time = "2026-04-30T19:32:16.964Z" } wheels = [ @@ -4973,14 +4973,14 @@ name = "umap-learn" version = "0.5.12" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "numba", marker = "python_full_version < '3.13'" }, - { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.13'" }, - { name = "pynndescent", marker = "python_full_version < '3.13'" }, - { name = "scikit-learn", version = "1.7.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "numba" }, + { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" } }, + { name = "pynndescent" }, + { name = "scikit-learn", version = "1.7.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or python_full_version >= '3.13'" }, { name = "scikit-learn", version = "1.8.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, - { name = "scipy", version = "1.15.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "scipy", version = "1.15.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or python_full_version >= '3.13'" }, { name = "scipy", version = "1.17.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, - { name = "tqdm", marker = "python_full_version < '3.13'" }, + { name = "tqdm" }, ] sdist = { url = "https://files.pythonhosted.org/packages/02/ee/af4171241117f85c74b5ca6448ea1033cc28d599c13651d67289bacd4083/umap_learn-0.5.12.tar.gz", hash = "sha256:6aff02ecac5f2aad9f3c65ee518d7ae93e1a985ae38721fdcffceee4232c33c7", size = 96672, upload-time = "2026-04-08T20:03:54.012Z" } wheels = [ @@ -5001,8 +5001,8 @@ name = "uvicorn" version = "0.47.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "click", marker = "python_full_version < '3.11' or sys_platform != 'emscripten'" }, - { name = "h11", marker = "python_full_version < '3.11' or sys_platform != 'emscripten'" }, + { name = "click" }, + { name = "h11" }, { name = "typing-extensions", marker = "python_full_version < '3.11'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/f6/b1/8e7077a8641086aea449e1b5752a570f1b5906c64e0a33cd6d93b63a066b/uvicorn-0.47.0.tar.gz", hash = "sha256:7c9a0ea1a9414106bbab7324609c162d8fa0cdcdcb703060987269d77c7bb533", size = 90582, upload-time = "2026-05-14T18:16:54.455Z" }