From cdf31f90295e4ce025c67340981ba39529f9d7a3 Mon Sep 17 00:00:00 2001 From: mmercuri Date: Sun, 10 May 2026 10:46:08 -0700 Subject: [PATCH 1/2] docs(adapters): fix three user-doc inaccuracies surfaced by spec PR #148 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - frameworks-embedding.md: vector_store.query -> retrieval.query (source emits literal "retrieval.query" at vector_store_adapter.py:167,199,233 for Pinecone/Weaviate/Chroma wrappers respectively). Added taxonomy note clarifying that all backends normalize to retrieval.query, and cross-referenced the canonical spec at docs/adapters/embedding.md. - frameworks-benchmark_import.md: removed non-existent "case-insensitive heuristic" schema-detection claim. Source actually short-circuits with "if not mapping: return record" at adapter.py:423-434 — explicit-mapping-only behavior in v1.x. Noted v1.8 roadmap for heuristic detection consistent with spec PR #148 §7. - frameworks-benchmark_import.md: BenchmarkImportAdapter is a bare class, not a BaseAdapter subclass (adapter.py:70 declares "class BenchmarkImportAdapter:" with no superclass). Retitled doc to "Benchmark import (data importer)", added architectural note matching spec PR #148 §1, and clarified it has no connect/disconnect lifecycle, no capability declarations, and emits no telemetry events. Cross-referenced PR #148 specs (embedding.md, benchmark_import.md) as canonical for both files. No source code changes; doc-only fix. --- docs/adapters/frameworks-benchmark_import.md | 33 ++++++++++++++++---- docs/adapters/frameworks-embedding.md | 14 +++++++-- 2 files changed, 39 insertions(+), 8 deletions(-) diff --git a/docs/adapters/frameworks-benchmark_import.md b/docs/adapters/frameworks-benchmark_import.md index d859f168..1c09bf60 100644 --- a/docs/adapters/frameworks-benchmark_import.md +++ b/docs/adapters/frameworks-benchmark_import.md @@ -1,10 +1,21 @@ -# Benchmark import framework adapter +# Benchmark import (data importer) `layerlens.instrument.adapters.frameworks.benchmark_import.BenchmarkImportAdapter` imports external benchmark datasets into Stratix evaluation spaces. Unlike -the other framework adapters, this is a **data importer**, not a runtime -instrumentation adapter — it reads benchmarks from disk or from -HuggingFace and produces normalized rows. +the other classes in `layerlens.instrument.adapters.frameworks.*`, this is +a **data importer**, not a runtime instrumentation adapter — it reads +benchmarks from disk or from HuggingFace and produces normalized rows. + +> **Architectural note:** `BenchmarkImportAdapter` is deliberately a +> **bare class** — it does NOT extend `BaseAdapter` +> (`src/layerlens/instrument/adapters/frameworks/benchmark_import/adapter.py:70` +> declares `class BenchmarkImportAdapter:` with no superclass). It has no +> `connect()` / `disconnect()` lifecycle, no `AdapterCapability` +> declarations, no sinks, and emits no telemetry events. It is a one-shot +> ETL pipeline rather than a runtime instrumentation adapter. See the +> canonical specification at +> [`docs/adapters/benchmark_import.md`](./benchmark_import.md) §1 for the +> full architectural carve-out. ## Install @@ -84,8 +95,18 @@ Stratix evaluation schema: | `difficulty` | `difficulty`, `level` | | `category` | `category`, `subject`, `topic` | -When no mapping is provided, the adapter applies a small set of automatic -heuristics (case-insensitive name match against the canonical fields). +**Schema mapping is explicit-only in v1.x.** When no `schema_mapping` is +provided, source field names pass through unchanged — +`_apply_schema_mapping` short-circuits with `if not mapping: return record` +(`src/layerlens/instrument/adapters/frameworks/benchmark_import/adapter.py:423-434`). +There is **no automatic heuristic detection** (no case-insensitive +matching, no fuzzy aliasing) in the current implementation. To map a +source column to a canonical field, you must list it explicitly in +`schema_mapping`. + +Automatic heuristic detection is on the v1.8 roadmap — see +[`docs/adapters/benchmark_import.md`](./benchmark_import.md) §3.3 and §7 +("v1.8 — Schema-heuristic detection"). ## Persistence diff --git a/docs/adapters/frameworks-embedding.md b/docs/adapters/frameworks-embedding.md index 0ba431b5..528a5f80 100644 --- a/docs/adapters/frameworks-embedding.md +++ b/docs/adapters/frameworks-embedding.md @@ -3,7 +3,17 @@ `layerlens.instrument.adapters.frameworks.embedding.EmbeddingAdapter` and `VectorStoreAdapter` instrument embedding-creation calls and vector-store operations across the common providers. They emit `embedding.create` and -`vector_store.query` events with dimension, batch size, and latency metadata. +`retrieval.query` events with dimension, batch size, and latency metadata. + +> **Event taxonomy note:** vector-store retrieval is normalized to +> `retrieval.query` regardless of backend — Pinecone, Weaviate, and Chroma +> wrappers all emit the same event type (the literal string +> `"retrieval.query"` is emitted at +> `src/layerlens/instrument/adapters/frameworks/embedding/vector_store_adapter.py` +> lines 167, 199, and 233). Backend-specific differences live in the event +> payload (e.g. Pinecone's `namespace`, Weaviate's `query_type`, Chroma's +> `distance_*`). See the canonical specification at +> [`docs/adapters/embedding.md`](./embedding.md) for the full contract. ## Install @@ -76,7 +86,7 @@ vs_adapter.connect() | Event | Layer | When | |---|---|---| | `embedding.create` | L3 | Per embedding call. Payload: `provider`, `model`, `batch_size`, `dimensions`, `total_tokens`, `latency_ms`. | -| `vector_store.query` | L3 | Per vector-store query. Payload: `provider`, `top_k`, `result_count`, `latency_ms`, `index_name`. | +| `retrieval.query` | L3 | Per vector-store query. Payload varies by backend (`provider`, `top_k`/`limit`/`n_results`, `result_count`/`match_count`, `has_filter`/`has_where`, `namespace` (Pinecone), `query_type` (Weaviate), `score_min`/`score_max`/`score_mean` (Pinecone), `distance_min`/`distance_max` (Chroma), `latency_ms`). See [`docs/adapters/embedding.md`](./embedding.md) §3.3 for the full payload schema. | ## Dimension tracking From b5bdeed76a9ff5c47ad7bf5f2b7432f5b1e6fa6a Mon Sep 17 00:00:00 2001 From: mmercuri Date: Sun, 10 May 2026 12:04:53 -0700 Subject: [PATCH 2/2] docs(adapters): fix Persistence section in frameworks-benchmark_import.md (4th inaccuracy) The Persistence section claimed save_benchmark(metadata, records) but source uses insert_row(table, row) at adapter.py:441-444. Surfaced as bonus finding by agent a56bee25024184e15 while fixing the prior 3 inaccuracies in this PR. Cross-reference PR #148 canonical spec at docs/adapters/benchmark_import.md. --- docs/adapters/frameworks-benchmark_import.md | 29 +++++++++++++++++--- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/docs/adapters/frameworks-benchmark_import.md b/docs/adapters/frameworks-benchmark_import.md index 1c09bf60..872fe37d 100644 --- a/docs/adapters/frameworks-benchmark_import.md +++ b/docs/adapters/frameworks-benchmark_import.md @@ -110,10 +110,31 @@ Automatic heuristic detection is on the v1.8 roadmap — see ## Persistence -If you pass a `store=` argument to `BenchmarkImportAdapter(...)` (something -that exposes `save_benchmark(metadata, records)`), the adapter writes -imported benchmarks through it. Otherwise records are returned to the -caller and held in `adapter._benchmarks` keyed by `benchmark_id`. +If you pass a `store=` argument to `BenchmarkImportAdapter(...)`, the +adapter persists imported benchmarks through it. The `store` is +duck-typed: it must expose a single method +`insert_row(table_name: str, row: dict)`. The adapter writes one row to +the `"benchmarks"` table containing `metadata.model_dump()`, then one +row per imported record to the `"benchmark_records"` table (each record +is augmented with `record["benchmark_id"] = metadata.benchmark_id` +before insertion). See +`src/layerlens/instrument/adapters/frameworks/benchmark_import/adapter.py:436-446` +for the `_persist` implementation. Persistence failures are swallowed +and logged at debug level — `import_*` will still return +`success=True` even if the store raises. + +If `store` is `None`, the adapter only retains `BenchmarkMetadata` +objects in-memory (`adapter._benchmarks`, keyed by `benchmark_id`, used +by `list_benchmarks()` / `get_benchmark()`). The imported records +themselves are NOT retained on the adapter — they are produced inside +`import_*`, returned via `ImportResult.records_imported` (a count, not +the rows), and otherwise discarded. Callers that need the rows without +a real store should pass a shim whose `insert_row` captures them (e.g. +an in-test list). + +For the deeper persistence contract, see +[`docs/adapters/benchmark_import.md`](./benchmark_import.md) §3.4 +("Persistence contract"). ## Events emitted