Skip to content

docs(adapters): fix user-doc inaccuracies (closes spec PR #148 audit)#150

Closed
mmercuri wants to merge 2 commits into
feat/instrument-frameworks-agentsfrom
docs/fix-frameworks-user-doc-inaccuracies
Closed

docs(adapters): fix user-doc inaccuracies (closes spec PR #148 audit)#150
mmercuri wants to merge 2 commits into
feat/instrument-frameworks-agentsfrom
docs/fix-frameworks-user-doc-inaccuracies

Conversation

@mmercuri

Copy link
Copy Markdown
Contributor

Summary

Fixes three pre-existing inaccuracies in the user-facing adapter docs that
spec PR #148 (docs/adapters/embedding.md, docs/adapters/benchmark_import.md)
surfaced. PR #148 added new spec docs and called out the inaccuracies in
those new specs but did NOT modify the inaccurate user docs. This PR
closes that loop.

Base note: This PR is based on feat/instrument-frameworks-agents
(PR #97) because the user docs being fixed (docs/adapters/frameworks-embedding.md
and docs/adapters/frameworks-benchmark_import.md) only exist on that
branch — they are not yet in main. Targeting that branch keeps this
PR's diff clean (a single commit, two file changes).

Inaccuracies fixed

1. docs/adapters/frameworks-embedding.md — wrong event name

Was: "emit embedding.create and vector_store.query events"
Now: "emit embedding.create and retrieval.query events"

Source emits the literal string "retrieval.query":

  • src/layerlens/instrument/adapters/frameworks/embedding/vector_store_adapter.py:167 (Pinecone wrapper)
  • src/layerlens/instrument/adapters/frameworks/embedding/vector_store_adapter.py:199 (Weaviate wrapper)
  • src/layerlens/instrument/adapters/frameworks/embedding/vector_store_adapter.py:233 (Chroma wrapper)

Updated 2 occurrences (intro + Events-emitted table). Added a taxonomy
note clarifying that vector-store retrieval is normalized to
retrieval.query regardless of backend, with cross-reference to the
canonical spec at docs/adapters/embedding.md.

2. docs/adapters/frameworks-benchmark_import.md — non-existent heuristic

Was: "When no mapping is provided, the adapter applies a small set of
automatic heuristics (case-insensitive name match against the canonical
fields)."
Now: Explicit-mapping-only behavior, with v1.8 roadmap reference.

Source short-circuits when no mapping is provided:

# src/layerlens/instrument/adapters/frameworks/benchmark_import/adapter.py:423-434
@staticmethod
def _apply_schema_mapping(
    record: dict[str, Any],
    mapping: dict[str, str] | None,
) -> dict[str, Any]:
    """Apply field name mapping to a record."""
    if not mapping:
        return record
    ...

There is no case-insensitive matching, no fuzzy aliasing — pass-through
only. v1.8 roadmap reference is consistent with spec PR #148 §7
("v1.8 — Schema-heuristic detection").

3. docs/adapters/frameworks-benchmark_import.md — implies BaseAdapter inheritance

Was: Title "Benchmark import framework adapter"; body grouped it with
"the other framework adapters", implying the runtime-instrumentation
contract (BaseAdapter lifecycle, capabilities, sinks).
Now: Title "Benchmark import (data importer)"; new architectural note
explicitly calls out the bare-class shape.

Source declares no superclass:

# src/layerlens/instrument/adapters/frameworks/benchmark_import/adapter.py:70
class BenchmarkImportAdapter:
    """
    Imports external benchmark datasets into Stratix evaluation spaces.
    ...

This matches the architectural carve-out in spec PR #148 §1 (the adapter
is a one-shot ETL pipeline, not a runtime instrumentation adapter — no
connect() / disconnect() lifecycle, no AdapterCapability enum, no
sinks, no telemetry events).

Out of scope (not changed)

The persistence section in frameworks-benchmark_import.md describes a
save_benchmark(metadata, records) store API, but the source actually
uses insert_row(table, row) (adapter.py:441-444). This is an
additional inaccuracy not in the original audit and was not requested
in the task scope. Recommend filing a follow-up issue.

Test plan

  • Verified each source line cited above by reading the actual file
    content on the feat/instrument-frameworks-agents branch.
  • Verified no remaining vector_store.query references in
    docs/, samples/, src/, or tests/ after the edit
    (grep -rn 'vector_store\.query' returns 0 matches).
  • Cross-checked against PR docs(adapters): embedding + benchmark_import specs #148 spec framing for §1 (architectural
    note), §3.3 (event payload schema reference), and §7 (v1.8
    roadmap).
  • Markdownlint not run (not installed in worktree); visually
    reviewed both files for broken markdown.

References

  • Spec PR docs(adapters): embedding + benchmark_import specs #148 (docs/adapter-specs-embedding-benchmark-import branch)
  • Source: src/layerlens/instrument/adapters/frameworks/embedding/vector_store_adapter.py
  • Source: src/layerlens/instrument/adapters/frameworks/benchmark_import/adapter.py



- frameworks-embedding.md: vector_store.query -> retrieval.query (source
  emits literal "retrieval.query" at vector_store_adapter.py:167,199,233
  for Pinecone/Weaviate/Chroma wrappers respectively). Added taxonomy
  note clarifying that all backends normalize to retrieval.query, and
  cross-referenced the canonical spec at docs/adapters/embedding.md.

- frameworks-benchmark_import.md: removed non-existent
  "case-insensitive heuristic" schema-detection claim. Source actually
  short-circuits with "if not mapping: return record" at
  adapter.py:423-434 — explicit-mapping-only behavior in v1.x. Noted
  v1.8 roadmap for heuristic detection consistent with spec PR #148 §7.

- frameworks-benchmark_import.md: BenchmarkImportAdapter is a bare class,
  not a BaseAdapter subclass (adapter.py:70 declares
  "class BenchmarkImportAdapter:" with no superclass). Retitled doc to
  "Benchmark import (data importer)", added architectural note matching
  spec PR #148 §1, and clarified it has no connect/disconnect lifecycle,
  no capability declarations, and emits no telemetry events.

Cross-referenced PR #148 specs (embedding.md, benchmark_import.md) as
canonical for both files.

No source code changes; doc-only fix.
@mmercuri mmercuri requested a review from m-peko May 10, 2026 17:50
…t.md (4th inaccuracy)

The Persistence section claimed save_benchmark(metadata, records) but
source uses insert_row(table, row) at adapter.py:441-444. Surfaced as
bonus finding by agent a56bee25024184e15 while fixing the prior 3
inaccuracies in this PR.

Cross-reference PR #148 canonical spec at docs/adapters/benchmark_import.md.
@mmercuri

Copy link
Copy Markdown
Contributor Author

Adding follow-up commit for the 4th inaccuracy (Persistence section save_benchmark vs insert_row) that agent a56bee25024184e15 surfaced during the original fix. Now consolidated in this PR per reviewer ergonomics.

@m-peko m-peko closed this May 21, 2026
@m-peko m-peko deleted the docs/fix-frameworks-user-doc-inaccuracies branch July 2, 2026 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants