docs(adapters): embedding + benchmark_import specs#148
Closed
mmercuri wants to merge 1 commit into
Closed
Conversation
Closes the spec gap flagged by the depth audit at A:/tmp/adapter-depth-audit.md sec 1.17 (embedding) and sec 1.18 (benchmark_import). These two adapters were the only framework adapters lacking dedicated specifications. The new specs sit alongside the existing user-reference docs (frameworks-embedding.md, frameworks-benchmark_import.md) and describe what the adapter is contracted to do, the explicit non-goals, BYOK/multi-tenancy posture, current test coverage, and the v1.7 / v1.8 roadmap. Per CLAUDE.md, every claim traces back to a line in the source; aspirational behavior is called out as such (e.g. the user-reference doc claim of automatic schema-heuristic mapping in benchmark_import is not implemented). Embedding spec also corrects the event-name discrepancy: the source emits 'retrieval.query', not 'vector_store.query' as the existing user-reference doc claims.
4 tasks
mmercuri
added a commit
that referenced
this pull request
May 10, 2026
…t.md (4th inaccuracy) The Persistence section claimed save_benchmark(metadata, records) but source uses insert_row(table, row) at adapter.py:441-444. Surfaced as bonus finding by agent a56bee25024184e15 while fixing the prior 3 inaccuracies in this PR. Cross-reference PR #148 canonical spec at docs/adapters/benchmark_import.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the spec gap flagged by the depth audit at
A:/tmp/adapter-depth-audit.mdfor the two framework adapters that previously had no dedicated specification:docs/adapters/embedding.md(~2,260 words / 332 lines) — coversEmbeddingAdapter(OpenAI / Cohere / sentence-transformers) andVectorStoreAdapter(Pinecone / Weaviate / Chroma) under audit sec 1.17.docs/adapters/benchmark_import.md(~2,380 words / 361 lines) — coversBenchmarkImportAdapter(HuggingFace / HELM / CSV / JSON / Parquet) under audit sec 1.18, including the architectural carve-out for "data importer" adapters that do not extendBaseAdapter.These docs sit alongside the existing user-reference docs (
frameworks-embedding.md,frameworks-benchmark_import.md). The reference docs are quick-start guides; these specs are the contractual document — Overview, Capability surface, Contract (public API + events emitted + lifecycle + error handling), explicit non-goals, BYOK / multi-tenancy, test coverage (current + target), and v1.7 / v1.8 roadmap.Honesty notes (per CLAUDE.md "no fake claims")
Each spec calls out aspirational behavior in the source/companion-doc as such, rather than restating it as fact:
retrieval.query(verified atvector_store_adapter.py:169 / 201 / 235). The companion user-reference doc claimsvector_store.query. Spec sec 3.3 names the source as authority.benchmark_importautomatic schema heuristics — companion doc claims case-insensitive aliasing; source_apply_schema_mappingshort-circuits on empty mapping with no heuristic fallback. Spec sec 3.3 calls this out and §7 puts heuristic detection on the v1.8 roadmap.gen_ai.*semconv, andorg_idenvelope propagation are flagged where applicable (audit §2 findings 1, 2, 4) and linked to the in-flight PRs that close them (PR feat(instrument): OpenTelemetry GenAI semantic conventions for all LLM-call adapters (spec 07) #125, fix(instrument): Propagate org_id through all event emissions (multi-tenancy CLAUDE.md fix) #118, fix(instrument): Brand leak in agentforce trust layer YAML + missing STREAMING/REPLAY capability declarations #119, feat(instrument): Typed Pydantic event foundation + agno reference (1/17 adapters) #129).What is explicitly NOT supported
The "do NOT support" sections enumerate non-goals so reviewers and customers can avoid building on assumptions the source does not back:
BaseAdapter, no telemetry events, no live mid-run feedback, no MTEB / MMLU / HumanEval native importers (work viaimport_huggingface), no automatic heuristic mapping, no incremental import / cursor, no signing or attestation, no per-record validation.Test plan
embedding_adapter.py,vector_store_adapter.py,benchmark_import/adapter.py).A:/tmp/adapter-depth-audit.md§1.17 / §1.18 verdicts (no behavior is claimed that the source does not implement).