compass test ready#1
Merged
Merged
Conversation
added 30 commits
May 18, 2026 23:57
…dead frozen class
…ad_id
C1: resume_pending now derives state_store status from the FINAL graph state's
human_approved, not the input decision — a graph that auto-rejects on resume
(SCORE_THRESHOLD edited between pause and resume, hitl_node short-circuits
before consuming interrupt()) no longer leaves state_store='approved' while
the JobNote says hitl_decision='auto_rejected'.
C1 sticky-threshold: score_node now captures SCORE_THRESHOLD into CompassState
at score time. hitl_node and vault_write_node prefer the captured value
over the live config, so threshold edits between pause and resume can no
longer silently flip the decision.
C2: _append_run_log self-migrates 5-col pre-1.B.1 pipeline-runs.md to 6-col
on next append (Paused=0 inserted into historical rows).
I2: _thread_id_for includes os.getpid() to disambiguate cross-process same-
microsecond collisions (defensive for 1.B.3 Modal cron + local CLI race).
C3: gap_aggregator.regenerate() was only called at end of run_pipeline(). Resume paths (MCP approve + timeout_checker) bypassed it, so CompanyNote roles_seen and SkillNote appears_in_jobs drifted from JobNote ground truth. Adding the call inside resume_pending closes the single source of truth. Defers I4 (double-resume race claim_pending) to Phase 1.B.3 alongside Modal cron — the real-world race trigger.
…tion leak Raw 'ss._now = lambda: ...' assignments in test_state_store.py bypassed monkeypatch's tracking, leaking the frozen time into subsequent tests in the suite. Caused timeout_checker tests to see a stale _now() value and mis-classify backdated rows as not-yet-stale. Pre-existing bug from Phase 1.B.1 — passed the full suite by ordering luck; exposed when 1.B.2's new tests shifted the test interaction.
…r bugs Live behavior review surfaced 4 false-negatives at intake_filter — added: - engineering manager / lead / director / head / vp of engineering - solutions engineer (the bare title — 'sales engineer' already caught) - security engineer / application security / infrastructure security - operations specialist / product operations / program manager 'Solutions architect' intentionally NOT added — borderline per Phase 1.A spec. Three deeper LLM-behavior bugs (extract under-extraction, OR-as-AND, and candidate-strong-skill-marked-missing) documented in docs/KNOWN_DATA_QUALITY_ISSUES.md for Phase 2.A eval-harness work.
…r, tailor hallucination, portfolio-claim risks Second adversarial pass surfaced two new bug classes (derived-field staleness asymmetry on role_family; gap_aggregator includes auto_rejected/ timed_out jobs at full weight) plus three operational portfolio-claim risks (skill assessor never exercised, applications never created, concurrent runs not protected).
B6 from KNOWN_DATA_QUALITY_ISSUES.md: role_family is set once at intake; when the OUT keyword list expanded (commit 3828d8f), stale JobNotes kept contributing to the gap plan. Two-part fix: 1. scripts/migrate_role_family.py — one-time migration. Re-runs keyword_classify on each JobNote title; rewrites role_family to 'out-of-scope' for titles the current classifier rejects. In-scope re-classifications (e.g. body-signal upgrades) are left alone since this script doesn't re-evaluate the JD body. 2. gap_aggregator.load_jobs() now skips JobNotes with role_family= 'out-of-scope'. Without this, the migration alone would only rename the field but the gap math would still include them. Migrated 5 stale entries (Sierra/Ramp Security Engineer x2, Decagon Senior Solutions Engineer, Decagon Engineering Manager, Ramp AI Operations Specialist) — all were leaking through the pre-3828d8f filter.
Day-1 Obsidian P1: writer now emits a `## Skills` block between the LLM summary and `## Full JD`, rendering required / nice-to-have / matched / missing as `[[Python]] · [[LangGraph]]` wikilinks so the Obsidian graph view edges JobNotes to SkillNotes. Empty categories are omitted; skill names with characters that get rewritten by `_safe_segment` use the alias form `[[AWS_Bedrock|AWS Bedrock]]` so the link resolves. Backfilled 18 existing JobNotes via scripts/migrate_jobnote_skills_section (idempotent — replaces an existing block if present, inserts before `## Full JD` otherwise).
added 16 commits
May 19, 2026 15:44
P2 — `gap_aggregator._sync_skill_backlinks` writes a `## Jobs requiring this
skill` block to every SkillNote body, listing every JobNote whose required/
nice-to-have skills include the canonical name. Sorted by match_score DESC.
Preserves existing body content above the block (seed notes carry category
descriptions). Idempotent — re-runs replace the block in place.
P3 — `vault_write_node` now auto-generates Obsidian-tag-pane-filterable tags
on every JobNote write: `#tier/{apply-now,opportunistic,...}`,
`#fit/{strong,decent,stretch,weak}`, `#role/{agent-engineer,...}`,
`#decision/{approved,timed_out,...}`. Empty role_family and absent hitl_decision
correctly omit their tag.
Dashboard panels for skill rarity + rare/specialized skills added to
compass-vault/dashboard.md (zero-code Dataview blocks).
8 new tests. 270 passing. Ruff clean.
Item 1 — `compass.vault.reader.load_reject_rules` parses `reject_if_title_contains` and `reject_if_jd_contains` YAML blocks out of preferences.md. `intake_filter_node` applies them as substring matches BEFORE the keyword classifier, so senior/staff/principal titles and `5+ years` / `PhD required` JDs are dropped at zero LLM cost. On a 41-board scrape this is ~40-60% of incoming volume. Item 2 — `_scrape_all` filters each board's results to `date_posted` within the last 30 days then sorts by date_posted DESC (None-dated jobs sink to the bottom) before round-robin interleave + MAX_JOBS_PER_RUN cap. Each board now contributes its freshest postings to the cap rather than arbitrary order. Undated postings still get a shot — many ATSes don't expose date_posted and we can't distinguish stale from undated. 7 new tests. 278 passing. Ruff clean.
…y fields Item 4 — `interview_difficulty` + `cisco_adjacency` fields added to JobNote and CompanyNote schemas. `vault_write_node` populates JobNote fields from `target_companies.get_interview_difficulty` / `get_cisco_adjacency` at write time so each posting carries a snapshot. Tier literal expanded with `opportunistic` and `backend-prep` to match `_profile/preferences.md::tier_weights` (the 3-month pivot tiers were unrepresented and would have failed Pydantic validation). `compass.vault.target_companies` now reads `_profile/target-companies.yaml` alongside the legacy markdown. New helpers: `get_company_meta`, `get_interview_difficulty`, `get_cisco_adjacency`, `get_ats`, `list_yaml_companies`. Invalid Literal values in YAML collapse to "unknown"/"none" so hand-edit typos don't crash the pipeline. Item 3 — `scripts/seed_companies_from_yaml.py` bulk-creates CompanyNotes from the YAML, pre-populating tier + geos + why_interesting + interview_difficulty + cisco_adjacency. write_company_note's existing merge logic preserves any prior human edits. Applied: seeded 41 CompanyNotes. 11 new tests. 289 passing. Ruff clean.
Item 5 — `writer._normalize_full_jd` runs a belt-and-suspenders HTML strip on the JD body at vault-write time. Scrapers already strip at their boundary (greenhouse/lever `_strip_html`, ashby `descriptionPlain`); this is the backstop for any future scraper or hand-built RawJob that bypasses that. A `_looks_like_html` guard ensures JDs containing literal `<` (code snippets, ASCII art, `<200ms latency`) pass through untouched — only visibly-HTML inputs get normalized. Item 6 — langfuse 4.x split credentials away from `CallbackHandler.__init__` which now only accepts `public_key` + `trace_context`. Passing `host=` / `secret_key=` raised TypeError and silently disabled all tracing. Fixed by initializing the `Langfuse(...)` singleton once with full creds, then constructing the bare `CallbackHandler()` which inherits config from the singleton. The refresh will now emit traces. Also: ruff format pass across the files touched in earlier commits this session. 291 passing. Ruff clean.
…ody gate
Three changes that together make the next refresh actually reflect the
user's stated target market (per _profile/target-roles.md::JD-master-boolean):
1. **Positive scraper filter from YAML** — `_scrape_all` now reads
`_profile/target-companies.yaml` (apply-now + opportunistic tiers) and
groups by ATS provider. Each scraper is called with the YAML-derived
slug list. Static `GREENHOUSE_BOARDS` / `LEVER_COMPANIES` /
`ASHBY_BOARDS` config lists are fallback only (tests + safety net when
YAML is missing). 17 greenhouse + 24 ashby boards picked up from the
current YAML.
2. **role_family.IN_TITLE_KEYWORDS expanded** with the master-boolean
title set:
- agent-engineer: AI Agent Engineer, Agentic AI Engineer, Software
Engineer (Agents/Agentic), AI Native Engineer, MTS (frontier-startup
flat-hierarchy signal — Sierra/Decagon/Cognition/Cursor/Mistral/xAI)
- applied-ai: GenAI Engineer, AI Enablement Engineer, AI/ML Engineer
- infra-llm: AI Platform Engineer
"Member of Technical Staff" test flipped from None→in-scope to
in-scope→agent-engineer per the new mapping.
3. **JD-body agent-signal gate** — after title routes to agent-engineer
/ applied-ai / infra-llm, intake_filter scans the JD body for any of
~17 agentic terms (LangGraph, MCP, tool calling, multi-agent, etc.).
Zero hits → drop the JD even though title looked right; the user's
target market wants agents-in-production roles, not generic ML/RAG.
`agent_signal_count` propagates through state into a
`#signal/agent-strong` (3+) or `#signal/agent-mention` (1-2) auto-tag
on the JobNote. SWE-family titles (swe-backend, swe-fullstack) pass
through unfiltered — generic SWE titles at AI-native companies are
often real agent-eng roles.
Also: zeroed `cisco_adjacency` across all YAML entries (the dimension was
built on a wrong premise — user's Cisco role is hardware test development,
not networking/security); removed the cybersec-ai block from
`needs_workday_scraper`.
298 passing. Ruff clean.
… dashboard panel
A — Audit fixes:
- DEFAULT_TIER_WEIGHTS now includes opportunistic (0.85) and backend-prep
(0.5) to match the expanded Tier literal and preferences.md
- _initial_state initializes agent_signal_count=None so the state key is
always present even on graph paths that error before intake_filter
B — Removed `cisco_adjacency` field cleanly:
- Schema literals + JobNote/CompanyNote fields gone
- vault_write_node + target_companies + seed script + tests updated
- Field stripped from 41 existing CompanyNotes
Dimension was built on a wrong premise (user's Cisco role is hardware
test dev, not networking/security). Cleaner schema, no dead weight.
F — YAML re-tier + Austin local startup expansion:
- Distyl AI demoted apply-now → opportunistic (LC-medium-hard, frontier-
grade bar — not realistic for 2-month landing at 1.5 YoE)
- Added 8 new apply-now entries (probed + verified scrapable):
Self Financial (Austin gh), AlertMedia (Austin gh), Diligent (NYC+Austin
gh), Apptronik (Austin gh), Roboflow (NYC+Austin ashby), Maven Clinic
(NYC gh), Maven AGI (Boston ashby), Vapi (SF ashby)
- 49 companies now in YAML (was 41)
- Re-seeded 49 CompanyNotes from YAML
H — Application velocity panel in dashboard.md:
- Apps this week / month / by tier
- "Easy-loop apply-now roles" filter (HackerRank/case/lc-easy/takehome)
so the 8-week sprint surfaces LC-light landings first
295 passing. Ruff clean.
C — `compass/scrapers/workday.py` — posts to the public
`{tenant}.{wdN}.myworkdayjobs.com/wday/cxs/{tenant}/{site}/jobs` JSON
endpoint that every Workday tenant exposes behind their SPA. Slug format
in YAML is `subdomain/tenant/site` (e.g. `wd5/citi/2`). Two-stage fetch:
list endpoint returns summaries → per-posting detail endpoint returns the
full JD body. Pagination via offset/limit; capped at 10 pages × 50 jobs
per board.
Verified scrapable tenants added to YAML apply-now tier:
Wells Fargo · wd1/wf/WellsFargoJobs hackerrank
Citi · wd5/citi/2 hackerrank
Morgan Stanley · wd5/ms/External lc-medium
BlackRock · wd1/blackrock/BlackRock_Professional lc-medium
Adobe · wd5/adobe/external_experienced lc-medium
Capital One, JPM (Oracle Cloud), GS, BofA, USAA, ServiceNow, Atlassian,
HubSpot, Intuit, IBM, Oracle, SAP, Splunk all have Workday tenants but
non-discoverable site IDs — those need the `add_job_from_url` tool
(item D, next).
`Source` literal extended with "workday".
E — score_node prompt now sees a COMPANY TARGETING CONTEXT block with
tier + interview_difficulty + section + notes from the YAML. Prompt
language instructs the LLM to treat this as secondary signal (skill
match still primary) but acknowledge:
- apply-now + easy loop = realistic landing, don't over-penalize gaps
- opportunistic/backend-prep + lc-hard = real stretch, score honestly
309 passing (14 new Workday tests + 5 score-prompt test updates). Ruff clean.
D — Two new MCP tools for sites Compass can't auto-scrape (JPM Oracle
Cloud, LinkedIn, custom careers pages):
add_job_from_url(url, company?, title?) — static-fetches the page,
strips HTML, runs intake_filter → extract → score → vault_write.
Returns score + matched/missing skills + agent_signal_count. Skips
tailor (Sonnet cost) and HiTL (implicit approval by adding manually).
add_job_from_text(company, title, url, jd_text) — for JS-rendered
pages where the static fetch returns near-empty body. User pastes
the JD body from the browser; Compass scores it.
`compass/pipeline/add_url.py` detects ATS provider by URL pattern
(greenhouse/lever/ashby/workday/generic) and runs static httpx fetch with
a browser User-Agent. Bodies under 200 chars return None, signaling
"likely JS-rendered — use paste-text instead."
G — Cover-letter generator (`compass/pipeline/cover_letter.py`):
Pydantic-structured 3-section draft (opening / body / closing), 250-400
words, anchored on specific projects from resume.md + role-clarifications.md.
Uses Sonnet (TAILOR_MODEL). Reads company notes from target-companies.yaml
to ground the "why this company" close.
MCP tool: generate_cover_letter(job_filename) — writes to
vault/cover-letters/{date}-{company}-{title}-{hash}.md. Not part of the
per-job pipeline (too expensive); user invokes per-application.
Strict system-prompt rules: no clichés ("I am writing to apply for…",
"passionate about", "team player"), no invented projects/numbers, plain
prose (no bullets/emoji/markdown), match candidate's resume voice.
318 passing. Ruff clean.
…cleanups
Audit #4 — Deleted dead `update_skill_note` function from
`compass/vault/writer.py`. `gap_aggregator._sync_skill_counters` has
owned the counter since Phase 0.B; the function had no callers in
production code (tests-only). Removed the 2 tests as well; pruned now-
unused SkillNote/SkillCategory/category_for imports. (-23 / +0 lines)
Added 19 companies to YAML under `provider: manual` — sites Compass
can't auto-scrape (Oracle Cloud / Workday-undiscovered / iCIMS /
SmartRecruiters / brassring). Each entry carries interview-loop
research from public sources so the score node can weight them
correctly when the user adds specific roles via `add_job_from_url` /
`add_job_from_text`:
Banks (apply-now):
JPMorgan (NYC, Plano) — hackerrank, 2 problems / 1hr
Capital One (NYC, Plano, McLean) — lc-medium PowerDay, 70-min CodeSignal
Goldman Sachs (NYC, Dallas) — lc-medium FAANG-lite
Bank of America (NYC, Charlotte) — hackerrank, easy
USAA (San Antonio) — hackerrank, lower bar
Consulting (apply-now):
Deloitte (NYC, Austin, Dallas) — case + Python basics, 2.6/5 diff
Accenture (NYC, Austin, Dallas) — lc-medium + Docker/K8s, 3.3/5
EY, PwC, KPMG (NYC, Dallas) — case + light tech
Mid-SaaS (apply-now):
ServiceNow (Austin), Atlassian (Austin), HubSpot, Intuit (Plano TX)
Enterprise (apply-now):
IBM watsonx, Oracle OCI GenAI, SAP Joule
NYC (apply-now):
Bloomberg (NYC HQ) — Terminal agent features
Updated `get_tier()` precedence — now reads YAML first (3-month-pivot
source of truth), falls back to markdown. This is needed because the
new manual-add entries live in YAML only; without the change they'd
resolve to tier="unknown" when JobNotes get written via
`add_job_from_url`. Verified: JPMorgan/Capital One/Deloitte/IBM all
correctly resolve to apply-now now.
Re-seeded 72 CompanyNotes (was 49 + 23 new manual entries).
`provider: manual` entries are correctly skipped by
`_yaml_scraper_slugs()` (auto-scrape excludes), but `get_company_meta`
+ `get_tier` + score-node context still find them.
316 passing. Ruff clean.
Phase 2.A eval harness, scaffolded so it works BEFORE any hand-labels exist.
compass/evals/dataset.py — EvalRecord Pydantic schema (id / jd_text /
source / expected_score / expected_skills / notes). load_dataset() returns
[] when the JSON file is missing so first-run isn't blocked. save_dataset,
add_example (auto-increment id), add_from_jobnote (label a real JobNote
from the vault).
compass/evals/metrics.py — pure-function primitives:
score_mae, score_rmse, score_bias (signed bias tells which direction
to tune the prompt — over vs under)
skill_recall, skill_precision (case-insensitive set match)
aggregate(...) (rolls per-JD into a ScoreMetrics dc)
compass/evals/judge.py — LLM-as-judge that produces synthetic expected_*
labels without manual annotation. ~$0.002/JD on Flash. Useful for first-
pass sanity checks on the upcoming refresh's 50-150 fresh JobNotes
before you hand-label any of them. Explicitly NOT a substitute for
hand-labels (two LLMs may share biases) but catches gross failures
like extract finding 2/12 skills.
compass/evals/runner.py — two modes:
run_against_labels() → compares Compass output to EvalRecord.expected_*
run_against_judge() → compares to live judge LLM output (no labels needed)
CLI: `uv run python -m compass.evals.runner [--judge] [--limit N]`
Writes `compass/evals/results-{mode}-{ts}.json` per-record + summary.
MCP tool `run_evals(mode, limit)` exposes the runner programmatically.
Per-record output includes `missed_skills` + `extra_skills` (set diff between
extracted and expected) so a user reading the JSON can immediately see which
skills extract_node systematically misses — directly drives the B1 prompt-
tuning loop the spec promised.
26 new tests (metrics primitives + dataset roundtrip + runner integration
with mocked extract/score/judge + extract-failure resilience).
342 passing. Ruff clean.
Systematic adversarial probe surfaced these bugs — all fixed with regression tests so they can't silently come back. 1. **URL deduplication too naive** (HIGH) — `https://example.com/job`, `https://example.com/job/`, `https://example.com/job?utm_source=google`, `http://EXAMPLE.com/job`, and `https://example.com/job#apply` were treated as 5 distinct URLs. The same JD via Google + LinkedIn referrals + RSS would dup-write 5 JobNotes and re-burn LLM cost 5 times. Fix: new `compass.vault.url_dedup.normalize_url` collapses scheme case, host case, default ports, trailing slash, tracking params (utm_*, gclid, fbclid, msclkid, ref, source, campaign), fragments, and sorts remaining query params. http and https of the same URL collapse to https. Applied in `_vault_url_set()` and `write_job_note()` URL match. 13 new tests in tests/vault/test_url_dedup.py. 2. **Agent-signal gate had false positives** (HIGH) — single hits on "agent" or "agents" alone passed the gate even when the JD body was talking about "change agent", "User-Agent header", or "outage management agent" in irrelevant context. The gate was meant to drop non-agent-eng roles with AI-sounding titles; it was letting them through on noise. Fix: tiered signals into STRONG (LangGraph, MCP, multi-agent, tool calling, agentic-AI as a phrase, etc.) and WEAK ("agent" / "agents" / "agentic" alone). Gate now requires at least ONE strong signal. Weak hits still increment `agent_signal_count` for the `#signal/*` tag but don't satisfy the gate alone. Confirmed: "change agent" + "User-Agent header" titles now drop. New tests: TestAgentSignalFalsePositives + TestAgentSignalCountConsistency. 3. **`agent_signal_count` was None on reject paths** (LOW) — title/JD-reject paths returned `{"in_scope": False, "role_family": "out-of-scope"}` and left `agent_signal_count` as None in state. Now consistently set to 0. 4. **Path traversal in `generate_cover_letter` MCP tool** (MED) — `job_filename` was joined to VAULT_PATH/jobs/ with no validation. A relative path like `../_profile/resume.md` resolved OUTSIDE jobs/ and would load + cover-letter-ize whatever was there. Fix: resolve both paths + `relative_to(jobs_dir)` check before reading. Confirmed: "../_profile/resume.md" now rejected with clear error. 3 new tests in tests/test_mcp_path_traversal.py. 5. **YAML cache stale in long-running MCP server** (MED) — `get_company_meta` and `get_tier` cached YAML at module-import time. User editing the YAML in Obsidian while the MCP server runs got stale data until restart. Fix: `_maybe_reload_yaml()` stat-checks mtime on every accessor call (sub-millisecond) and reloads if changed. Cache survives reads, refreshes on writes. 3 new tests in tests/vault/test_target_companies_aliases.py. 6. **No aliases for tenant slug ↔ display name** (MED) — `get_tier("JPMC")` returned "unknown" even though "JPMC" is the universal short name for JPMorgan. The substring-fuzzy fallback didn't bridge "jpmc" → "jpmorgan". Workday/Oracle tenant tokens are short codes (wf=WellsFargo, ms=Morgan Stanley, COF=Capital One, GS=Goldman Sachs); without aliases, the scraper board_token would route to tier=unknown even when the company is in YAML. Fix: YAML `aliases:` field on companies; `_parse_yaml` indexes by primary name + all aliases. Added aliases to JPMorgan, Capital One, Goldman Sachs, Bank of America, Morgan Stanley, Wells Fargo. Confirmed: get_tier("JPMC") = "apply-now", same for COF/MS/WF/etc. 7. **`add_job_from_url` accepted any URL scheme** (MED) — javascript:, file://, ftp://, data: all reached httpx unchecked. httpx would presumably error but it's not validated upfront; could mask a copy-paste mistake or accidental local-file read. Fix: scheme allowlist {http, https} enforced before fetch attempt. Also rejects URLs with no hostname (e.g. "https://" alone). 6 new tests in tests/pipeline/test_add_url_scheme.py. 8. **Eval results filename collision when run twice in same second** (LOW) — `results-{mode}-{HHMMSS}.json` would overwrite if the runner was invoked twice within one second (rapid iteration during prompt tuning). Now `HHMMSS-microseconds` for collision-free ID. 342 → 374 tests passing (+32 regression tests). Ruff clean.
Audit of Phase 2.A against the master spec found 1 real correctness bug
and 1 missing spec item. Fixed both.
**Production-fidelity bug** — the runner called `_extract(jd_text)` and
`_score(req, profile, job)` directly. Those are the raw LLM functions.
But the production graph wraps them with post-processing the runner skipped:
extract_node wraps `_extract` with:
- `_normalize_skill_list` — canonicalizes skill names against the
taxonomy AND drops skills the LLM emitted but the JD body doesn't
actually contain (anti-hallucination guard)
- `_seniority_with_title_fallback`
score_node wraps `_score` with:
- `_score_with_retry` — retries once if reasoning looks truncated
- `_constrain_to_jd_skills` — drops matched/missing skills outside
the JD's required ∪ nice_to_have universe (LLM hallucination guard)
The runner was measuring un-normalized, un-constrained output. The recall
metric would have reported worse than the user actually experiences, and
the B1 prompt-tuning loop would have optimized against the wrong baseline.
Runner now applies the same normalization + constraint layers post-LLM-call.
`_extract` / `_score` remain the patchable surface so existing test stubs
work; production helpers are wrapped around the stub output.
Two regression tests pin this:
test_runner_applies_extract_normalization — hallucinated skills are
dropped before metrics computation
test_runner_applies_score_constraint — score matched_skills
outside the JD universe are filtered before match_skill_recall
**Missing spec item — interactive labeling CLI** — `scripts/label_jd.py`,
called for in spec but never built. Two modes:
uv run python -m scripts.label_jd <JobNote>
Reads compass-vault/jobs/<JobNote>, shows the body, runs Compass's
current extract on it, displays the agent's normalized skill list,
prompts for expected_score + expected_skills + notes. Press Enter
to accept the agent's list as ground truth (one-keystroke labeling).
uv run python -m scripts.label_jd
Paste-mode without a JobNote — useful for labeling JDs found
manually outside the vault.
Path-traversal guard mirrors the MCP fix from the previous commit. Late-
binds VAULT_PATH via `cfg.VAULT_PATH` (not `from compass.config import
VAULT_PATH`) so the temp_vault test fixture's monkeypatch works.
3 new tests for the labeler CLI.
374 → 379 passing. Ruff clean.
**Still missing from spec, documented as deferred:**
- Cost-per-run tracking — pydantic-ai exposes usage on the run result;
surfacing it requires a small refactor of `_extract`/`_score` return
signatures. Defer until we actually want cost charts.
- Langfuse Dataset Run integration — the Langfuse 4.x callback is wired
for the pipeline; eval traces would need explicit propagation through
`_run_extract_and_score`. Defer until 2.B (public-trace polish).
- `docs/EVAL_BASELINE.md` — correctly deferred; needs labeled data first.
…pers
Dispatched 4 parallel adversarial subagents across Phase 1.B (HiTL,
checkpointing, RAG) + Phase 1.A pipeline nodes + Phase 2.A eval harness +
scraper layer. 20 candidate findings; 12 verified as real bugs; 8 fixed
here (most-impactful); 7 reported as false alarms or already-handled.
**CRITICAL fixes:**
1. **HiTL race condition** (`compass/hitl/{state_store,resume}.py`) —
Modal cron's timeout-checker and an MCP `approve_job` call could both
pass the `get_pending` check on the same thread_id, both call
`graph.ainvoke` on the same checkpoint, and write conflicting JobNotes
(one "timed_out", one "approved"). New `claim_pending(thread_id) -> bool`
atomically transitions `pending → resuming`. `resume_pending` calls it
before invoking the graph; second caller raises cleanly. `mark_resolved`
updated to finalize from either `pending` or `resuming`. CHECK
constraint on SQLite table updated to include 'resuming'.
The Phase 1.B.3 `claim_pending atomic transition` spec item is now
shipped. 5 regression tests in tests/hitl/test_claim_pending.py.
2. **Stale RAG chunks after skill rename/delete** (`compass/rag/indexer.py`)
— `build_index` used `upsert` only, never `delete`. If a skill was
renamed or removed from skill-inventory.md, the old chunk's id stayed
in the collection forever and polluted future retrieval. Now after
every build, any id in the collection that isn't in the current
sections list is dropped. Force-rebuild flag no longer needed for
day-to-day correctness.
3. **Workday relative URLs corrupted via dedup** (`compass/scrapers/workday.py`)
— when detail-fetch's `externalUrl` was null, `_to_rawjob` stored the
bare `externalPath` (e.g. "/job/abc"). `normalize_url` then produced
the invalid string `https:///job/abc` — broken vault apply-link AND
broken dedup. Now `_to_rawjob` accepts `base_url=` and absolutizes
the relative path. If no usable URL at all, drop the job rather than
persist a broken row. 4 regression tests.
4. **`swe-mobile` + `swe-frontend` promotable to `agent-engineer`**
(`compass/pipeline/role_family.py:GENERIC_FAMILIES`) — a React Native
job at an LLM-native startup whose JD mentions "LangGraph" + "multi-
agent" once anywhere in the body got upgraded to agent-engineer.
User isn't a mobile or frontend specialist; promoting these clutters
the vault with mis-classified roles. Both families removed from the
upgrade-eligible set. 4 regression tests.
5. **`tailor_node` fired on any approved job regardless of score**
(`compass/pipeline/nodes/tailor.py`) — only checked
`state.get("human_approved")`. A mis-click in the HiTL UI on a
score=1.0 row would burn a Sonnet call (~$0.05). Added explicit
score-vs-threshold gate after retrieving the score_result. Uses
sticky `state["score_threshold"]` (set by score_node) over the
live SCORE_THRESHOLD config. 3 regression tests.
**IMPORTANT fixes:**
6. **`list_yaml_companies` returned duplicate entries** for companies
with aliases (`compass/vault/target_companies.py`). `_yaml_meta`
indexes by primary name AND every alias, so `list(_yaml_meta.values())`
yielded N+1 copies for N aliases. JPMorgan (5 aliases) appeared 6
times. seed_companies_from_yaml + gap_aggregator + MCP tools saw
inflated counts. Now deduplicated by object identity.
7. **Judge skill normalization mismatch deflated eval recall**
(`compass/evals/runner.py:run_against_judge`) — judge prompt instructs
"use exact phrases from the JD" so it returns raw forms like
"pydantic-ai". Compass's extract returns canonical form ("Pydantic AI").
Naïve lowercased set match between them missed legitimate hits on
every multi-word punctuated skill. Now applies `_normalize_skill_list`
to the judge's output before comparison so both sides are canonical.
8. **Ashby silently dropped jobs without `descriptionPlain`**
(`compass/scrapers/ashby.py`) — some boards populate `description` (HTML)
without `descriptionPlain`. Mirrored Lever's HTML-fallback pattern.
**Bugs the reviewers flagged that turned out to be FALSE:**
- claim_pending was on a real check-in-table named `writes`, not the
non-existent `checkpoint_writes` reviewer guessed. Verified by probing
the actual langgraph schema (`['checkpoints', 'writes']`).
- `score_threshold` IS set in state by score_node (lines 176 + 181) —
reviewer missed it. Sticky-threshold guarantee already holds.
**Bugs filed as known gaps (not fixed here):**
- Workday detail-fetch rate-limit handling (50 parallel requests per
page, no 429 detection)
- Chroma PersistentClient singleton (multiple instances on same path
with high concurrency)
- Empty `_company_context_block` crash potential
- `taxonomy.load_taxonomy` LRU cache not invalidated in tests that
monkeypatch TAXONOMY_PATH
- Lever scraper is dead code (no YAML entries use it)
379 → 396 passing (+17 wave-2 regression tests). Ruff clean.
…he/cold-start Dispatched 4 more parallel adversarial agents across surfaces NOT deeply reviewed in waves 1+2: analysis layer (assessor/gap/learning_bridge/taxonomy), MCP server + applications + scripts, graph orchestration + node interactions, and cold-start + concurrency + encoding edge cases. 20+ candidate findings; ~12 verified as real; 9 fixed here. Agent C's most- alarming claim (`__interrupt__` reads from wrong object → "HiTL entirely broken") was verified FALSE — LangGraph DOES populate `__interrupt__` in return state. Current code works. **SECURITY (path traversal — actively leaks .env contents pre-fix):** 1. **`learning_bridge.resolve()` path traversal** — `learning-vault://../../.env` resolved through bare Path-join (no `..` normalization), returned contents of any sibling file. VERIFIED live before fix: leaked `OPENROUTER_API_KEY` from a sibling `secret.env`. Now `_parse_uri` resolves both candidate and root, checks `relative_to(root)`, raises ValueError on escape; `resolve()` catches and returns None. 5 regression tests in tests/vault/test_learning_bridge_traversal.py. 2. **`get_profile(section)` MCP tool path traversal** — same threat class as the cover_letter fix from wave 1, but unfixed. `../.env` resolved to vault root with `.md` suffix. Now resolves + `relative_to(profile_dir)` guard; returns clear error on escape. Late-binds VAULT_PATH via `cfg.VAULT_PATH` so the temp_vault fixture's monkeypatch actually applies (CLAUDE.md lesson #2 footgun was present here too). 3 regression tests. **CORRECTNESS:** 3. **`normalize(None)` crashed `gap_aggregator.regenerate()`** — a JobNote with a YAML list-entry written as `- ` parses as `[None]`, and `taxonomy.normalize(None)` raised AttributeError on `.lower()`, taking down the whole batch's gap-plan regeneration. Now `normalize()` accepts None/empty/non-str defensively and returns None. 5 tests. 4. **`master-gap-plan.md` non-atomic write** — `Path.write_text` is truncate-then-write; SIGKILL or disk-full mid-write left a half-written markdown file that the MCP `get_master_gap_plan` tool returned as garbage. Now writes to `.tmp` sibling + `os.replace()` for atomic swap. 5. **`save_dataset` non-atomic write** — same class for the eval dataset JSON. Pre-fix, a SIGKILL during `label_jd.py` could corrupt the labeled dataset to unparseable JSON, making the whole history unloadable. Same `.tmp + os.replace` pattern. 6. **`add_job_from_text` accepted any URL scheme** — `add_job_from_url` already enforces http/https; sibling tool didn't. Now matches. 7. **`seed_companies_from_yaml.py` empty-string company name** — would produce `companies/.md` dotfile carrying no information. Now skipped with a visible warning. **CACHE / TEST-MOCKABILITY:** 8. **`taxonomy.load_taxonomy` + `_synonym_index` LRU caches never cleared** — long-running MCP server kept stale taxonomy after the user edited `skill-taxonomy.md`. Added `refresh_taxonomy()` helper that clears both LRU caches; ready to be called from MCP analysis tools that depend on fresh taxonomy. 9. **`llm.get_model_id` bypassed `compass.config`** — read os.environ directly, so tests that monkeypatched `compass.config.SCORE_MODEL` silently used the production model. Now reads `cfg.<NAME>` first, falls back to os.environ. Existing env-patching tests still work after they also clear the cfg attr. Added a regression test that pins the cfg-monkeypatch path. **Bugs flagged FALSE (verified by direct probe before fixing):** - "HiTL `__interrupt__` reads from wrong object — entirely broken" — Probed langgraph directly: `ainvoke` DOES return state with `__interrupt__` populated when `interrupt()` fires. Current code works. **Bugs filed as known gaps (not fixed here — acceptable per scope/risk):** - `skill_assessor._apply` SkillNote read-modify-write race (sequential per call, MCP vs cron race unlikely without Modal deployed) - `append_agent_log` concurrency interleaving (< PIPE_BUF on Linux, rare on macOS, low impact) - `write_company_note` race for same company across 5 concurrent jobs (acceptable per existing code comment — roles_seen owned by gap_aggregator anyway) - Workday detail-fetch rate-limit handling (no 429 detection) - Chroma PersistentClient singleton (multiple instances per process) - `build_graph(checkpointer=None)` no guard (footgun but documented) - `intake_node` doesn't set in_scope on null current_job (downstream recovers via intake_filter's same check) 396 → 411 tests passing (+15 wave-3 regression tests). Ruff clean. 14 commits past the `phase-1b2-rag` tag.
…ough Plain-English explanation of what's in the project, how the pieces fit together, and how data flows from a scraped JD to a JobNote. Replaces the 3-day-stale ARCHITECTURE.md as the canonical "explain this project to me" doc. Covers: - The three vaults (compass-vault, learning-vault, code repo) and their separate ownership - ASCII data-flow diagram from scraper → vault_write → gap_aggregator - Per-node behavior (intake_filter / extract / score / reflect / hitl / tailor / vault_write) including the post-LLM cleanup layers each one applies - Per-module description with file paths - The 12 MCP tools and what they're for - The skill-assessor meta-loop — the differentiated angle that makes Compass interesting as a portfolio artifact - Vault file layout with directory tree - Cost expectations per LLM-touching node - What's NOT in the pipeline (deferred Modal cron, Cisco internal, Oracle Cloud / iCIMS scrapers, auto-apply, LinkedIn) - Honest "what Compass does well vs doesn't do yet" closing summary Companion to but distinct from PROJECT_OVERVIEW.md (which is the 510-line phase-by-phase narrative). This doc is the current-state architecture; PROJECT_OVERVIEW is the historical phase walkthrough.
Self-contained doc for the next agent + the user to pick up tomorrow morning. Captures: - Current branch/HEAD state + 90-second pre-flight check - Summary of the long session that ended 2026-05-19 (24 bugs fixed across 3 adversarial review waves; 411 tests passing) - The single uncompleted task that blocks the refresh: the user's resume + skill-inventory pass - Tomorrow's plan in time-ordered steps: pre-flight → resume pass → first refresh → Obsidian verification → eval --judge baseline → HiTL approvals → label 10 JDs → rigorous eval baseline → first applications - Communication style preferences (verbatim from user's CLAUDE.md) - Known data-quality issues likely to surface in the first refresh (B1 under-extraction, Workday rate limits, ML/Vision over-ranking, Anthropic/Sierra dominance, Adobe scrape breadth) - Cost expectations (~$0.70 for day 1) - Reading order for context - Deferred items with rationale (Modal cron, claim_pending callers, RAG chunking, portfolio polish, blog post) - The single most likely opening scenario + step-by-step response Doc is intentionally action-oriented (tomorrow is for running Compass + applying, not engineering). Closes with a reality check: code is done; the bottleneck moved to runtime + user action.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
should be testable and semi-ready to use.