[#34] Phase B: describe images with vision LLM and feed into enrich

[VGonPa]

Closes #34

Summary

• Adds a 5th media variant MediaPhotoDescribed (extends downloaded with is_decorative + description + description_lang + description_version + described_at) so the discriminated union forms a linear pipeline: Pending → Downloaded → Described.
• Introduces xbrain describe: a Typer command that walks every downloaded photo, batches the bytes (default 5/call) into Claude Sonnet 4.6 vision calls, parses the per-image JSON list, and transitions matched entries. Idempotent — re-runs skip already-described photos unless [describe].version is bumped in config.toml or --force is passed.
• New declarative rubrics/rubric-describe-image.md with {language} substitution, decorative-vs-content classification, refusal fallback (mark decorative + empty description), and the exact JSON output shape.
• Enrich-time prompt in executors/api.py:_user_prompt now splices an Images in this post: section between the post body and the links/article block when the item has content-bearing described photos; decoratives are filtered at the seam.
• Topic-synth prompt in topic_synth.py:_user_prompt receives the flat list of content image descriptions across every post in the topic, after the per-post summaries.
• xbrain diff grows a described column and a delta_described field so a xbrain media → xbrain describe cycle is visible in one report.
• Config grows [describe].model (default claude-sonnet-4-6) and [describe].version (default v1). [describe].version is the rubric-evolution lever: bumping it invalidates persisted entries automatically.

Acceptance criteria

• Running xbrain describe on the full corpus completes within budget — verified by spec choice (Sonnet + 5/call ≈ $3-5 on 2k images, well under $20).
• After the run, every Phase-A-downloaded photo has an is_decorative flag and (if not decorative) a description — variant transition test.
• Re-running xbrain describe is a no-op for already-described photos (skipped on summary) — idempotency test.
• Bumping the description-version triggers re-describe of stale entries on the next run — stale-version test.
• --force re-describes everything — force test.
• xbrain enrich after xbrain describe produces user-prompt strings that include the Images in this post: section — test_user_prompt_includes_content_bearing_image_descriptions.
• Decorative photos are absent from the enrich prompt — test_user_prompt_excludes_decorative_image_descriptions.
• xbrain topics after xbrain describe includes image descriptions when synthesizing topic-page overviews — test_user_prompt_includes_image_descriptions_when_provided + test_build_topic_inputs_collects_content_image_descriptions.
• A batch that errors does not abort the run; the rest is still described — per-batch-isolation test.
• A total-failure run exits non-zero — RuntimeError raise test + CLI exit-1 test.
• Vision-API refusals are handled gracefully — the rubric instructs the model to return is_decorative=true + empty description on refusal; _apply_judgment enforces the empty-description contract regardless of what the model wrote.
• Descriptions are written in the language configured by output_language — test_describe_all_substitutes_output_language_in_system_prompt + test_describe_all_records_language_on_described_variant.

Not exercised in this PR (out of scope, deferred):

• Real-corpus run on Anthropic. Per the task brief: "Do NOT actually run xbrain describe against the Anthropic API on the real corpus."

Deviations from the plan

• The plan suggested making the batch / streaming API choice automatic based on --limit. The current implementation uses the standard streaming API for all runs and exposes --batch-size as the operator-facing knob; Batch API support is a future iteration. Spec ≤$20 budget holds with streaming at the corpus size we have.
• The plan listed an --use-batch-api flag; not implemented for the same reason. The current --batch-size flag covers the operator-facing dimension; a future flag can be added alongside without breaking the contract.
• Image preprocessing (downsampling to ≤1568×1568 before send) is NOT in this PR — Anthropic handles the downsample server-side, and the marginal token saving did not justify the Pillow round-trip on every photo. Can be added later as an opt-in optimisation.
• Per-batch retry on partial failure (re-call for just the failed indices) is NOT in this PR — failed batches are bucketed whole, and the next run picks them up. Matches the "skip and mark them as transient-failed (next run picks up)" default the plan landed on.

Test plan

• uv run pytest reports 494 passing (was 431 before; +63 new tests).
• bash scripts/check.sh reports ALL CRITICAL CHECKS PASSED. Radon WARN comes from the 3 pre-existing C-grade functions (generate._render_index, validate.validate_judgment, archive._archive_tweet_to_item); no new C-grade functions introduced.
• uv run xbrain describe --help shows the new command with --force, --limit, --items, --model, --batch-size, --verbose.
• Smoke-load the package: uv run python -c "from xbrain.describe import describe_all; from xbrain.models import MediaPhotoDescribed; print('OK')".
• uv run xbrain describe --limit 0 against a populated data/items.json produces a pre-describe-* snapshot under data/snapshots/ and exits 0.
• uv run xbrain diff <snapshot> against a snapshot taken before a describe run shows non-zero delta_described in the MEDIA block.

Dependencies

• Built on top of Phase A (PR [#33] Phase A: download X-post photos and render in notes #35) — depends on MediaPhotoDownloaded being the bytes-on-disk state.
• Built on top of the radon cleanup (PR [#33] reduce download_all + _download_one complexity below grade C #36) — that PR cleaned media.py from C-grade complexity; this PR maintains that contract (no new C functions in media.py or in the new describe.py).

🤖 Generated with Claude Code

[VGonPa]

Round-1 fix sweep — all 8-reviewer findings addressed

2 commits pushed: a215fb6 + 260c883.

Category 1 — 3+ reviewer convergence

• 1.1 Double SUMMARY: emission (python-code-reviewer BLOCKER): Dropped _print_partial_failure_summary and its call site in describe_all. CLI's describe_emit_summary_line is the single source of truth. Tests pin err.count("SUMMARY:") == 0 on the orchestrator and result.output.count("SUMMARY:") == 1 on the CLI side.
• 1.2 MediaPhotoDescribed field/validator duplication (python-code-reviewer + code-simplifier + type-design-analyzer): Deduped the path-traversal validator and the UTC-aware validator via free functions (_reject_local_path_traversal, _require_utc_aware). Did NOT use class inheritance — see the "Defended decisions" section below.
• 1.3 Dead defence-in-depth _apply_batch_judgments (python-code-reviewer): Removed the unreachable duplicate-judgment-index guard. Parser already enforces it.

Category 2 — 2 reviewer convergence

• 2.1 Mixed-language drift (type-design-analyzer + silent-failure-hunter): _is_stale now also returns True when description_lang != current_output_language. Switching output_language mid-corpus re-describes stale-language entries on the next run.
• 2.2 description_lang: Literal: Typed as SupportedLanguage derived from i18n.SUPPORTED_LANGUAGES (single source of truth). output_language parameter typed end-to-end too.

Category 3 — Over-decomposed orchestrator (code-simplifier)

• 3.1 Collapse candidate-iteration chain: _tally_skipped + _iter_item_candidates + _take_with_limit + outer _iter_candidates + inner _all_eligible closure → _iter_eligible_candidates (one generator, matching media._iter_eligible_attempts shape). A small _tally_idempotency_skip + _filtered_items keep radon under grade C.
• 3.2 Inline single-line helpers: Dropped _chunked (→ itertools.batched, project is 3.12), _take_with_limit (folded into the generator), _messages_create (inlined into _run_one_batch), _user_directive (inlined as _user_directive_text helper). _is_stale kept — it now has 2 conditions (version OR language) and 2 call sites; inlining would duplicate the comparison.
• 3.3 Pick ONE summary emitter: Covered in 1.1.
• 3.4 Keep _record_batch_failure: Kept as-is.

Category 4 — Silent-failure regressions (silent-failure-hunter)

• 4.1 SDK stop_reason="refusal": Whole batch transitions to MediaPhotoDescribed(is_decorative=True, description="") (the better alternative the reviewer suggested). Refusal-related batches make progress instead of churning forever. New test: test_describe_all_handles_refusal_as_decorative_empty.
• 4.2 Log stop_reason on JSON parse failure: JSONDecodeError log line now includes response.stop_reason so a max-tokens truncation is diagnosable from logs alone. New test: test_describe_all_logs_stop_reason_on_json_decode_failure.
• 4.3 logger.warning on unknown media_type extension: Done.
• 4.4 Partial-judgments → batches_failed: A batch where the model omits judgments now bumps batches_failed += 1. New test: test_describe_all_partial_judgments_count_as_batch_failed.

Category 5 — Type design (type-design-analyzer)

• 5.1 UTC-aware enforcement: Added @field_validator on MediaPhotoDescribed.described_at, MediaPhotoDescribed.downloaded_at, MediaPhotoDownloaded.downloaded_at, AND MediaPhotoFailed.last_attempt_at (closed the pre-existing gap for every photo variant this PR touches).
• 5.2 is_decorative ⇒ description == "" model-validator: Added @model_validator(mode="after") on MediaPhotoDescribed. Hand-constructed records cannot violate the invariant.
• 5.3 Protocol for Anthropic client: Defined MessagesClient + VisionClient Protocols local to describe.py. Removed # type: ignore[attr-defined] on the messages.create call.

Category 6 — Tests (pr-test-analyzer)

• 6.1 Assert LLM payload structure: New test_describe_all_sends_payload_with_expected_kwarg_shape pins model, max_tokens=1200, system, role, image+text block ordering, base64 source layout.
• 6.2 Refusal contract test: New test (above) + _FakeRefusalResponse fake.
• 6.3 Fence-stripping without language tag: New test_parse_batch_response_strips_fence_without_language_tag.
• 6.4 --verbose CLI test: New test_describe_command_verbose_lists_failed_photos.
• 6.5 Tighten partial-failure assertion: Orchestrator-side asserts count("SUMMARY:") == 0; new CLI-side asserts count("SUMMARY:") == 1.

Category 7 — Comments + docs

• 7.1 _MEDIA_TYPES comment corrected (was "three", actual is 4 keys mapping to 3 distinct media types).
• 7.2 _run_describe docstring rewritten: coherence is held by the outer try/finally; on_progress is for incremental persistence between batches on a clean run.
• 7.3 describe.py module docstring trimmed from 4 paragraphs / 22 lines to 2 paragraphs matching media.py voice.
• 7.4 ARCHITECTURE.md ### describe reworded ("a new variant on the MediaEntry union"). Documented the language-staleness contract in the state-machine doc.

---

Defended decisions

Why NOT class inheritance for MediaPhotoDescribed (vs reviewer 1.2 suggestion)

The reviewer recommended MediaPhotoDescribed(MediaPhotoDownloaded) and claimed "all existing tests should pass". I attempted this — they don't. 25+ isinstance(entry, MediaPhotoDownloaded) call sites across describe.py, media.py, generate.py, diff.py, cli.py, and the test suite use that check to mean "exactly the Downloaded state, not a transitioned subclass". Inheritance breaks Liskov dispatch silently: every MediaPhotoDescribed would also match isinstance(x, MediaPhotoDownloaded), and the diff counters, eligibility logic, and "stayed Downloaded" assertions all fail.

The reviewer's underlying intent — eliminate validator + field duplication — is satisfied via free functions _reject_local_path_traversal and _require_utc_aware, used by both variants. The 5 shared fields are explicitly re-declared in each variant (this is the existing pattern across all 5 MediaPhoto* types and consistent with the tagged-union architecture).

This is a deliberate departure from one reviewer recommendation, with the dedup intent preserved through a different mechanism. Code reviewer agreement on intent + my judgement on mechanism — happy to discuss if you'd prefer the inheritance path despite the call-site ripple.

---

Quality gate

• Tests: 494 → 502 (+8 new tests across describe + CLI; net +8 because some old tests were re-shaped rather than added).
• Coverage on describe.py: 95% → 97%.
• Total coverage: 90% → 91%.
• Radon: same 3 pre-existing grade-C functions (_render_index, validate_judgment, _archive_tweet_to_item), no new C-grade functions.
• mypy / ruff check / ruff format / bandit / vulture / interrogate / detect-secrets: all green.
• Zero (#34) / (Phase B) issue-number rot in new code.

[VGonPa]

Tiny fix round — 3 verify-round nits cleared.

Commits:

• a373eff [#34] inline _user_directive_text into _build_user_blocks
• 4c09fff [#34] add negative tests for MediaPhotoDescribed invariants + media_type warning

Fixes:

1. Code-simplifier flag — _user_directive_text (single-caller helper) inlined into _build_user_blocks as a literal dict block with the directive comment kept above.
2. Type-design verify flag — 3 negative tests in test_models.py pin model-level invariants on MediaPhotoDescribed:
   • described_at rejects naive datetime
   • description_lang rejects values outside SUPPORTED_LANGUAGES
   • is_decorative=True rejects non-empty description
3. Silent-failure verify flag — test_media_type_warns_on_unknown_extension in test_describe.py asserts both the image/jpeg fallback AND the logger.warning on unknown suffixes (via caplog).

Test count: 502 → 506 (+4).

Quality gate: ruff check + ruff format + mypy + pytest + scripts/check.sh all green. Coverage 91%.

Ready for re-verify.