Codex/port hubs from main#1
Closed
cheesss wants to merge 86 commits into
Closed
Conversation
…, investigation packet
- event-dashboard.html: Add Simulate button for approval items (dryRun preflight)
→ calls POST /api/approval-queue/{id}/review with dryRun:true, renders [DRY RUN] banner
→ item stays in inbox (not removed), shown in info tone with blue badge
- event-dashboard.html: Add .inbox-result.info CSS for DRY RUN tone
- event-dashboard.html: Expand inboxAction catch → structured log + POST /api/runtime-issues
- event-dashboard.html: Fix inboxBulkAction silent catch {} → console.warn with id/error
- scripts/event-dashboard-api.mjs: Add POST /api/runtime-issues (capture issue envelope)
and GET /api/runtime-issues (list last 50 issues)
- scripts/_shared/runtime-issue-writer.mjs (new): captureRuntimeIssue(), classifyIssue(),
safeToAutoFix() — writes data/runtime-issues/YYYY-MM-DD/{id}.json
- scripts/_shared/surface-route-map.mjs (new): static surface→file map + getFilesForIssue()
- scripts/generate-codex-investigation-packet.mjs (new): CLI tool — reads issue by ID,
collects file excerpts + server log, builds compact packet, optionally sends to Claude API
- docs/CODEX_REMEDIATION_LOOP_PLAN_2026-04-15.md (new): full plan with per-step Codex prompts
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…n, banners, bulk guards - surface navigation: all 5 nav buttons switch active surface, URL hash updates - Simulate button: appears only for approval items, calls dryRun API, shows [DRY RUN] banner in info tone, item stays in inbox (not removed) - action result banners: EXECUTED / REJECTED / FAILED on API response variants - bulk action guards: buttons disabled with no selection; mixed approval+triage selection disables incompatible bulk accept - stale badge: risk-snapshot with old oldestInternalUpdatedAt triggers stale chip - all tests use page.route() mocking — no live DB required Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…erts - src/components/DeckGLMap.ts: default MAP_INTERACTION_MODE flat (was 3d) → terrain source only added in 3d mode, reducing cold-start GPU load - src/theme-map-lens.ts: add renderPausedByHost flag — refreshDynamicData() and applyContext() bail early when iframe is hidden; resume triggers full re-render; add wm-map-lens-visibility message handler; remove 'military' layer from crossDomain + geopolitical presets (reduces default layer count) - scripts/generate-structural-alerts.mjs: add resolveBaselineContext() helper; lifecycle alert severity downgraded to 'medium' for small/near-zero baselines; alertScore capped at 46/52 for low-baseline signals; adds baselineNote to detail Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…pdate - scripts/event-dashboard-api.mjs: event-uplift-grades query adds LATERAL join to article_event_map+articles to return top 3 source articles per signal - event-dashboard.html: normalizeSignalToReviewItem maps sources array from API - event-dashboard.html: renderReviewItemCard renders [1][2][3] citation chips using existing .signal-cite CSS; click shows source title/date/link popover - event-dashboard.html: showSignalCitePopover() — per-card popover from inline JSON data, reuses .cite-popover CSS; no global cite registry pollution - event-dashboard.html: today feed title now dynamic — "Recent Events (7d fallback)" when meta.window !== '24h'; fallback badge text clarified - event-dashboard.html: daily-digest header shows source name when fallback (e.g. "Source: article_fallback_72h (72h-fallback)") - docs/DASHBOARD_DATA_CONTINUITY_AUDIT_2026-04-14.md: updated status table — P1 #5 structural alerts resolved, P1 #8 transmission freshness resolved; remaining acceptance criteria listed Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tasks, opensky path - scripts/discover-emerging-tech.mjs: source filter now supports LIKE patterns (e.g. 'google news' → LIKE 'google news:%', 'foo*' → LIKE 'foo%'); default sources list includes 'google news' - scripts/generate-tech-report.mjs: loadTopicContext rewritten — uses buildTopicArticleProfile for keyword-based candidate matching; loads up to 300 linked articles; excludes arxiv for geopolitics topics - scripts/master-daemon.mjs: taskGenerateStructuralAlerts runs all 4 periods (week/month/quarter/year) sequentially; adds taskRefreshEventMarketTransmission at 2h interval; updates startup log - scripts/proposal-executor.mjs: handleAddRss imports evaluateAndRegisterFeed from _shared/discovered-source-registry.mjs (was importing from TS server file) - src/services/military-flights.ts: OpenSky proxy URL corrected to /api/opensky/states/all (was /api/opensky) - src/services/server/autonomous-discovery.ts: load rss-allowed-domains.json via readFileSync instead of static import (avoids bundler JSON assert issues) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Ops surface: new Diagnostics section with Runtime Issues panel
- Loads GET /api/runtime-issues on surface switch (lazy, same pattern as inbox)
- Renders each issue: classification badge (External/API/UI/Action/Freshness),
surface, action, timestamp, error message (truncated to 120 chars)
- Issue ID shown at bottom — click to copy to clipboard (for use with
generate-codex-investigation-packet.mjs --issue-id)
- Badge count in section title updates on load
- CSS: .runtime-issue-row, .cls-external/api/ui/action/other classification colors
- switchSurface('ops') now triggers refreshRuntimeIssues()
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Phase 0 — source-probe.mjs
- New shared module: probeSource() with 9-adapter cascade
- Adapters: direct-feed, html-alternate-feed, wordpress-rss,
sitemap-news, json-ld, open-graph, html-list, playwright(stub), llm-selector(stub)
- Quality breakdown: itemCount, recentItemCount, titleDiversity,
duplicateRate, spamRate, themeRelevance, sourceFreshness
- Never throws — all failures return structured SourceProbeResult
- Per-adapter 8s timeout, total 15s probe deadline
Phase 1 — self-heal-sources.mjs
- Probe gate before queueForApproval for untrusted URLs
- Only quality-passing (recentItems≥3, nextAction=register|review)
candidates enter approval queue
- Rejected/needs-adapter candidates logged to automation log
- Approval payload now includes probe evidence metadata
Phase 2 — proposal-executor.mjs
- add-rss dryRun performs real Source Probe instead of generic response
- Returns resolvedUrl, connectorKind, qualityScore, sampleItems,
wouldRegister, wouldSeedCount
Phase 3 — approval-queue.mjs + event-dashboard-api.mjs
- approval-queue: needs-fix status supported as reviewable decision
- getPendingApprovals returns needs-fix items alongside pending
- API: execution.skipped=true marks approval as needs-fix, not executed
Phase 4 — proposal-executor.mjs handleAddRss
- Uses resolvedUrl from probe for registration and article seeding
- probe.sampleItems reused directly when count≥3 (avoids re-fetch)
- Returns resolvedUrl, connectorKind, probeTraceId in result
Phase 5 — event-dashboard.html
- DRY RUN banner shows probe metadata: resolvedUrl, connectorKind,
qualityScore, recentItemCount, sample headlines, warnings
- SKIPPED result shows warning tone, Accept button disabled
- needs-fix rows stay visible with NEEDS FIX banner
- Approval cards show probe evidence from payload
Phase 6 — source-adapter-proposal.mjs
- CLI: probe → Claude API → adapter repair proposal
- buildRepairPrompt() generates structured repair context
- dry-run mode prints packet without API call
- Tests: 6 source-probe + 3 adapter-proposal (all pass)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Phase 0 (semantic contract): - deriveResponseMeta now derives valueOrigin (observed/estimated/research) and validAsOf from the response mode, keeping withMeta as the single patch point for all 13 callers. - inferResponseMode accepts the extended mode enum (nowcast/imputed/composite/ mirrored/backfill/replay) via an explicit whitelist; unknown modes fall back to 'live' so payloads stay safe. - MODE_STALE_THRESHOLD_HOURS gains per-mode thresholds; mirrored is always stale with an explicit reason. Phase 0.5 (signal_history origin tagging): - Migrations add value_origin + writer_id columns to signal_history and tag legacy GDELT-proxy rows (marketStress/transmissionStrength/eventIntensity/ gpr) as 'proxy' with writer_id='master-pipeline-step0-gdelt'. - New scripts/_shared/signal-history-writer.mjs centralises writes so that every INSERT carries explicit origin metadata. refresh-fred-signals-to-nas, refresh-market-quotes-to-nas, refresh-event-market-transmission, master-pipeline STEP 0, and backfill-new-sources all route through it. - classifySignalQuality returns proxy/composite/imputed statuses in addition to observed/mirrored/stale; buildSignalSummary exposes signalOrigin map alongside signalQuality for per-signal provenance. Phase 0.6 (source hygiene): - add-articles-source-metadata.mjs migration adds wire_source, publisher_group, market_relevance columns + check constraint + indexes. - shared/publisher-groups.json maps 70+ domains into canonical groups with high/medium/low relevance tiers (collapses EuroNews 7 langs, France24 4 langs, Le Monde en/fr into single groups; tiers 93% of feeds). - scripts/_shared/source-classifier.mjs resolves publisher_group + wire_source from URL/title/body. backfill-article-source-metadata.mjs populates legacy rows in 2k-row batches. - scripts/_shared/source-concentration.mjs computes HHI, effectiveSourceCount, wireDominated, topShare; wire duplicates collapse into a single bucket so AP/Reuters syndication no longer inflates diversity. - add-canonical-events-hhi.mjs migration adds source_hhi / effective_source_count / wire_dominated / top_source_share columns. recompute-canonical-events-hhi.mjs rebuilds them from publisher_group annotations. Tests: 29 new assertions across event-dashboard-freshness-contract, source-classifier, and source-concentration suites. All pass. Planning docs: docs/NOWCAST_PLAN_ISSUES_2026-04-17.md captures the full issue inventory driving these phases. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…egistry
Phase 1 (storage):
- create-nowcast-tables.mjs provisions estimated_signal_nowcasts,
nowcast_reconciliation, nowcast_training_snapshots on NAS.
- buildSignalSummary fuses estimated_signal_nowcasts into /api/kpi-summary
for any signal whose observed value is missing or non-observed. Response
meta.mode flips to 'nowcast' when any KPI is estimated; per-signal values
are exposed under signals.nowcasts with method/confidence/interval fields.
Phase 2a-d (rate + commodity + fx + composite nowcasts):
- train-rates-nowcast.py trains vintage-aware ridge models for
hy_credit_spread, treasury10y, yieldSpread, ig_credit_spread. Walk-forward
holdout with 90% interval and baseline (carry-forward) comparison.
- train-commodity-fx-nowcast.py handles oilPrice (XLE/USO/XOM/CVX) and
dollarIndex (UUP/FXE). Drops weekend rows implicitly via ETF close alignment.
- compute-rates-nowcast.(py|mjs) runs inference every 30 minutes, respects
the market session floor via market-calendar.mjs, and writes confidence
bounded to [session_floor, 1.0].
- compute-composite-nowcasts.mjs derives marketStress from nowcasted inputs
via RSS interval combination and session-scaled confidence. Never
overwrites observed values.
Phase 2e (clean eventIntensity):
- compute-event-intensity-nowcast.mjs counts canonical events where
contributing articles have market_relevance in {high, medium}, collapses
wire duplicates, and normalises against the 30-day hour-of-day baseline.
Asian session blackouts no longer register as low intensity.
Phase 3 (UI):
- event-dashboard.html gains a new NOWCAST trust chip (sky-blue) rendered
above the freshness chip when valueOrigin='estimated'. Chip title shows
method + confidence + market session. KPI strip propagates
nowcasts[0].confidence/intervalLow/intervalHigh into renderTrustRow.
Phase 4 (source gate + regime mask):
- create-nowcast-source-gate.mjs provisions nowcast_source_eligibility with
18 seed rules mapping each nowcast target to its allowed proxies,
max_lag_hours, holdout_mae_max, and regime_mask. ig_credit_spread disables
shock-regime inference because the HYG/VIX relationship breaks down.
- scripts/_shared/nowcast-source-gate.mjs detects current regime from VIX
and returns abstain + reason when eligible sources < 2 or 24h MAE exceeds
drift threshold.
Phase 5 (registry + reconciliation):
- create-model-registry.mjs adds model_registry with candidate → shadow →
active → deprecated lifecycle.
- reconcile-nowcasts.mjs runs every 15 minutes, pairs unreconciled
estimated_signal_nowcasts with the first observed signal_history row on
or after target_ts, writes nowcast_reconciliation with abs_error/
pct_error/within_interval/calibration_bucket, and emits alerts when
24h coverage falls below 70%.
- promote-nowcast-model.mjs gates candidate→shadow on holdout MAE < baseline
* 0.85 AND coverage≥0.80, and shadow→active on 30-day live MAE ≤ holdout
* 1.2 AND coverage≥0.80.
Master-daemon registry gets four new tasks:
- rates-nowcast (30 min)
- composite-nowcasts (30 min)
- event-intensity-nowcast (30 min)
- reconcile-nowcasts (15 min)
Tests: 35 nowcast-related assertions + 96 CI core tests all pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Review by user correctly flagged two gaps in the previous commit series: Phase 4 had the gate defined but not enforced by any writer, and tests covered only the semantic contract and source hygiene surfaces, not the full estimated_signal_nowcasts → reconcile → fusion path. Gate enforcement: - Extract pure evaluateGate() from nowcast-source-gate.mjs so the same logic can be exercised without a live client. - Add Python port scripts/_shared/nowcast_source_gate.py (evaluate_gate + check_eligible_sources) that mirrors the JS implementation field-by-field. - compute-rates-nowcast.py now calls check_eligible_sources() before INSERTing into estimated_signal_nowcasts; abstain short-circuits the write path and returns a structured reason. - compute-composite-nowcasts.mjs gates marketStress via checkEligibleSources before writeComposite(); the shock-regime gate flips marketStress to abstain per its seed rules. - compute-event-intensity-nowcast.mjs calls the gate with its two proxy sources (article_count_hourly, gdelt_event_count) so the task aborts cleanly when the gate rejects the setup. - create-nowcast-source-gate.mjs grows seed rules for marketStress (shock-disabled) and eventIntensity. Pure-function refactors for testability: - classifyReconciliation + classifyDriftAlert extracted from reconcile-nowcasts.mjs. Coverage and bucket thresholds stay in one place. - evaluateCandidateGate + evaluateShadowGate extracted from promote-nowcast-model.mjs; the legacy evaluateCandidate/Shadow wrappers now delegate to the pure functions. - fuseNowcastsIntoLookup extracted from event-dashboard-api.mjs's buildSignalSummary so the observed-vs-estimated merge can be tested without a pg connection. New test suites (42 assertions across 6 files): - nowcast-source-gate.test.mjs: evaluateGate abstain paths, regime mask, drift threshold, fake-client checkEligibleSources with shock regime. - reconcile-nowcasts.test.mjs: bucket classification, interval coverage, pctError handling, drift alert severity ladder. - promote-nowcast-model.test.mjs: candidate/shadow gate thresholds and failure reasons. - nowcast-fusion.test.mjs: observed always wins, proxy gets replaced, anyEstimated flag propagation. - compute-composite-nowcasts.test.mjs: source-level wiring check proving the gate is consulted before writeComposite. - compute-rates-nowcast-gate-wiring.test.mjs: ensures the Python INSERT cannot run before check_eligible_sources(); locks lag_hours derivation. Totals: 77 nowcast-related assertions + 96 CI core tests, all passing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace per-row UPDATE loops with UNNEST-array bulk UPDATEs. backfill-article-source-metadata.mjs: 2000 UPDATEs/batch → 1 UPDATE/batch via UNNEST($1::bigint[], $2::text[], $3::text[], $4::text[]). Observed: 68k rows, 13 min → seconds. recompute-canonical-events-hhi.mjs: per-event SELECT+UPDATE → single JOIN-load all article-event pairs, group in memory, bulk UPDATE 5000 events/batch via UNNEST of 6 parallel arrays. Observed: 62k events, 100+ min projected → ~30 s. Logic identical; WHERE/COALESCE semantics preserved. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Close the structural gap where the nowcast pipeline assumed market_quotes
had 180-day history it had no way to populate. Three disconnected layers
shared the same wrong premise: daemon refresh collected only 8 hardcoded
symbols, trainers/inference required 10+ additional feature symbols, and
the handoff jumped straight to --window 180 training with no bootstrap
step.
Single source of truth
scripts/_shared/market-quote-symbols.json
coreSnapshots + nowcastFeatures per target + signalMappings
scripts/_shared/market-quote-symbols.mjs
Node accessors (getAllRequiredSymbols, getSignalMappings, etc.)
Consumers now read from the JSON
refresh-market-quotes-to-nas.mjs — DEFAULT_MARKET_QUOTE_SYMBOLS derives
from coreSnapshots ∪ flatten(nowcastFeatures). Daemon polls the full
required set, not just the original 8.
train-rates-nowcast.py, train-commodity-fx-nowcast.py — TargetSpec
.proxy_market_symbols derives from nowcastFeatures[target].
Bootstrap
scripts/bootstrap-market-quotes-history.mjs (new)
1. Fuse from worldmonitor_intel.historical_raw_items warm store
(yahoo-{SYM} and yahoo-{SYM}-5y dataset_ids).
2. Patch the remainder via Yahoo chart API (range=1y, interval=1d).
3. Idempotent — skip rows already at (symbol, observed_at).
Verified: 18/18 required symbols ≥120 trading days in last 180d after
one run.
Coverage audit (blocks training/inference on insufficient history)
scripts/_shared/market-quote-coverage.mjs (JS, pure + DB wrapper)
scripts/_shared/market_quote_coverage.py (Python port, kept in sync)
Hooked into:
train-rates-nowcast.py main() — abort_if_missing → RuntimeError
train-commodity-fx-nowcast.py main() — same
compute-rates-nowcast.py predict_and_write() — abstain per-target
Inference abstain is per-target so a gap on one signal doesn't kill
the whole rates-nowcast task.
Docs
docs/NOWCAST_HANDOFF_2026-04-18.md — added §5.0 bootstrap step,
marked §6-#1/#5 as resolved, added §6-#8/#9 for the structural fixes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two defensive layers between failing models and the live dashboard:
1. Acceptance gate now blocks .pkl save (not advisory).
scripts/_shared/nowcast_acceptance_gate.py — thresholds moved out of
prose into a shared Python helper (MAE < baseline * 0.85, cov90 >= 0.80,
N >= 120). Both trainers evaluate the gate after train_and_validate and
exit 3 before calling save_model when it fails. --validate still prints
the verdict but bypasses the exit so humans can iterate.
Interval halfwidth also switched from Gaussian 1.645σ to empirical p90
of |train residuals| — more honest calibration for heavy-tailed
residuals, and exposes genuine holdout-train variance mismatch instead
of papering over it.
2. fuse filter requires promotion_state ∈ ('shadow','active').
scripts/event-dashboard-api.mjs loadLatestNowcastsForSignals — now
JOINs model_registry and drops any estimated_signal_nowcasts row whose
model version isn't registered as shadow or active. Candidate or
unregistered models accumulate reconciliation data but never reach
buildSignalSummary → dashboard. promote-nowcast-model.mjs is the only
path to surface a model.
Phase C first-run outcome (documented in handoff §5.2): 0/6 targets
cleared the gate. Ridge + ETF proxies can't beat naive carry-forward on
1–3 bp/day FRED moves. oilPrice had +51% MAE but its holdout variance
exceeds train variance — no in-sample interval calibration reaches the
0.80 coverage target. Added §6-#10 and §6-#11 with the structural
implications.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase E partial activation. With 0/6 targets clearing the acceptance gate in the first training run, leaving rates-nowcast on a 30m cron produces only "no trained model" abstains for all 4 rates targets every cycle — pure log noise. Disable by default; operator opts in with NOWCAST_RATES_ENABLED=true once the rates redesign track produces usable models. Other three nowcast tasks (composite-nowcasts, event-intensity-nowcast, reconcile-nowcasts) stay active on restart. Phase D smoke test confirmed they behave correctly with no trained rates models — composite gate- abstains on stale inputs, event-intensity writes valid rows, reconcile no-ops when nothing to pair. Startup banner now logs "[disabled] rates-nowcast ..." so the absence is visible in ops logs rather than silently missing. Handoff §5.4 updated with partial-activation instructions and Phase D smoke test verdict. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase C showed ridge + ETF proxies cannot beat naive carry-forward on slow-moving FRED rate targets — this is an architectural mismatch, not a data or tuning problem, so it belongs in its own track rather than mixed into the operational activation work. The track doc captures the evidence (per-target MAE / cov / N from Phase C validate), the mismatch taxonomy (scale, signal-to-noise, residual distribution, tight baseline), and candidate directions (change-target regression, regime-conditional, conformal intervals, non-linear models, intraday features, or dropping un-beatable targets) without prescribing a ranking. Hard constraints for future rates work: acceptance gate stays enforced (not advisory), promote-nowcast-model.mjs is still the only path to dashboard, and market_quote_coverage.abort_if_missing() must front every new trainer. Timing: pick up 2–4 weeks after the three active nowcast tasks settle in production. Not this session. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Doc inventory + reorganization after the nowcast activation session:
Status markers on superseded design docs
NOWCAST_PLAN_ISSUES_2026-04-17.md — superseded by NOWCAST_HANDOFF_2026-04-18
NOWCAST_ESTIMATION_ARCHITECTURE_PLAN_2026-04-17.md — superseded, all phases
landed across commits 62825c9 / 746a0e5 / 8bce577 / 87c21b6
(Plan doc was also untracked until now — add to git alongside the status.)
DOCUMENTATION.md canonical-docs table
Added NOWCAST_HANDOFF_2026-04-18, NOWCAST_RATES_REDESIGN_TRACK_2026-04-18,
CONNECTED_SYSTEM_WORKFLOW_DETAILED_EXPLAINED_2026-04-18, and
CONNECTED_SYSTEM_WORKFLOW_VISUAL_2026-04-18 so the index matches the
post-nowcast state. Companion render artefacts for the detailed
workflow (.html / .svg / .png / .mmd) added as one collapsed row.
CONNECTED_SYSTEM_WORKFLOW docs
Previously untracked — pulled into git. Narrative explainer covers
layer-by-layer roles, source-add / keyword+theme-add paths, the
observed/estimated split, and which paths actually matter in ops today.
Zoomable HTML + SVG + PNG + Mermaid source are companion renders.
CLAUDE.md — rules 9 through 13
9. Nowcast acceptance gate is enforced (.pkl save refused on fail)
10. fuseNowcastsIntoLookup filters to promotion_state shadow|active
11. Symbol SoT is scripts/_shared/market-quote-symbols.json
12. >5000-row migrations must use UNNEST bulk UPDATE, not per-row loops
13. New external data stores need bootstrap-first, not natural accumulation
Also refreshed the NAS-tables and scripts tables with the nowcast entries
(estimated_signal_nowcasts, nowcast_reconciliation, nowcast_source_eligibility,
model_registry, market_quotes, plus the trainer/compute/gate helpers).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…k INSERT
Yahoo chart API helper consolidation
Three separate implementations (parseYahooQuotePayload + fetchYahooQuote
in refresh-market-quotes-to-nas.mjs, fetchYahooDaily in bootstrap-
market-quotes-history.mjs, seed-side parseYahooChart + fetchYahooChart
ViaRelay in _seed-utils.mjs) had drifted — different timeouts, different
parser output shapes, different error handling.
New scripts/_shared/yahoo-chart.mjs:
fetchYahooChart(symbol, { range, interval, timeoutMs })
parseQuoteSnapshot(data, symbol) — latest-snapshot parser
parseDailyBars(data, symbol) — full daily time series parser
Non-seed consumers now import from the shared module:
refresh-market-quotes-to-nas.mjs — uses fetchYahooChart + parseQuoteSnapshot
bootstrap-market-quotes-history.mjs — uses fetchYahooChart + parseDailyBars
Seed scripts (_seed-utils.mjs, seed-market-quotes.mjs, seed-gulf-quotes.mjs)
intentionally NOT touched — their parsers are tuned to the seed cache
format and migrating them is out of scope for this pass.
parseYahooQuotePayload re-exported from refresh-market-quotes-to-nas.mjs
for backward compat with any external importer of that name.
proposal-executor handleAddTheme bulk INSERT
Previous implementation ran a 3-nested loop (articles × symbols × horizons)
with up to 12k round-trips per large proposal: per-article published_at
lookup, per-(symbol,horizon) price lookup, and per-row INSERT into
labeled_outcomes. Worst case hit ~6000 INSERTs gated by ~6000 price
queries.
Rewrite:
1. One SELECT over articles.id = ANY($1) → Map(id → published_at).
2. One price-series query per symbol covering the full window
[min published_at, max published_at + max_horizon+2d]. Price filter
per (article, horizon) now runs in JS against the in-memory series.
3. Single UNNEST bulk INSERT into labeled_outcomes, ON CONFLICT DO NOTHING.
Round-trips reduced to 1 + resolvedSymbols.length + 1 (~20-ish for a
typical proposal). Existing proposal-executor tests (7) still pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ens, cache, topic-scoring) First wave of the mega-file split planned in NOWCAST_HANDOFF §refactor. Target file drops from 3273 → 2925 LOC (-11%) while staying API-compatible. _shared/dashboard-map-lens.mjs (70 LOC) MAP_LENS_FILTER_TERMS, MAP_LENS_ANCHORS, TRANSMISSION_TARGETS + normalizeLensFilter / normalizeLensText / matchesLensFilter / inferMapLensAnchor. Pure, no DB coupling. _shared/dashboard-cache.mjs (70 LOC) CACHE_DIR + readJsonCache / writeJsonCache / hasRenderableData / toCacheToken / buildCacheKey / hasDynamicSinceParams / buildSinceToken. resolveWithCache stays in main (couples to withMeta + buildJsonResponse). _shared/dashboard-topic-scoring.mjs (160 LOC) TOPIC_ARTICLE_GENERIC_TERMS / TOPIC_ARTICLE_STOPWORDS / GEO_CONTEXT_PATTERNS + sanitizeTopicText / splitTopicTerms / buildTopicArticleProfile / buildTopicRecentArticleScore. Re-exported from event-dashboard-api.mjs to keep tests/event-dashboard-topic-article-matching.test.mjs import path stable. Tests unchanged: 24/24 pass across the three dashboard-touching test files. No behavioural change — pure extraction plus import rewiring. Two more modules left in the planned split (signal-quality, nowcast-fusion). Those land separately because signal-quality is bigger and nowcast-fusion has to land last to reference the extracted classifySignalQuality helpers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… nowcast-fusion
Second wave of the mega-file split. Target drops from 2925 → 2622 LOC
(now 19.9% below its pre-split 3273-LOC starting point).
_shared/dashboard-signal-quality.mjs (299 LOC)
SIGNAL_LABELS / KPI_SIGNAL_CHANNELS / SIGNAL_STALE_THRESHOLD_HOURS,
DATA_TIMESTAMP_KEYS, MODE_STALE_THRESHOLD_HOURS, ALLOWED_RESPONSE_MODES,
OBSERVED_MODES, ESTIMATED_MODES +
toIsoTimestamp, collectPayloadTimestamps, latestInternalTimestamp,
firstTimestamp, inferResponseMode, deriveValueOrigin, signalAgeHours,
classifySignalQuality, deriveResponseMeta, withMeta.
All pure. loadLatestSignalsWithQuality + detectLiveQuoteFeed stay in
main because they need the pg Pool + safeQuery.
_shared/dashboard-nowcast-fusion.mjs (52 LOC)
fuseNowcastsIntoLookup (pure combiner — the one the
tests/nowcast-fusion.test.mjs suite exercises).
loadLatestNowcastsForSignals stays in main — DB-coupled reader +
the shadow/active promotion_state filter.
Re-exports added to main for all four test-imported names
(classifySignalQuality, withMeta, deriveResponseMeta, fuseNowcastsIntoLookup)
so no test file needs a path change.
Net effect of the full split (5 modules, two commits):
3273 → 2622 main LOC (-651, -19.9%)
651 new LOC across 5 _shared/dashboard-*.mjs modules
24/24 tests pass throughout — no behavioural change
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the remaining bulk-INSERT candidates flagged in the 2026-04-18 audit. incremental-event-engine.mjs (STEP 4, ~50-200 rows/run) Per-event INSERT into event_features (26 columns) → single UNNEST bulk INSERT with 26 parallel arrays. Daily daemon cycle now takes one round-trip for the whole event_features refresh instead of N. incremental-event-engine-fast.mjs (STEP 4, parallel path) Same pattern applied to the "fast" variant. Both variants now bulk. proposal-executor.mjs handleAddTheme — 2 remaining loops L934-942 auto_article_themes per-article INSERT → bulk with UNNEST (bigint[], text[], double precision[], text[]). L952-959 auto_theme_symbols per-symbol INSERT → bulk with 6-array UNNEST. Together with the earlier labeled_outcomes rewrite (commit 329e907), handleAddTheme for a large proposal now runs in ~23 queries instead of ~12k. No behavioural change — same ON CONFLICT semantics, same output rows. Existing tests (proposal-executor + proposal-executor-guardrails) pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… 10 plan files
Three sub-docs carved out of CONNECTED_SYSTEM_WORKFLOW_DETAILED_EXPLAINED
per the user's roadmap, each focused on a single path with code +
table-level detail the parent doc intentionally avoids:
WORKFLOW_SOURCE_ADD_PATH_2026-04-18.md
Detectors (discover-emerging-tech, self-heal-sources, backfill-new-sources)
→ quality gate (isLowSignalAddRssProposal) → codex_proposals →
Decision Inbox → proposal-executor handleAddRss → discovered-source-registry,
plus the self-heal cascade into source-repair / source-retire proposals.
WORKFLOW_KEYWORD_THEME_ADD_PATH_2026-04-18.md
Distinguishes system-generated theme candidates (generate-codex-theme-proposals,
fast-keyword-extractor, label-discovery-topics) from operator-proposed add-theme.
Walks the post-rewrite handleAddTheme bulk pipeline: match articles,
auto_article_themes, auto_theme_symbols, labeled_outcomes, sensitivity
matrix — now ~23 queries instead of ~12k. Covers downstream propagation
into event engine + evidence surfaces.
WORKFLOW_OPENCLAW_INTEGRATION_PATH_2026-04-18.md
Compresses the 1402-line OPENCLAW_INTEGRATION_ARCHITECTURE into four
operational surfaces: channels, TaskFlow, source-repair delivery,
scheduler retry escalation, briefing. Emphasises the boundary rule
(Lattice = state of record, OpenClaw = orchestration) and the read-only
tool surface exposed via event-dashboard-api.
DOCUMENTATION.md index updated — all three new docs added to the canonical
table directly below the parent workflow doc.
Status marker normalization (10 files)
Added `> **Status**:` blockquote header to dated phase-plan docs that
were missing one, and converted inline `Status:` lines to the same
blockquote convention. Covers backtest quickstart / deep-dive / explainer,
NAS_BACKTEST_STORAGE_PLAN, TEMPORAL_FEATURE_UPGRADE, ANALYST_WORKSTATION_REFRAME,
CODEX_REMEDIATION_LOOP_PLAN, DYNAMIC_ISSUE_DETECTION_AND_CODEX_REMEDIATION_PLAN,
LIVE_BACKFILL_DATA_BOUNDARY_PLAN, SOURCE_PROPOSAL_INGESTION_REDESIGN.
DOCUMENTATION.md's archive-note convention now matches reality across
all dated plan files.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Backend (7 new API routes):
- /api/event-timeline — 90d events + regime + VIX
- /api/event-narrative/:id — Codex-authored 2-paragraph summary (cached)
- /api/similar-events/:id — pgvector cosine over article embeddings
- /api/regime-scenario — VIX/yield/oil slider predictions
- /api/asset-dossier/:sym — per-ticker theme/uplift/events
- /api/weekly-digest — Codex structured weekly brief
- /api/correlation-breaks — 30d vs 90d Pearson break detector
Frontend — Investigate surface AI Analysis Lab:
- Event Timeline SVG: dot radius ∝ √|t|, hover tooltip (title/theme/
grade/|t|/|uplift|/articles/sources), drag-to-zoom with adaptive
month→week→day ticks, reset-zoom button + double-click reset
- Correlation Breaks: table → 45° scatter (x=corr 90d, y=corr 30d);
color/size by |Δ|; hover tooltip with pair/direction/Δ/values
- Weekly Digest panel, Regime Scenario Lab sliders, Asset Dossier,
clickable timeline dots → peek overlay (with narrative + similar)
Session-scope additions (pre-AI-Lab):
- event-intelligence-builder (hot events / theme-impact / explain-event
/ source-diversity / meta-model-health)
- nowcast-status-builder (model registry + drift + gate verdicts)
- event-decision-alerts webhook emitter
- incremental-event-engine SQL fix (UPDATE FROM JOIN alias correlation)
- plugins/openclaw-lattice-control-plane (5 new MCP tools)
- OpenClaw persona files (IDENTITY/SOUL/TOOLS/USER/HEARTBEAT.md)
- interactive-verify / verify-ai-interactive / screenshot-ai-interactive
Docs cleanup:
- delete 12 shipped/historical plan docs (AUTOMATION_RAG pair, backtest
roadmap + operator quickstart + experiment queue, compute-language,
emerging-tech, gdelt/guardian, resilience, map-redesign, bootstrap +
discovery upgrade logs)
- fix dangling refs in BACKTEST_SYSTEM_EXPLAINER, HANDOFF_BRIEFING,
AGENT_QUICKSTART
- update DOCUMENTATION.md index
Verification: scripts/_shared/verify-ai-interactive.mjs passes 15/15
(hover tooltip, brush zoom, reset, scatter hover, dot sizing, dblclick).
Known: 2 Codex endpoints require CODEX_MODEL env override because
~/.codex/config.toml defaults to gpt-5.5 (not accessible on this account).
Set CODEX_MODEL=gpt-5.4 when spawning the API server.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…, webhook signing Reliability fixes from project audit. Top priorities first. HIGH severity: - intelligence-automation.ts:3262 — worker loop now wraps cycle in try/catch with consecutive-failure exponential backoff (cap 30min). Previous code exited the do-while on any throw, leaving the process alive but never running another cycle. This matches the Apr 22 incident: scheduler started 17:49 KST, produced 0 logs for 4 days while staying alive in the process table — first cycle threw on init, loop died, process kept by open pg pool. - intelligence-automation.ts:863 — writeJsonFile is now atomic (write to .tmp.<pid>.<ts> then rename). Previous naive writeFile could corrupt the 263KB intelligence-scheduler-state.json mid-write if API server's read raced with scheduler's write. MEDIUM severity: - event-dashboard-api.mjs:165 — pg Pool max bumped 6 → 20 (env-overridable), added idle/connect timeouts. AI Lab introduced 7 new routes; dashboard refresh + concurrent users could saturate at max=6. - openclaw-webhook-emitter.mjs:357 — added HMAC-SHA256 signing (x-lattice-signature: t=<ts>,v1=<hex>) alongside the existing Bearer token. Receiver must verify timestamp + body signature to prevent replay/forge if the bearer token leaks. - codex-agent.mjs:150 — removed unconditional shell:true; only enable for .cmd entrypoints on Windows. Prevents stray cmd.exe windows + small metacharacter injection surface in future refactors. LOW severity: - event-dashboard.html:3629 — date math now in UTC (setUTCHours, ms arithmetic) instead of local-midnight + setDate. Fixes ±1 day visual misalignment between event dots (UTC ISO from API) and timeline grid (was local-midnight) for users not in UTC. - scripts/migrations/add-articles-embedding-index.mjs — new idempotent migration. Adds IVFFlat index on articles.embedding (cosine ops). Current scale (73k articles) does ~200ms full scan; not a hard problem yet, but scales linearly. Index is offline-create, run when convenient. - verify-ai-interactive.mjs — pointed at Vite :3000 instead of dead :8088 static helper. Verification: TS check passes, AI Lab interactive regression 15/15 passing. Skipped (false positives from initial audit): - master-daemon PG client release pattern — try/finally is correct. - Frontend listener accumulation — innerHTML rebuild GCs old DOM + listeners. - DEFAULT_ALLOWED_FILES env override — actually Object.freeze hardcoded. - pg pool exhaustion in event-dashboard-api — uses pool.query() (auto release). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…tion, HMAC verify helper 1. package.json: register master-daemon as `npm run daemon` (and `daemon:once`, `daemon:task`). Apr 22 incident root cause was that `master-daemon.mjs` had no npm script — when the dev environment was restarted, only intelligence-scheduler / vite / sidecar came back; master-daemon was silently missed and stayed dead for 4 days. With a script, future restarts will pick it up via the standard launcher pattern. 2. scripts/migrations/add-articles-embedding-index.mjs: now idempotent. First run discovered an existing `articles_embedding_idx` (lists=100, created out-of-band, not committed in any migration script) and created a duplicate `articles_embedding_cos_idx` (lists=200). I dropped my duplicate; updated the script to skip when any IVFFlat index on articles.embedding already exists. Also exposed via `npm run migrate:articles-embedding-index`. 3. openclaw-webhook-emitter.mjs: added `verifyLatticeWebhookSignature()` helper for receivers. Parses the `x-lattice-signature: t=<ts>,v1=<hex>` header, enforces a tolerance window (default 300s replay protection), and uses `timingSafeEqual` for constant-time HMAC comparison. Round-trip tested against 6 cases (valid, wrong secret, tampered body, expired, missing header, missing body — all expected outcomes). Note: webhook receiver in OpenClaw gateway itself is upstream (node_modules/openclaw, not in this repo) and runs on localhost only, so HMAC is defense-in-depth here. Helper exists so future in-repo receivers (e.g., a custom plugin webhook handler) can call it directly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After running full Playwright walkthrough across all five surfaces
(home/inbox/investigate/geo/ops) plus the OpenClaw web UI, found 11
issues. Fixing 7 in this commit; 4 deferred (OpenClaw auth = upstream,
date format unification = invasive, Geo legend counts = needs DeckGL
deep dive, widget loading timeout = needs per-widget audit).
HIGH severity:
- schema-automation.mjs: replace DROP+ADD CONSTRAINT pattern with
idempotent DO block + IF NOT EXISTS check. Old pattern was
non-transactional and raced when multiple workers (master-daemon +
intelligence-scheduler + API server) called ensureAutomationSchema()
concurrently — second writer's ADD failed with
"constraint approval_queue_status_check already exists". This was
surfacing as runtime issues blocking inbox.accept clicks for users.
- master-daemon.mjs: bump auto-pipeline-labels limit 200 → 1500
(env-overridable). 8,435 articles unlabeled vs 18 labeled in 24h
window meant labels were never going to catch up at 200/2h while
ingest is ~800/h. New cadence: 1500 labels per 2h cycle = 750/h,
outpaces ingest rate. Existing backlog drained separately via
one-shot 10k batch (manual).
MEDIUM severity:
- event-dashboard.html renderSelectedProofCard: branch by item.type.
Approval/proposal items now show URL + reason instead of the empty
"DIRECTION -- UPLIFT -- T-STAT -- CONTROLS --" grid that confused
users into thinking the analysis failed. Triage shows keyword pills.
E2 signals keep the full proof chain. Each non-event type also gets
an inline hint pointing at the action buttons in the preview panel.
- DeckGLMap.ts borderStream marquee: replace alarming
"AWAITING NEXT DISCOVERY CYCLE" placeholder with informative
"READY :: live map active · no new source ops this cycle · next
refresh in ~5min". Previously made users think the map was empty
even when 36+ markers were plotted; the marquee is just source-ops
history, not the main data layer.
LOW severity:
- event-dashboard.html loadEventNarrative: include event id + ISO UTC
timestamp ("Event #99523 · 2026-04-26 04:12 UTC") instead of bare
locale time string. Tooltip shows full ISO for copy-paste.
- event-dashboard.html refreshScenarioLab: add column headers
(theme/symbol/avg %/hit rate/×regime) above the predictions list.
Title attribute clarifies semantics: "if event in this theme fires
while regime matches, expected reaction over post-event window."
- event-dashboard.html refreshWeeklyDigest: replace static "60-90s"
placeholder with animated progress bar + "Ns elapsed" counter +
pulsing dot. Bar tops out at 95% (so it never claims completion
before the actual fetch resolves) and clears on both success and
error paths. Prevents users from concluding the page is frozen
during the long Codex round-trip.
Verification: AI Lab interactive regression 15/15 still passing,
TS compile clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same root-cause fix as master-daemon registration: data-accumulator.mjs was a third standalone scheduler (yahoo prices + GDELT backfill into worldmonitor_intel.historical_raw_items) that had no npm script and was silently missing from the running fleet. Discovered via labeling pipeline diagnostic: auto-pipeline step 3 needs forward-window prices to label outcomes, but the warm yahoo-chart store was 24 days stale (max valid_time_start = 2026-04-02 vs today 2026-04-26) because nobody had data-accumulator running. Started one instance as continuous (every 2h, 5 symbols/cycle, 65 total) in this session. Now also exposed via: npm run daemon:accumulator # continuous npm run daemon:accumulator:once # single cycle Coverage will catch up over ~26h as the cycle works through 65 symbols. auto-pipeline-labels limit bump (200→1500) from previous commit will then start clearing the 8,435-article unlabeled backlog. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Audit found 30 widgets across Investigate (17) and Ops (13) surfaces that visually appeared "loading" forever even though they had real content rendered inside. Root cause: render functions populated the elements via innerHTML but never stripped the .loading class, so the .loading CSS (color:transparent + skeleton sweep + ::after "loading" overlay) kept hiding the actual data. Two ways to fix: (a) Touch every render function in 30+ places to strip .loading (b) Make the CSS conditional on actually-empty content This commit takes (b): use :not(:has(*)) so the skeleton applies only when the element has zero child elements (i.e., still in its bootstrap "Loading…" text state). As soon as innerHTML is replaced with a table, chart, or any element-bearing content, the skeleton disappears automatically — no JS bookkeeping required. Verified via Playwright across all surfaces: total .loading elements found: 17 withSkeletonOverContent: 0 ← was 30+ before withoutSkeleton: 17 emptyAndStillLoading: 0 Also commits scripts/_shared/widget-loading-audit.mjs which catches this regression class — future audits can re-run it to detect new widgets that fall into the same trap. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Systematic audit found that 24h master-daemon was running fine, but
4 NAS tables were silently going stale because their feeders had no
schedule. Cascade impact: no new E2-grade events for 41 days.
Audit results (NAS tables on 2026-04-27):
| Table | Stale | Cause |
|--------------------------------------|-------|------------------------------|
| worldmonitor_intel.* (yahoo-chart) | 24d | data-accumulator unscheduled |
| worldmonitor_intel.* (gdelt-doc) | 28d | same |
| worldmonitor_intel.* (fred raw) | 27d | same |
| market_returns | 30d | build-market-returns.py |
| | | unscheduled |
| labeled_outcomes | 27d | cascade from above |
| event_uplift (last 60d window) | 99rows| cascade — matched_controls |
| | | can't build for recent dates |
| model_eval | 16d | train-meta-model unscheduled |
Three-store anti-pattern: same data lives in `market_quotes` (live,
15min refresh — fine), `worldmonitor_intel.historical_raw_items`
(warm store — stale), and `market_returns` (aggregate — stale). Each
has its own feeder with separate schedule. Live works for the dashboard
display, warm + aggregate are needed for the analytical pipeline (event
grading, alpha decay, scenario lab) — those broke.
Changes:
1. data-accumulator.mjs — make YAHOO_BATCH_SIZE / cycle interval /
GDELT rate / sidecar port / max backfill days all env-overridable.
Default batch raised 10 → 30. At 5 syms/2h cycle (the previous
default) the warm store fell 24+ days behind ingest.
2. master-daemon.mjs TASKS — add two new scheduled tasks:
- `build-market-returns` (HOUR_6_MS): rebuilds market_returns from
labeled_outcomes + Yahoo. Runs `python build-market-returns.py`.
Verified live: completed in 5.8s on first cycle.
- `train-meta-model` (WEEK_1_MS): weekly retrain.
`data-accumulator-tick` was tried as an inline task but it blocked
the entire master-daemon for 10+ min per cycle. Removed — comment
in TASKS dict points operators to `npm run daemon:accumulator`
for the standalone continuous worker.
3. build-market-returns.py SQL fix — same UPDATE FROM JOIN alias bug
we fixed in incremental-event-engine.mjs last week was still
present here. PostgreSQL doesn't allow target-table alias in JOIN
ON; moved alias-dependent predicates (`mr.horizon = lo.horizon`,
sector CASE) to the WHERE clause. Without this fix the script
built market_returns rows but failed before updating labeled_outcomes
(caught it on first live run today).
Runtime verification:
- Manual `python scripts/build-market-returns.py` after fix:
SPY: 2,751 returns rows. Sector ETFs: 14,568. abnormal_return
updated 408. sector_return updated 93,274. Coverage 528,423/620,862
(85.1%). market_returns advanced 2026-03-27 → 2026-03-30.
- master-daemon's build-market-returns task ran successfully (5.8s,
ok=true) on first cycle after restart.
- event_uplift grew 341k → 345k (+4k); E2 28,571 → 29,394 (+823);
matched_controls 239k → 242k (+2k) after engine recompute.
Recovery is partial because of cascade: market_returns is bounded by
labeled_outcomes which is bounded by warm yahoo. data-accumulator
standalone (now running batch=30) needs ~4-6h to fully catch up
warm yahoo, after which the rest of the chain unblocks naturally.
ETA for fresh E2 on dashboard: ~12-24h.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… embedding throughput Audit found two more pipeline gaps causing user-visible noise: 1. Hot Events panel was dominated by arXiv-style singleton entries (emerging-tech: 14,515 events / 28 days = 518/day, mostly article_count=1 source_count=1) plus 1,547+ dt-* hash-named dynamic theme codes that hadn't been promoted to canonical taxonomy. Result: dashboard's top 12 was 11x emerging-tech singletons + 1x real graded event. Other themes (cybersecurity, geopolitics, supply-chain-security) were getting clipped out. 2. Embedding refresh was running at 500/cycle but the unembedded backlog had grown to 5,931 articles last 7 days (probably ollama transient failures or rate-limit during catchup). Without embeddings, similarity search and theme classification for those articles is degraded. Changes: - event-intelligence-builder.mjs (buildHotEventsPayload): add `article_count >= 2` AND `theme NOT LIKE 'dt-%'` predicates to both pools (recent-by-volume + top-by-grade) of the UNION query. Singletons aren't bona-fide events — they're individual publications. Dynamic-theme codes belong to the discovery pipeline, not user-facing surfaces. Verified live: top-12 went from 11x emerging-tech singletons to a balanced mix of 8 themes with multi-article confirmation each. - master-daemon.mjs taskGenerateEmbeddings: limit env-overridable with raised default (500 → 2000) and batch overridable. At 2000/h the 5,931-article backlog clears in ~3 cycles, then catches incoming ~800/day comfortably. Previous 500 default was tuned for steady state, not catchup. Verified: - Hot Events theme distribution: clean-energy 2, emerging-tech 2, supply-chain-security 1, cybersecurity 1, space 2, technology-general 1, conflict 2, environment-general 1 (vs previously almost all emerging-tech) - AI Lab interactive regression 15/15 still passing - TS compile clean - Sam Altman E2 (41-day-old) still on top — it's the only graded event in window. Will be replaced naturally as warm yahoo backfill produces fresh E2s in the next 12-24h cascade. Remaining open items (deferred — separate ML PR): - model_registry: 0 rows (ML inference not running) - meta-model-server.py port 8100 not running - canonical_events historical pollution (6,874 dup theme/date groups) — display filter handles user-facing impact; cleanup is optional. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User-value gap: with the validated lane sometimes empty (data state, not
code), the dashboard could show nothing actionable for hours. This commit
adds an always-on trending signal driven by raw canonical_events +
article_count — works even when event_uplift is sparse.
scripts/_shared/event-intelligence-builder.mjs
+ buildTrendingThemesPayload(pool, { windowDays, limit, minArticleCount })
Window-over-window article volume change ranking. Per theme:
events_now / events_prev
articles_now / articles_prev
sources_now
latest_event_at
articlesChangePct (null when prev = 0 → trendStatus 'newly-active')
eventsChangePct
trendStatus surge | rising | stable | cooling | newly-active | no-baseline
(≥100% / ≥25% / ±25% / <-25% bands)
SQL is two CTEs — recent vs previous window — joined and ordered by
pct change DESC, articles_now DESC.
Filters: article_count >= minArticleCount, theme NOT LIKE 'dt-%'.
scripts/event-dashboard-api.mjs — new route:
GET /api/themes/trending?window=7&limit=12&min=2
Returns the ranking. Window capped at 30d, limit at 50.
src/dashboard/sl-banners.mjs — new banner:
buildTrendingBanner(payload) renders top 5 themes as
"↑↑ supply-chain-security (328a · +369%)"
"↑↑ clean-energy (178a · +456%)"
...
Uses arrow icons by trend-status band so the user sees acceleration at
a glance even before clicking through to a theme brief.
Refreshed on every 60-s sl-banners tick alongside ops/status and
hot-events. Always rendered (not just on validated=0) so it complements
the validated lane rather than replacing it.
Live verification on the current data state (no validated/pending events
recently):
/api/themes/trending?window=7&limit=8 returned:
surge biotech now=52 Δ+642.9%
surge aerospace now=61 Δ+510%
surge clean-energy now=178 Δ+456.3%
surge conflict now=99 Δ+395%
surge supply-chain-security now=328 Δ+368.6%
surge monetary-policy now=8 Δ+300%
surge emerging-tech now=246 Δ+261.8%
surge cloud-infrastructure now=137 Δ+234.1%
This is exactly the "엉뚱한 대시보드를 벗어나서 쓸 만한 브리핑 도구" framing
the user pushed for — the dashboard now always shows actionable themes
ranked by real volume change, even when the deeper validation pipeline
is still catching up on the controls + uplift backlog.
68/68 product-side tests still pass; typecheck clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…illars
Four user-facing additions, all hooked into the existing sl-banners overlay
so event-dashboard.html stays untouched.
A1 — Today's Top Decision hero card
src/dashboard/sl-banners.mjs: buildTopDecisionBanner(hot, trending)
Picks first validated > pending > top-trending. Renders title + theme +
grade + score + lane pill, plus a "Show evidence ▾" toggle button. When
validated lane is empty, falls back to top trending theme so the
dashboard always opens with something concrete instead of a blank.
A2 — Evidence drawer expanded inline
attachEvidenceToggleHandlers(stack) binds click on the toggle button.
Calls existing /api/explain-event/<id>, then injects a sibling row with:
- up to 3 uplift rows: "▸ NVDA 1d — grade E2, |t|=2.78, uplift 1.45%, n_controls=3"
- up to 3 article citations: "▸ Reuters: <title>"
- matched controls regime summary: "Matched controls cover regimes: ..."
Re-clicking the button collapses. matched_controls + |t|-stat + uplift +
citations on one screen — the previously-buried evidence is now one
click from the hero.
A3 — /api/dashboard/now-do prescriptive recommendation
scripts/event-dashboard-api.mjs: priority queue:
1. ops critical actionable → "⚠ <condition>" + run command
2. validated > 0 → "✓ N validated signal(s) ready for review"
3. pending > 0 → "○ N pending validation — blocked on controls or t-stat"
4. trending top theme → "✦ <theme> surging — N articles, +X%"
5. idle → "· System idle"
src/dashboard/sl-banners.mjs: buildNowDoBanner renders at the very top
of the stack so the operator always sees one prescriptive line.
A4 — /api/dashboard/health-summary unified pillars
scripts/event-dashboard-api.mjs: 4 pillars (data / pipeline / model /
product) each with their own level (ok/warning/critical) plus an
overall roll-up. Pulls ops/status, product-quality, hot-events.modelTrust
in parallel, classifies each pillar, returns:
overall: 'critical' | 'warning' | 'ok' | 'unknown'
pillars: { data, pipeline, model, product }
actionables: ops actionableInstructions[]
laneCounts, themeFraming
src/dashboard/sl-banners.mjs: buildHealthPillarsBanner renders 4 dots
+ per-pillar one-line detail. Gives the operator (and any external
evaluator) one place to see system health instead of stitching four
endpoints.
sl-banners stack order, top to bottom = priority:
1. Now-do (A3)
2. Today's top decision + evidence drawer (A1+A2)
3. System health pillars (A4)
4. Ops actionable (existing)
5. Hot Events framing + lane breakdown (existing)
6. Trending themes fallback (N7)
Live verification:
/api/dashboard/now-do → "✓ 2 validated signals ready for review"
/api/dashboard/health-summary → overall=warning,
data=warning, pipeline=ok, model=ok, product=ok
Vite serves updated sl-banners.mjs (9 new function symbols visible).
68/68 product-side tests still pass; typecheck clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The user added a domain dictionary (THEME_RELEVANCE_TERMS) to
event-product-score.mjs that gives computeThemeKeywordOverlap a richer
signal than pure tokenization. Two existing tests assumed the old
token-only behaviour:
- "word boundaries" test used a title with 'NVIDIA' under theme 'ai-ml'.
'nvidia' is a legitimate ai-ml domain term, so the new dictionary fires
domain-overlap correctly. Updated the title to one with no domain hits
("Investors gain on consumer chip-free demand surprise") so the test
still pinpoints the word-boundary check.
- "themeRelevance boosts when overlap is strong" expected rationale tag
/keyword-overlap:\d/. The dictionary path emits domain-overlap:NN%, so
the assertion now accepts either prefix: /(keyword|domain)-overlap:\d/.
31/31 N1 tests now pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A 5-step intro modal that runs once per browser (localStorage flag
'lattice-onboarded'). Skips silently on subsequent loads, in iframes,
and during E2E (?e2e=1).
src/dashboard/sl-onboarding.mjs (new)
- STEPS array, 5 entries:
1. Welcome — "Validated market signals with proof, not noise"
2. Five surfaces, one workflow (Home / Inbox / Investigate / Geo / Ops)
3. Lanes tell you trust (validated/pending/watch/noise color coding)
4. Decision Inbox keyboard (J/K/A/R/S) + audit trail
5. Self-monitoring system (health pillars, stale auto-refresh)
- Modal-based (not coachmarks) because the inline 7,971-line dashboard
has dynamic surfaces — positional anchors would break across resize
and surface switches. Plain modal works until the G2 split.
- Esc / click-outside / Enter / Space all advance or dismiss.
- Skip button records "onboarded" so it never shows again.
event-dashboard.html: ONE-LINE addition just before </head>:
<script type="module" src="/src/dashboard/sl-onboarding.mjs"></script>
Verified: vite serves the module (HTTP 200, function symbols visible).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces templated brief content with Codex-generated specific narrative
on demand, with caching + daily budget guard + graceful fallback.
scripts/_shared/ai-analysis-builder.mjs
+ buildThemeNarrativePayload(pool, { theme, period, briefPayload, forceRefresh })
Returns { ok, theme, period, narrative: { whatChanged, whyMatters,
evidence, caveats, monitor, related }, generatedAt, cached }.
+ Cache: theme_narrative_cache table (auto-created), 24h TTL keyed by
(theme, period). PRIMARY KEY allows clean upsert.
+ Cost guard: LATTICE_LLM_DAILY_BUDGET (default 20) — counts today's
cache rows and refuses new generations once exceeded. Returns
{ ok: true, exhausted: true, reason: '...' } so callers can fall
back without retry storms.
+ Codex prompt buildThemeNarrativePrompt: passes the templated brief
as context with explicit "replace, don't paraphrase" instruction.
Output schema is enforced JSON; parsed safely with Array.isArray
checks + slice() bounds (4 lines per section, 6 for evidence).
+ Codex CLI failure (CLI missing, JSON malformed, etc.) returns
{ ok: false, error: '...' } — never throws.
scripts/event-dashboard-api.mjs — /api/theme-brief/<theme>?narrative=llm
+ Detect ?narrative=llm; bypass resolveWithCache (LLM path has its
own cache layer in theme_narrative_cache).
+ enhanceWithNarrative(briefPayload):
- Calls buildThemeNarrativePayload.
- On success, MERGES narrative into briefStructure: replaces a
section's content only when LLM produced ≥1 line for it; original
template stays as fallback for empty sections.
- For evidence: replaces .items but keeps existing .classes from
the templated builder.
- For related: maps narrative array → related.entities.
+ Returns { ...brief, narrativeSource: 'cache'|'llm-fresh'|
'budget-exhausted'|'llm-failed', narrativeGeneratedAt,
narrativeError } so the dashboard can show provenance.
Live verification on /api/theme-brief/clean-energy?period=quarter&narrative=llm:
narrativeSource: llm-fresh (first call) → cache (subsequent within 24h)
Sample LLM output:
whatChanged:
- "Clean-energy coverage snapped back into growth at 382 stories,
reversing the prior quarter's softer trend."
- "It still owned 93.15% of Environment General coverage; the
0.6-pt share dip is minor, not a breakdown."
whyMatters:
- "This is more than noise: volume, +174.82% y/y growth, and
strong category share improved at the same time."
- "The visible corporate signal is deployment-led: shipping
decarbonization activity points to retrofit spending, not
headline-only interest."
caveats:
- "Base effects matter: a weak prior quarter can make the rebound
look cleaner than the underlying demand really is."
- "Breadth is not yet deep; source diversity is 0.8763 and
geography spans 7, so leadership may still be narrow."
monitor:
- "Check whether next-quarter gains spread beyond shipping into
grid, storage, utilities, and industrial efficiency names."
- "Prioritize harder proof: project awards, filings, patents,
earnings commentary, and regulator actions tied to deployment."
This replaces the template "Monitor whether X keeps positive
acceleration" lines with specific, sector-naming, context-aware analysis
— a meaningful step from B-grade brief content toward A-grade.
Default behaviour (no ?narrative=llm) is unchanged so existing dashboards
keep getting templated briefs. Opt-in path lets the dashboard or LLM
narrative track ramp incrementally.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The existing settings.html serves desktop runtime config (Tauri secrets,
Ollama models, runtime feature flags) — that 1,360-line module is for
power-user / desktop deployment configuration. THIS commit adds an
orthogonal "in-app preferences" surface for normal dashboard users:
display language, lane filter default, alert thresholds, polling cadence.
scripts/_shared/user-prefs.mjs (new)
- File-backed at data/user-prefs.json (gitignored).
- Single-user mode for now (user_id='default'); shape is forward-
compatible with the deferred multi-tenant track.
- getUserPrefs / setUserPrefs (validating partial merge) / resetUserPrefs.
- Validation: language ∈ {en, ko}; defaultLane ∈ {all, validated,
pending, watch, noise}; refreshIntervalMs clamped 10..600 s; alert
thresholds clamped to sensible ranges.
scripts/event-dashboard-api.mjs — new /api/user-prefs route:
GET /api/user-prefs[?user=] — read merged-with-defaults prefs
POST /api/user-prefs — partial merge update
DELETE /api/user-prefs — reset to defaults
src/dashboard/sl-prefs.mjs (new)
- Adds a fixed gear button at bottom-left (#sl-prefs-toggle).
- Click → modal with three sections (Display / Alerts / Refresh) +
"Re-show onboarding tour" action.
- Save persists via /api/user-prefs and falls back to localStorage on
API failure so the UX feels responsive offline.
- Reset button confirmable, then calls DELETE.
- Modal styled with the shared token system; honours --bg-surface,
--accent, etc.
event-dashboard.html: ONE-LINE addition before </head>:
<script type="module" src="/src/dashboard/sl-prefs.mjs"></script>
.gitignore: data/user-prefs.json (runtime artifact).
Live verification:
GET /api/user-prefs → defaults
POST /api/user-prefs {language:"ko", ...} → persisted
Vite serves the new module (HTTP 200).
Distinct intentionally from settings.html — that file remains the
desktop runtime config surface; this is the in-app UX preferences pane.
A future cleanup can fold both into a single "preferences" surface, but
the dashboard's WIP state on settings.html makes this slice safer for now.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scripts/build-public-demo-snapshot.mjs (new) — single command to extract
a sanitized N-month slice from NAS PostgreSQL into one JSON file. The
file can be inspected by external evaluators or loaded by a sandbox
build of the dashboard (LATTICE_DEMO_MODE=1, see C3).
What the snapshot contains:
- themes[] theme aggregates (event_count, article_count,
latest_event_date)
- canonicalEvents[] id, theme, title, eventDate, articleCount,
sourceCount (article_count >= 2, no dt-* themes)
- articles[] id, title, source, publisherGroup, publishedAt,
url, marketRelevance, summary (truncated 280ch)
- eventUplift[] canonicalEventId, symbol, horizon, uplift,
tStat, evidenceGrade, nControls
- modelPredictions[] canonicalEventId, symbol, horizon, modelVersion,
alphaProb, expectedAlpha, downsideRisk, timeToPeak,
evidenceGrade, createdAt
- attribution sources list + license note
PII / sanitization:
- email-like and phone-like patterns scrubbed from titles and summaries
- article body truncated to 280 chars (avoids verbatim republication)
- URLs kept (linking to original publishers is fine and required by
most data licenses)
Defaults:
--months=6 window
--output=data/public-demo/lattice-snapshot-<YYYY-MM-DD>.json
.gitignore: data/public-demo/ (regenerated artifact, large)
Live verification with --months=2:
✓ 2.13 MB JSON in 163 ms
counts: 33 themes / 328 events / 3,653 articles / 443 uplift /
3,312 predictions
Sample event: ai-ml "Elon Musk's worst enemy in court is Elon Musk"
articleCount=21, sourceCount=13
Sample uplift: ICLN 2w, t=-8.29, grade=E0, n_controls=2
Sample prediction: GOOGL 1w, alpha_prob=0.518, expected_alpha=2.09
Bug fix during dev: model_predictions schema is (alpha_prob,
expected_alpha, downside_risk, time_to_peak, evidence_grade) — not a
single 'prediction' column. Updated the SELECT.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When LATTICE_DEMO_MODE=1, the API serves the read-only public snapshot
and refuses every mutation. Lets the dashboard run on Vercel / any
static host without NAS access — external evaluators see real signal
output in 5 seconds without us shipping DB credentials.
scripts/_shared/demo-mode.mjs (new)
- isDemoMode() — env LATTICE_DEMO_MODE === '1' | 'true'
- blockIfDemoMode(method) — returns { status:403, body:{...} } for
POST/PUT/DELETE/PATCH; null otherwise.
- loadDemoSnapshot() — finds + caches the latest snapshot from
data/public-demo/ (1h TTL, env override
via LATTICE_DEMO_SNAPSHOT_PATH).
- clearDemoSnapshotCache() — for tests.
scripts/event-dashboard-api.mjs
- resolveEventDashboardResponse intercepts ALL writes early when in
demo mode → 403 with explanatory body.
- New /api/demo/snapshot — metadata + themes (compact).
- New /api/demo/snapshot/<section>?limit=&offset= — paginated section
access (events|articles|uplift|predictions). 2 KB requests, demo-
friendly.
src/dashboard/sl-banners.mjs
- On every refresh tick, fetches /api/demo/snapshot. When demoMode is
true, prepends a warn-style banner at the top of the stack:
"Demo data · read-only · X-month slice · N events / N articles /
N uplift rows. Writes return 403."
docs/SANDBOX_DEPLOY.md (new)
- Local dry-run instructions:
LATTICE_DEMO_MODE=1 node scripts/event-dashboard-api.mjs &
LATTICE_API_BASE=http://127.0.0.1:46200 npm run dev:vite
- Vercel deploy recipe + env vars to set:
LATTICE_DEMO_MODE=1
LATTICE_DEMO_SNAPSHOT_PATH=public/lattice-snapshot.json
- What the evaluator sees / what's blocked / refresh cadence.
Live verification:
✓ LATTICE_DEMO_MODE=1 → /api/demo/snapshot returns
{ ok:true, demoMode:true, counts:{themes:33, events:328,
articles:3653, uplift:443, predictions:3312} }
✓ POST /api/watchlist while in demo mode → 403 with
"Demo mode is read-only. Writes (review / watchlist mutations /
preferences) are disabled..."
✓ Without LATTICE_DEMO_MODE → POST /api/user-prefs returns ok:true
(writes restored).
This unblocks the public sandbox URL track (Sprint 3 Vercel deploy is
configuration, not code).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three Sprint 3 deliverables wrapped into one commit because they
together complete the "investor demo readiness" track.
B1 — /api/model-comparison + model-comparison.html
scripts/_shared/model-comparison.mjs (new):
buildModelComparisonPayload(pool) reads model_eval, groups by
model_version, computes mean and worst across folds for Brier,
ECE, log-loss, top20 precision, alpha hit rate, deflated Sharpe.
Adds a synthetic "naive-random-top20" baseline (Brier 0.25, ECE
0.10, top20 0.20, hit rate 0.50) as comparison anchor.
lift{} envelope exposes Brier improvement %, ECE improvement %,
top20 lift %, hit rate lift % vs the baseline.
scripts/event-dashboard-api.mjs:
GET /api/model-comparison route.
model-comparison.html (new):
- Hero: 4 cards (Active model, Brier improvement, top20 lift,
hit-rate lift) — green for positive, red for negative.
- Table: every model_version side-by-side with mean/worst Brier
and ECE, top20 precision, hit rate, deflated Sharpe, sample count.
Best model highlighted in green; baseline row muted.
- Methodology block: promotion gate (CLAUDE.md §10), walk-forward
details, baseline assumptions.
- Links to / dashboard, raw JSON, /license.html, /data-sources.html.
Live verification:
meta-v1-20260411-0710 Brier 0.216 ECE 0.099 top20 0.250 ← best
meta-v1-20260430-0633 Brier 0.281 ECE 0.162 top20 0.313
meta-v2-20260411-1506 Brier 0.287 ECE 0.267 top20 0.263
naive-random-top20 Brier 0.250 ECE 0.100 top20 0.200
lift over naive: Brier +13.6%, ECE +1.1%, top20 +25%, hit rate +36.5%
B2 — comparison_lift metric in /api/product-quality
scripts/_shared/product-quality-metrics.mjs:
+ computeComparisonLiftMetric(pool) imported from model-comparison.
+ metrics.comparison_lift = mean(brierLift, top20Lift), clamped 0..1
+ details.comparison_lift exposes brierLift, top20Lift, activeModel,
sample count
Live: comparison_lift = 0.193 with brierLift 0.136, top20Lift 0.25.
> 0.10 threshold = "model meaningfully beats naive baseline".
E1 — License / terms / data sources static pages
- license.html: AGPL v3.0 explanation + "what's in the source vs not"
+ "not financial advice" callout.
- data-sources.html: 4 attribution tables — News/RSS, Market, Research,
Models — with each provider's terms link.
- terms.html: appropriate vs inappropriate use cases, no-warranty
callout, demo-mode caveats, liability.
- All three load /src/dashboard/shared/tokens.css for visual coherence
with the dashboard. Cross-link to each other and back to /event-dashboard.html.
vite.config.ts: 4 new HTML entry points added to rollupOptions.input
(modelComparison, license, dataSources, terms) so production builds
emit them as proper static pages.
Verified:
✓ Vite serves all 4 new pages (HTTP 200).
✓ typecheck clean.
✓ 60/60 product-side tests pass.
✓ release-readiness 7 OK + 0 WARN + 0 SKIP. semantic-health FAIL is
data-state (12k stale predictions, modelTrust=disabled) — not a
code regression.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three overlay modules (sl-banners, sl-onboarding, sl-prefs) each duplicated:
- element factory (el / \$)
- API_BASE resolver
- JSON fetch with try/catch
- <style> block injection (~50 lines each, near-identical)
- DOMContentLoaded boot scaffolding
Centralised into:
src/dashboard/shared/dom-utils.mjs (new, 93 lines):
- el(tag, props, children) — class/style/data-*/Node-or-string aware factory
- API_BASE — single resolver from window.LATTICE_API_BASE
- fetchJson(pathOrUrl) — null on non-OK (except 503), never throws
- ensureOverlayStylesheet() — idempotent <link> injection
- deferUntilIdle(callback, delay) — DOMContentLoaded + setTimeout boot helper
src/dashboard/shared/sl-overlay.css (new, 405 lines):
Single shared stylesheet covering banner stack, onboarding modal, and
prefs modal. All selectors use var(--token, fallback) so design-token
changes propagate without code edits.
Net diff:
sl-banners.mjs 603 → 530 (-73)
sl-onboarding.mjs 196 → 142 (-54)
sl-prefs.mjs 281 → 227 (-54)
module subtotal 1080 → 899 (-181)
+ dom-utils.mjs +93
+ sl-overlay.css +405 (replaces ~150 lines of inline CSS across modules)
Wire-size win at runtime: the stylesheet is loaded once (cacheable),
versus three runtime-injected <style> blocks each parsed separately.
Behaviour parity:
- All public exports unchanged (refreshBanners, refreshBriefStructure,
startOnboarding, openPrefsModal).
- DOMContentLoaded timing preserved via deferUntilIdle.
- ensureOverlayStylesheet is idempotent so any module can call it without
fighting another.
Verified:
✓ typecheck clean
✓ Vite serves the new dom-utils.mjs and sl-overlay.css (HTTP 200)
✓ sl-* modules import the new helpers (visible in vite-served output)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
17 routes shared the same try/catch + logger.warn + 500 envelope:
try {
const payload = await build...();
return buildJsonResponse(payload, ...);
} catch (err) {
logger.warn('<route> route failed', { error: String(err?.message || err) });
return buildJsonResponse({ ok: false, error: String(err?.message || err) }, 500);
}
scripts/_shared/route-helper.mjs (new):
- runRoute({ name, logger, build, buildJsonResponse }) — one-shot wrapper.
- build() returning a payload → 200 envelope.
- build() returning { payload, statusCode } → custom status.
- build() throwing → logger.warn + 500 envelope.
- makeRouteHandler(logger, buildJsonResponse) — factory for the bound
`handle(name, fn)` callsite shape.
scripts/event-dashboard-api.mjs:
- Imports makeRouteHandler.
- resolveEventDashboardResponse instantiates `handle = makeRouteHandler(...)`.
- Refactored 6 simple routes to use it:
/api/model-comparison
/api/themes/trending
/api/meta-model-health (custom status: 503 on critical)
/api/explain-event/<id> (custom status: 404 on miss)
/api/theme-impact/<theme> (custom status: 400 on bad input)
/api/source-diversity-audit (custom status: 503 on critical)
Each route loses 4 lines of boilerplate. Routes with more complex
logic (custom error envelopes, multi-method handlers, dynamic since
cache, demo-mode interplay) were left as-is to keep the diff
reviewable.
Boilerplate count:
before: 17 'logger.warn ... route failed' occurrences
after: 11 (6 refactored)
Verified:
✓ typecheck clean
✓ 60/60 product-side tests pass
✓ All 6 refactored routes return HTTP 200 in live testing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The dashboard fires /api/dashboard/now-do, /api/dashboard/health-summary, and the underlying /api/ops/status, /api/hot-events, /api/themes/trending, /api/product-quality endpoints together every 60s. Without caching, the shared heavy SQL builders (ops-status, hot-events, trending-themes, product-quality) ran 2-3 times per tick for identical args. Added scripts/_shared/memoize-payload.mjs with: - memoize(name, builder, ttlMs) returning a wrapper that caches the Promise (so concurrent in-flight calls share one query) - stableArgsKey() that skips pool/client args (anything with .query) - failed-promise eviction so errors don't poison the cache Wired the four heavy builders through memoized closures in event-dashboard-api.mjs, with pool closed over so the cache key is computed from non-pool args only. Live test confirms identical generatedAt within TTL and fresh build after 11s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Foundational token vocabulary for the Palantir/Bloomberg-grade redesign. All --v3-* tokens are additive — older tokens above remain so existing inline event-dashboard.html style block keeps rendering until per-surface modules adopt v3. Tokens added: - Lane (categorical): validated/pending/watch/noise — Radix dark scales - Direction (Bloomberg-semantic, deuteranopia-safe): up=blue, down=red, amber=NEUTRAL data text only (corrects v1/v2 misuse of amber as accent) - Severity: critical/warning/info/ok - Background depth: 5-step slate-tinted scale - Borders: hairline/soft/strong/focus - Text: 4-step luminosity scale (no opacity tricks) - Evidence ladder: luminosity-based E0~E4 (fixes existing opacity-0.25 a11y bug) - Radius: 3-tier (8/12/16) only - Spacing: 4px base scale - Freshness pill: 3-state (Live/Stale/Paused, per Smashing 2025) - Motion: Linear-style timing + 3 easings Also created src/dashboard/v3/index.mjs entry that imports four phase modules; subsequent agents fill them in (parallel-safe, each owns its file). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four phases built in parallel by isolated agents (each owning their files under src/dashboard/v3/), then integrated via a single index.mjs entry and one <script type="module"> tag in event-dashboard.html. Phase 1 — State messaging (337 JS / 234 CSS): - mountFreshnessPill: 3-state Live/Stale/Paused per Smashing 2025 spec - mountSkeleton: Carbon-style staged skeleton (list/card/table/tile) - runOptimistic: SWR-style optimistic UI helper - pulseValue: value-change glow with up/down/neutral direction tint - Auto-attach to [data-v3-freshness] hosts (opt-in, no-op when absent) Phase 2 — Global chrome (662 lines / 5 files): - mountTickerBar: 6-cell status strip (model, articles/24h, validated, gate, ECE, sync); aria-live=off, opt-in via [data-v3-ticker] on body; fetches /api/dashboard/health-summary + /api/ops/status every 60s - mountCheatsheet: ?-toggled slide-in panel (Slack-style, NOT modal), static catalog of all Lattice key bindings - mountHistoryStack: Cmd+[/] back/forward over surface changes, MutationObserver on body[data-surface], 50-entry in-memory stack Phase 3 — Additive surface components (1,119 lines / 7 files): - mountSparkline: pure SVG, no deps - mountTreemap: d3-hierarchy squarified, fallback stacked-bar - mountMatchedDag: hand-rolled SVG DAG for treatment ↔ controls - mountTimeSeries: lightweight-charts wrapper, lazy-imported - mountDenseTable: tabulator-tables wrapper, lazy-imported - All opt-in (no auto-mount), called by surfaces in a future PR - Adds deps: d3-hierarchy ^3.1.2, lightweight-charts ^5.2.0, tabulator-tables ^6.4.0 Phase 4 — Motion primitives (297 JS / 28 CSS): - anim/slideIn/slideOut/fadeIn/fadeOut/pulseGlow/flashOnUpdate/ animateSurfaceSwap — Web Animations API, no deps - Honors prefers-reduced-motion: reduce (read at call time so OS toggles take effect mid-session) - Re-entry on same element + id cancels prior animation cleanly Integration: - src/dashboard/v3/index.mjs imports all four phase entries; each module self-mounts on deferUntilIdle - event-dashboard.html: 7 lines added (link + script tag + ticker opt-in attribute on body) - npx tsc --noEmit passes - Vite serves /src/dashboard/v3/* with HTTP 200, no compile warnings Total: ~2,800 lines across 17 v3 files. Old surfaces unaffected — all new components are opt-in or invoked by future PRs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds src/dashboard/v3/wire.mjs (and wire.css) which scans the live DOM after first idle and attaches phase 1-4 components to known surface selectors — without modifying event-dashboard.html's inline markup or inline <style> block. Removing this file removes only the polish, not any user-facing functionality. Wired in: 1. Freshness pill on each surface's hero card (#now-do-card, inbox stat strip, investigate hero, ops system-health) — falls back silently if the selector doesn't exist 2. .v3-num utility class stamped on .brief-stat-value, .kpi-chip-value, .story-rank, [data-numeric] etc. — tabular-nums + slashed-zero 3. Ops pillar sparklines (60-tick rolling buffer, fed by /api/dashboard/health-summary every 60s) 4. Inbox accept/reject/snooze: optimistic UI (.v3-row-* tints) + directional pulseGlow via capture-phase listener (additive, never cancels the page's own click handler) 5. Surface tab swap: animateSurfaceSwap fade via MutationObserver on body[data-surface] 6. Value pulse on any [data-pulse-on-update] cell — direction inferred from the parsed-number delta (up/down/neutral) wire.css fixes the audit-flagged a11y bug for E0 evidence rows (--text-muted opacity 0.25 was nearly invisible) by overriding to the v3 luminosity-based --v3-ev-e0/e1 ladder. Also adds 32px body padding-top when [data-v3-ticker] is set so the new ticker bar doesn't overlap content, and brightness(0.92) dim on the rest of the page when the cheatsheet sidebar is open. npx tsc --noEmit passes; Vite serves wire.mjs and wire.css at 200. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Refines wire.mjs after grepping the actual event-dashboard.html structure to use selectors that exist: - Freshness pills: #decision-cockpit / #hot-events-section / #lattice-trust-strip etc. (verified existing IDs, with priority-ordered fallbacks so the first match wins) - .v3-num utility class: extended to .tstrip-value, .since-value, .since-delta, .kpi-regime-badge, .count (surface-nav badges) — all numeric cells the monolith renders into - New wirePulseTargets(): stamps data-pulse-on-update on the live KPI cells (#kpi-vix, #kpi-risk, #kpi-spread, #kpi-oil, #kpi-dollar, #tstrip-*-value, #since-inbox, #since-synd, #inbox-total-count) so the existing MutationObserver in wireValuePulse() catches each refresh and pulses with up/down/neutral colour based on the parsed delta The page's own JS rewrites these cells on every poll (60s tick); the pulse glows the cell briefly so the eye catches changes — Geckoboard 2025 rule, "animate changes, not renders". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Upgrades the v3 token palette from "professional-restrained" to "high-end-fancy" matching Linear / Palantir / Vercel's tonal range, and re-routes the legacy --accent / --signal-green / --bg-* aliases via wire.css :root so the entire monolith picks up the new colours without any HTML edit. Tokens — tokens.css: - Lanes: emerald-400 / amber-400 / blue-400 (Linear-saturation, was Radix muted scales) - Direction: sky-400 up / rose-500 down / amber-400 neutral (hot, glowy) - Severity: rose-400 critical (was tomato), so glows pop - Background: indigo-tinted near-black (#08081a → #2a2e47) replaces flat slate (#020617 → #475569). Subtle warm tint reads premium. - Borders: indigo-tinted instead of slate - Evidence E4: mint-cyan #00f5d4 (matrix high-conviction tone) - New signature: cyan-400 primary (--v3-accent), violet-400 secondary (--v3-accent-2), pink-400 special (--v3-accent-3) - New glows: --v3-glow-validated/-critical/-accent (radial box-shadow presets) - New gradients: --v3-grad-aurora (cyan→violet→pink), --v3-grad-tile, --v3-grad-validated/-critical/-violet - New frosted glass: --v3-glass-bg + --v3-glass-blur (20px, 180% sat) Wire.css :root override — re-routes legacy tokens: - --accent: cyan signature (was lime-yellow #d8f99d) - --bg-void/-base/-surface/-elevated: indigo-tinted scale - --signal-green/-amber/-red/-blue: mapped to v3 lane/severity - Effect: all inline monolith CSS that reads var(--accent) etc. now resolves to fancy palette — page-wide shift, zero HTML edit Visual upgrades — wire.css: - Ticker bar: frosted glass + cyan-tinted bottom border + soft glow - Cheatsheet panel: frosted glass + violet edge tint, cyan kbd chips - Freshness pill: live dot has 8+16px emerald glow halo - Validated evidence rows (E3/E4): emerald-mint left bar with bloom - Cockpit/accent cards: aurora gradient (cyan → violet → pink) overlay - Surface nav active tab: cyan underline with linear-gradient + glow - Trust strip values: cyan text-shadow on tabular numbers - Focus ring: 4-layer cyan halo (Linear-style) - Inbox optimistic rows: glow + linear gradient instead of flat tint - Critical banners: rose box-shadow glow - KPI chip values: Bloomberg-correct amber-glow on neutral data - Sparklines: cyan stroke + drop-shadow filter - Theme/trust/since strips: subtle aurora-tone backdrop - Section kickers: linear-gradient text-fill (cyan → violet) prefers-reduced-motion: drops backdrop-filter to avoid GPU cost on low-end devices, keeps glows. npx tsc --noEmit passes; Vite serves all files at 200. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
P0 — must-fix
1. delete 4 Snapshot cards on Home (Risk/Macro/Validation/Source-Ops dup the Trust Strip)
2. inbox actions → "verb → destination" template (Accept→Followed, Reject→Archive, Watch→Tracking…)
3. cross-theme card front shows only claim+lane+score+evidence; novelty/seed-sim/pathway/
diagnostics/secondary actions move into <details>
4. Theme Brief: lifecycle/YoY/accel/source-div collapsed into a single ●●●○ Brief Trust pill
(hover surfaces full numbers + diagnostics); Related Entities + Adjacent Pathways demoted
into a single closed <details> with chip preview on the closed summary
5. Scenario Lab: per-slider why-copy on hover (VIX/Yield/Oil) + no-change explainer that
suggests specific knob moves when no theme classification flips
6. Ops: Today's Events / Issue Temp / Tech Trends / Codex Discoveries / Anomalies /
Pending Validation moved into a collapsed legacy-feeds <details> with destination labels
(→ Inbox, → Investigate); element IDs preserved so existing JS loaders still write
7. empty-state copy pass: "loading X" → "what would be here + how to fill it" rule
P1 — high leverage
1. Home reference drawer summary rewritten ("hot events · evidence chain · alerts · …"),
closed-by-default reaffirmed
2. inbox filter row: state filters → segmented control; type filters → chip strip;
zero-count badges hide via .seg-count.is-zero
3. inbox bulk operations folded behind a "Bulk… ▾" <details> menu with per-type
breakdown ("3 → Followed · 2 → Run · …"); Shift-S = custom snooze duration; B opens menu
4. Investigate: 3-column brief grid above the fold (Theme Brief · Current Regime ·
Scenario Lab); Scenario Lab moved up from AI Lab section
5. Investigate drawers: jump-link tab row above the two reference <details> with renamed
summaries (Discovery·Lifecycle·Breadth·Watchlist | Lab·Asset·Correlation·AI Digest)
6. Personal Tracking: compact chip strip on Home (loadTrackedTargets renders both chips
and full list); add form folded into a <details> with "+ Add tracking target" summary
7. Trust Strip absorbs Since Strip: original since-* IDs hidden but populated; compact
one-line since-summary replaces the multi-pill bar; deltas appended to DATA/SOURCE
pillar tooltips
8. canonical 4-state lattice-dot system (validated/candidate/stale/provisional) — applied
to inbox-list rendering and cross-theme cards; replaces ad-hoc fresh/status mini-chips
P2 — polish
1. scenario URL share: VIX/Yield/Oil encoded in URL params, applied on page load
2. cross-theme "Not useful" → review endpoint posts feedbackKind=not-useful for the
discovery model
3. inboxSnoozeCustom() prompts for hours/days/until-date; Shift-S keyboard shortcut
4. citation copy buttons in showCitationPopover (Markdown link + BibTeX entry)
5. reference drawer last-open persistence via localStorage (home-reference-drawer +
investigate-trend-drawer + investigate-analysis-drawer)
6. keyboard shortcuts: G T (Theme Brief deep link), D (toggle surface drawer), B (Inbox
bulk menu), O (open/collapse all <details> on active surface)
7. first-run modal replaces the "How To Use This Dashboard" home card; F key reopens it;
auto-shown once (gated by localStorage:lattice:first-run-shown)
Net: +1561 / -156 in event-dashboard.html. No JS loaders broken — all renamed/relocated
elements keep their original IDs so background fetchers continue to write to them.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
WHY
The post-IA dashboard still had ~163 inline font-size:9-10.5px hits, text-soft
at 45% opacity, and text-muted at 25% — all of which fail WCAG AA at small
sizes against the bg-void background. Letter-spacing was inconsistent across
uppercase labels and line-height defaulted to ~1.2 (too tight for analyst
reading sessions).
WHAT
1. tokens.css — bumped text opacity to .96 / .80 / .60 / .42 (was .95 / .75
/ .45 / .25). Added a semantic type scale (--fs-caps-xs / --fs-2xs / --fs-xs
/ --fs-sm / --fs-md / --fs-lg / --fs-xl / --fs-2xl), line-height tokens
(--lh-tight 1.2 / --lh-snug 1.35 / --lh-default 1.5 / --lh-relaxed 1.6),
letter-spacing tokens for uppercase, and font-weight tokens (Geist 400-700).
2. event-dashboard.html — inline :root mirrored. Body baseline 12px → 13px.
Added antialiasing + text-rendering:optimizeLegibility. Added a typography
refinement block at the end of the inline <style> that overrides ~30
commonly-used classes:
- section-kicker / brief-section-title / brief-stat-label / tstrip-label →
normalized to --fs-caps-xs (10.5px) with --ls-caps (.1em) and --fw-semi
- section-title → --fs-lg (16px) with --lh-tight
- card-title → --fs-md with --fw-semi
- meta-line / section-meta → --fs-xs (12px) with --lh-default
- badge → --fs-2xs (11px) instead of 9-10px
- inbox-shortcut / inbox-arrow / inbox-dest → 10.5-12px (was 9-10px)
- xtc-* claim, lane chip, score, reason → consistent scale + brighter colors
- brief-trust-pill → 12-15px (was 9.5-13px)
- scenario-empty-explainer / scenario-pred-row → readable
- inbox-filter-* → 11-12px with weight differentiation
- inbox-item-title → 13px with snug line-height
- tracking-chip → 12px with hover lift
- lattice-dot → 8px (was 7px), stronger glow on validated/provisional
- lattice-since-summary → 11px with proper hierarchy
- first-run modal → 13-14px with kbd styling
- cite-popover → readable copy buttons
3. Universal numeric tabular-nums slashed-zero applied to every value cell
(brief-stat-value, kpi-chip-value, tstrip-value, since-value, scenario
predictions, regime KPIs, seg-count, cite-num, etc.).
4. Color refinements:
- Cross-theme lane chips now have borders + 18% bg + brighter foreground
(lane-validated #5eebac, lane-research #7eb6fa, lane-noise #cbd5e1)
- inbox-filter-seg.active gains an inset ring for clearer state
- tracking chips: brighter green text + hover lift
- 4-state lattice-dot: validated green glow tightened, provisional violet
glow added
5. Accessibility:
- focus-visible outline (2px cyan ring) on every interactive element —
keyboard users always see focus
- @media (prefers-contrast:more) overrides text tokens to fully solid
white scale for users who request it
6. Defensive rule — any remaining inline font-size:9-10px gets forced up to
--fs-caps-xs (10.5px) via attribute selector, so legacy markup stays legible
without per-line audits.
7. wire.css — V3 ticker label aligned to --fs-caps-xs and --fw-semi (was 9.5px).
V3 luminosity-based text colors left unchanged — already accessible against
the indigo-tinted backgrounds.
Net: tokens.css +44/-5, event-dashboard.html +120/-6, wire.css +3/-2.
No JS changes. Cascade order ensures this pass overrides earlier rules without
editing them, so reverting is just removing the typography block at the end
of the inline <style>.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
WHY
Reports for subject="cloud-infrastructure" silently picked unrelated rows:
symbol_signal_report -> TLT (long-duration treasury, theme_exposure=0.05)
event_signal_report -> Medicare AI EFF lawsuit
cross_theme_report -> Vertiv (right by accident — top score)
The DB adapter filters were patterns like
WHERE ($1::text IS NULL OR ce.theme = $1)
which silently match all rows when the subject string can't be coerced into
the adapter's expected key shape. A pack labeled "cloud-infrastructure" is
worse than no pack — it actively misleads.
A second issue: theme_trend_aggregates rows reported article_count=0 in the
window with computed acceleration = -145%. The aggregate row was stale vs
the current article→theme mapping, but nothing flagged the inconsistency.
WHAT
1. NEW scripts/_shared/report-subject-fidelity.mjs
- resolveSubjectKey(input) — canonical {kind, key, raw} ('cloud-infrastructure'
classified as kind='theme' instead of being passed through ambiguously)
- buildNoBoundCandidateBundle({reportType, requestedSubject, reason}) —
structurally valid no-data bundle so the report explicitly says "no
subject-bound X candidate for Y" instead of falling back silently
- attachSubjectFidelity(bundle, {requestedSubject, matchStatus, ...}) —
stamps every bundle with metadata.requestedSubject + subjectMatchStatus
(subject-bound | no-bound-candidate | fallback-used | system-wide).
fallback-used appends CAV-SUBJECT-FALLBACK so the warning renders inline.
- classifySubjectMatch — hyphen/underscore tolerance for theme slugs
- isSystemWideReportType — regime + system_quality bypass binding
2. NEW scripts/_shared/report-data-diagnostics.mjs
Detects 5 inconsistency patterns and appends caveats:
- aggregate-orphan: article_count=0 with non-zero YoY/accel/novelty
- evidence-aggregate-mismatch: recent_evidence>0 but article_count=0
- baseline-distortion: |acceleration| > 500% (base period near zero)
- low-sample-overfit: regime_multiplier > 5 with sample_size < 10
- unsampled-tstat: t_stat present without sample_size
Stamps bundle.metadata.diagnosticSignals so quality.mjs can use the
booleans for grade caps.
3. report-db-adapter.mjs (modified, all 6 builders)
- Theme: strict subject filter; allowFallback gate before falling back to
top-by-article-count
- Event: derive theme from subject (kind='theme'); strict ce.theme = $1
filter; allowFallback gate before top-by-uplift fallback
- Cross-theme: include $themeKey = ANY(c.themes) match (theme arrays);
allowFallback gate
- Symbol: when subject is a theme slug, query stock_sensitivity_matrix
WHERE theme = $themeKey (was matching all rows when slug couldn't be
coerced to ticker)
- Regime / system_quality: matchStatus='system-wide' (always correct)
- All builders now call attachSubjectFidelity + applyReportDataDiagnostics
before returning
4. scripts/generate-intelligence-report.mjs
- parseArgs now handles --key=value syntax (was only --key value)
- allowFallback flag wired (default false — strict)
VERIFIED via DB regeneration of cloud-infrastructure pack:
theme_report → cloud-infrastructure (was: cloud-infrastructure by luck;
now: subject-bound + diagnostic flags hasAggregateOrphan=true,
hasEvidenceMismatch=true)
event_signal_report → 92891 "Data Center Tech Lobbyists" (was: 98244
Medicare AI EFF lawsuit — completely unrelated)
symbol_signal_report → GLD@cloud-infrastructure (was: TLT, exposure=0.05;
now: filtered to rows with theme=cloud-infrastructure +
hasOverfitRisk=true diagnostic flagged)
cross_theme_report → 217 Vertiv (still 217, but now via themes[] array
match — Vertiv genuinely is a data-center power company)
Negative case: --subject=nonexistent-fictional-theme → produces
"No event signal report bound to nonexistent-fictional-theme" report
with quality C (was: would have picked Medicare AI as fallback).
This is foundational. Phases 2-5 (data richness, analyst priors, Codex
synthesis, cross-asset linkage) all build on this fidelity layer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…adth, knowledge edges
The Phase 1 theme bundle had 4 metrics + 6 article evidence + a few
regime impact rows. That is enough to verify subject binding but not
enough for prose to say anything specific.
Phase 2 adds:
- top 8 subtopics from theme_evolution (lifecycle stage, momentum)
- up to 6 peer symbols from stock_sensitivity_matrix bucketed by
sign and significance (positive_high / neutral / negative_high)
- multi-regime symbol comparison (regime_conditional_impact grouped
by symbol; only symbols observed in 2+ regimes)
- top 8 knowledge_edges from theme node → connected entities
(suppliers, components, materials, technologies)
- 30-day event timeline from canonical_events + matched
event_hawkes_intensity (surge flag, normalized temperature)
- Hawkes daily intensity series (up to 30d)
The bundle keeps its existing shape — additions ride on the existing
metrics / evidence / marketReactions arrays plus a new
metadata.themeContext snapshot that downstream prose can read.
Verified: cloud-infrastructure bundle now ships with
metrics: 12 (was 4)
evidence: 12 (was 6)
marketReactions: 8 (was 5)
metadata.themeContext.subtopics: 8
metadata.themeContext.peerSymbols.negative: 3
metadata.themeContext.regimeBySymbol: 5
metadata.themeContext.knowledgeConnections: 8 (sample: theme→fiber
via 'requires' relation, confidence 0.58)
metadata.themeContext.events: 10 (sample: VMware Cloud Foundation)
metadata.themeContext.hawkesSeries: 27 daily points
Phase 3 will use this context to generate entity-specific prose
instead of meta-templates.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…st sections
The Phase 1+2 bundle now ships with rich themeContext (subtopics, peer
symbols, multi-regime impacts, knowledge edges, event timeline). Phase 3
makes the prose USE that data.
Before (Phase 1 baseline meta-prose):
"article_count is 0 articles in the week lens; this is the measurement
the report should explain before adding narrative interpretation."
After (Phase 3 typed prose):
"Cloud Infrastructure aggregate row reports article_count=0 but 10
canonical events fired in the last 30 days (0 surges). Read the
aggregate as stale; trust the event timeline and recent evidence
ledger instead."
"Symbol exposure (6 candidates): negative-zscore: GLD, CVX, COP.
Treat zscore<1 candidates as noise."
"GOOGL: +5.22% avg return, t-stat 0.95, sample_size=185. Status:
validated. Multi-period sample — usable as proxy."
"Catalyst lane: Cloud Infrastructure requires data center power
(component, evidence_count=20, source_diversity=9, confidence=0.55).
Watch data center power for theme-specific reads."
"Working stance is bearish-on-attention (subtopic in declining
lifecycle (momentum -100); aggregate row is orphaned — distrust the
multipliers). Bull case: ... Bear case: ... Invalidator: ..."
WHAT
1. NEW scripts/_shared/report-analyst-typed.mjs (~480 lines)
buildThemeAnalystSections — uses themeContext, subtopics, peers,
regimes, knowledge edges, events; produces stance + invalidator +
specific market transmission rows
buildEventAnalystSections — quotes event title, lists sources,
enumerates significant t-stats, defines decision-relevance rule
buildSymbolAnalystSections — names ticker + theme, classifies
exposure as positive/negative/neutral, surfaces overfit risk
buildCrossThemeAnalystSections — bottleneck-vs-adjacency frame;
seed-lock check; substitute-availability invalidator
buildTypedAnalystSections dispatcher
2. report-llm-analyst.mjs (modified)
generateDeterministicAnalystDraft now delegates section-by-section
to typed builder; null sections fall back to generic prose. Pulls
in 12 of 18 sections from typed for theme/event/symbol/cross-theme
reports. Model id changed to 'rule-based-typed-analyst' when typed
sections are populated.
3. report-validator.mjs (modified)
knownNumericStrings now also includes numbers from themeContext
(subtopic momentum/accel/rank, peer zscores, regime multipliers,
knowledge edge confidence/evidence_count, event articleCounts,
hawkes intensity), the sensitivity row, the candidate row, the
event row, plus standard analyst-prior thresholds (1, 1.5, 2, 3,
5, 7, 10, 14, 21, 30, 0.4-0.8). Negative + signed-percent forms
added (+5.22%, -5.22%).
Tokenization strips ISO dates ("2026-05-04") and quoted titles
("404 Media", "VMware Cloud Foundation") so numbers inside them
are not flagged as unknown numeric claims.
COMMON_UPPERCASE_WORDS expanded with business abbreviations (CEO,
CTO, PPA, EDP, AND, NA, MW, GW, GPU, etc.).
4. report-theme-context.mjs (modified)
loadSubtopics: DISTINCT ON sub_theme — keep only latest period per
subtopic (was duplicating with conflicting momentum readings)
loadPeerSymbols: dedup by symbol, keep max-|zscore| row (was
producing GLD twice in negative-zscore list)
loadRegimeImpacts: dedup by (symbol, regime, horizon); keep
symbols with 2+ rows (was filtering out same-regime / different-
horizon pairs)
VERIFIED with cloud-infrastructure pack regeneration:
theme_report: warning grade=B blockers=0
event_signal_report: warning grade=B blockers=0
symbol_signal_report: warning grade=B blockers=0
cross_theme_bottleneck_report: warning grade=C blockers=0
All 4 subject-bound types now produce validation-clean reports with
entity-specific prose. Quality stays at B/C because data caveats
(stale, low source diversity) appropriately cap the grade.
Phase 4 next: hook ai-analysis-builder.mjs as a real LLM provider
behind validator gates for narrative voice on top of the typed sections.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
WHY
The Phase 3 typed sections produce structured, evidence-bound prose, but
they read like a JSON renderer ("Subtopic ABC: rank 4 in parent theme XYZ,
lifecycle declining, momentum -100"). Bridgewater-style narrative requires
synthesis voice that reasons across the lenses (attention / transmission /
graph) and articulates explicit bull / bear cases with invalidators.
WHAT
1. NEW scripts/_shared/report-codex-narrator.mjs (~250 lines)
- buildBundleDigest: compresses bundle into a JSON skeleton suitable
for LLM context (metrics + evidence + reactions + caveats + figures
+ themeContext digest); strips raw article text to keep tokens bounded
- buildTypedDigest: passes the typed sections in as ground truth so
Codex synthesizes ON TOP, not from scratch
- generateCodexNarrative: spawns runCodexJsonPrompt with a strict
analyst prompt that forbids investment language, requires every
numeric claim to be cited from bundle ids, mandates explicit
invalidator thresholds. Returns 4 sections (narrativeHead /
bullThesis / bearThesis / invalidator)
- validateCodexNarrative: per-sentence allow-list check (numbers in
bundle metrics, tickers in marketReactions, no buy/sell phrases).
Hard-drops sentences with unknown numerics or forbidden phrases;
soft-warns on uppercase tokens (English caps were over-filtering
valid prose like "Through the more reliable lenses…")
- narrativeToAnalystBlocks: shape conversion to compiler-ready blocks
2. NEW scripts/_shared/report-validator-helpers.mjs
Extracted knownNumericStrings + allowedTickerStrings so both narrator
and validator share the same allow-lists. Avoids import cycle from
report-llm-analyst → narrator → validator.
3. report-llm-analyst.mjs (modified)
generateReportAnalystDraft now accepts options.provider='codex'.
Always starts from the typed/deterministic draft (the floor); when
codex is requested, generates narrative on top, validates, and
attaches blocks. Codex failure / validation failure falls back to
typed-only with diagnostic metadata (codexAttempted, codexError,
codexDropped).
4. report-compiler.mjs (modified)
HTML + Markdown renderers now emit:
- "Analyst Narrative" section (codexNarrativeHead) above Executive
Brief
- "Bull / Bear Synthesis" section after Working Thesis, with three
sub-headings (Bull thesis / Bear thesis / What would change my
mind)
Both render only when codex sections are present; typed-only reports
are unchanged.
VERIFIED with cloud-infrastructure theme report regeneration:
- Codex call took 3:43; output parsed cleanly
- Narrative head (excerpt):
"Cloud Infrastructure reads as bearish on attention, but the
aggregate is orphaned, so the event timeline and knowledge graph
are the trustworthy lenses rather than the trend row itself. The
core inconsistency is that the weekly aggregate shows 0 articles,
0 percent share, and 0 novelty while still carrying a -145.8333
percent acceleration print, even as the recent evidence ledger
still contains 6 fresh items..."
- Bear thesis ends with explicit invalidator citing MET-THEME-ARTICLES,
MET-PEER-COUNT, MRKT-PEER-GLD-1m, MRKT-PEER-CVX-2w, MRKT-PEER-COP-2w
- 7 sentences hard-dropped (unknown_numeric — full-precision tokens
Codex emitted that aren't in the bundle's exact decimal form), 26
soft-warns (capitalized English words). Remaining narrative is
coherent, evidence-bound, professional analyst voice.
Phase 5 next: cross-asset linkage paths + historical regime analogue
retrieval, so the narrator can frame "this looks like 2017's hyperscaler
cycle" type observations.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…logues
Bridgewater-style framing requires reasoning across hops ("data-center
buildout pressures copper") and historical pattern matching ("this looks
like the 2017 hyperscaler cycle"). Phase 5 supplies the underlying data
so the Codex narrator can produce that framing.
WHAT
1. NEW scripts/_shared/report-cross-asset-paths.mjs (~140 lines)
loadCrossAssetPaths(client, themeKey):
- Locates the theme node in knowledge_nodes
- 1-hop edges via knowledge_edges → connected entities (component,
material, supplier, technology, company)
- 2-hop expansion to entities connected to the 1-hop set, excluding
the theme itself; aggregated path-confidence = product of edge
confidences, top 8 by score
- tagTradableEndpoints: matches endpoint canonical_name against
stock_sensitivity_matrix.symbol; tradable endpoints can be cited
by ticker by the narrator
crossAssetPathsToBundleAdditions: emits MET-CROSS-ASSET-PATH-COUNT
metric + bundle.metadata.crossAssetPaths extension
2. NEW scripts/_shared/report-historical-analogues.mjs (~150 lines)
loadThemeHistoricalAnalogues:
- Pulls full event_hawkes_intensity series for the theme (default
365-day lookback)
- Slices into rolling 30-day windows; computes feature vector
[mean, max, surge_count, stdev] per window
- Cosine similarity between current 30-day window and all past
non-overlapping windows; top 3 analogues
tagAnalogueContext: enriches each analogue with the top 3 canonical
events that fired in that window so the narrator can reference
specific past episodes by title/date
Falls back gracefully if Hawkes history < 60 days (insufficient
data — current cloud-infrastructure case)
3. report-db-adapter.mjs (modified)
buildDbThemeReportBundle now also loads cross-asset paths and
historical analogues; their metric + extension contributions are
merged into bundle.metrics and bundle.metadata.
4. report-codex-narrator.mjs (modified)
buildBundleDigest passes crossAssetPaths and historicalAnalogues
into the LLM context.
NARRATOR_PROMPT extended with:
- Cross-asset framing rules: trace at least one 2-hop path with
relation_type and confidence in the narrative head
- Historical analogue framing: when top similarity > 0.85, cite
the analogue window's start date, similarity, and 1-2 of its
contextEvents
5. report-validator.mjs + report-validator-helpers.mjs (modified)
knownNumericStrings now also accepts path scores + confidence values
from crossAssetPaths and analogue similarity / profile values from
historicalAnalogues, so narrator can cite them without unknown-numeric
blockers.
VERIFIED on cloud-infrastructure (typed-only, no Codex call):
Cross-asset paths: 6 paths surfaced
- data center (requires, conf 0.58) → end
- cooling (requires, conf 0.58) → end
- fiber (requires, conf 0.58) → end
- 3 more 1-hop entities
2-hop expansion produced 0 paths because the existing knowledge_edges
graph is shallow for cloud-infrastructure — most components don't have
onward edges yet. This is a data-density issue, not a code bug. As the
research-os pipeline accumulates more edges, 2-hop paths will populate
automatically.
Historical analogues: skipped — 45 days of Hawkes history available,
needs 60+ for 2 non-overlapping 30-day windows. Will activate once the
event_hawkes_intensity table has 60+ days for the theme.
Total bundle metrics: 13 (Phase 1 baseline = 4)
+9 metrics added across Phase 2/3/5 enrichments
The report system is now structurally ready to produce Bridgewater-style
narratives whenever the data is dense enough. The remaining gap is data
maturity (knowledge_edges 2-hop density, multi-month Hawkes history),
not pipeline architecture.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Owner
Author
|
. |
…tion artifacts - Make visualStatus the primary dashboard/API readiness source - Demote productTier to evidence-tier display when closure is blocked - Reconcile raw discovery coverage against the accepted evidence matrix - Block pre-reconciliation artifacts from appearing review-ready - Add contradiction warnings and regression tests - Document readiness source-of-truth in the automation runbook
- Add mechanism seed generation, storage, evidence planning, closure, provider gap review, adapter proposals, and daemon/self-improvement flows - Wire research seed and provider gap review surfaces into the dashboard API/UI - Extend report backfill/provider execution and frontier parent readiness behavior - Update GitHub README, script guide, and runtime artifact ignore rules - Add regression coverage for seed lifecycle, provider gap closure, dashboard review, valuation readiness, and frontier parent flow
Owner
Author
|
Closing this oversized PR in favor of the split stacked PR set:
The split removes the generated/cache/backup churn from the review path and keeps each follow-up layer reviewable against its direct base. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Type of change
Affected areas
/api/*)Checklist
site/and/ordocs/for any public-facing behavior changenpm run docs:buildnpm run public:syncif this should be reflected in the public repo or siteapi/rss-proxy.jsallowlist (if adding feeds)npm run typecheck)Screenshots