Add product analytics as an evidence source and experiment-design mechanism

README.md

@@ -271,8 +271,8 @@ vocabulary:
   lens: [Commercial, Consumer, Investor]   # your audiences
   area: [Product, Go-to-market, ...]       # your domains
   audiences: [End user, Investor, Partner, Internal]
-evidence_sources: [web]       # + fireflies, slack, gmail, attio — whatever
-                              #   your harness actually has connected
+evidence_sources: [web]       # + fireflies, slack, gmail, attio, analytics
+                              #   — whatever your harness actually has connected
 ```
 
 No config at all still works: local files, web-only evidence. Skills

docs/writing-connectors.md

@@ -6,6 +6,18 @@ documents, not code: the **runtime connector** doc translates abstract registry
 operations into your backend's concrete tool calls, and the **schema guide**
 doc tells `/setup-validation-os` how to build or validate the backend.
 
+> **Note — this doc is for registry storage backends only.** Product
+> analytics tools (PostHog, Mixpanel, BigQuery, warehouse SQL) are
+> deliberately **not** modeled as a connector — their event/property
+> schemas are workspace-defined and discovered at runtime, not a fixed
+> field list a maintainer could map once (the schema-guide contract below
+> assumes the latter). They're wired in as an evidence source +
+> experiment-design mechanism instead:
+> `skills/find-evidence/references/product-analytics.md` and
+> `skills/experiment-design/references/analytics-instrumentation.md`,
+> sharing `skills/_shared/analytics-metric-resolution.md`. Don't build a
+> `connectors/posthog.md` — there's no stable schema to map.
+
 ## Before you start
 
 Read, in order:

examples/08-query-the-metrics.md

@@ -0,0 +1,60 @@
+# 08 · Query live telemetry for evidence — `/find-evidence` (analytics)
+
+Three weeks after DEC-002 launched flat £40/mo, Ledgerly connected
+Mixpanel and added `analytics` to `evidence_sources`. ASM-004 ("owners
+will pay £40/mo") still sits at Confidence 15 — interview-only, from
+EXP-003's Anecdotal rung. Real subscriptions can now settle it directly.
+
+**You type:** "check Mixpanel for how ASM-004 is actually holding up"
+
+The skill resolves ASM-004, and this time the flavour is obvious — a
+measured metric beats another round of interviews:
+
+▎ "Owners will pay £40/mo" is now directly measurable — real charges beat
+▎ stated intent. Sweeping analytics.
+
+Business context first, then the glossary — empty, this is the first
+analytics sweep — so it falls to introspection, and introspection comes
+back ambiguous:
+
+▎ Mixpanel lists two candidate events for "paying £40/mo": `Subscription
+▎ Started` (fires on card charge) and `Plan Selected` (fires on choosing a
+▎ tier, before payment succeeds). Which counts as the evidence for
+▎ ASM-004 — the charge, or the selection?
+
+You confirm `Subscription Started` — the costly one. The mapping is
+written to `analytics-glossary.md`:
+
+```markdown
+| owner pays £40/mo | mixpanel | event `Subscription Started`, `plan=flat_40` | 2026-08-25 | alex | ASM-004 |
+```
+
+The query runs — window 2026-08-04→2026-08-25 (launch to today), segment
+= qualified ICP trial starts, denominator = the same — and the rigor
+check runs before anything is written: definition stable (event unchanged
+since launch), no concurrent price change or promo in the window, N = 34
+trials, well above the floor.
+
+▎ 24 of 34 qualified ICP trials converted at flat £40/mo (71%) — no
+▎ confound in the window. Clears `We're right if` comfortably.
+
+One gated write:
+
+```markdown
+## EXP-004: Are ICP owners actually paying £40/mo, post-launch?
+- **Assumption**: ASM-004 · **Type**: Paying users
+- **Result**: Validated       · **Date**: 2026-08-25
+- **Source**: Mixpanel — `Subscription Started`, plan=flat_40,
+  2026-08-04→2026-08-25, 24/34 qualified trials
+```
+
+Confidence on ASM-004: 15 → 99 (the rollup is a `max` over proven
+evidence — the strongest concluded experiment wins, weak evidence doesn't
+stack).
+
+**What this shows:** telemetry is read live through Mixpanel's own tools,
+never a hardcoded event name; an ambiguous mapping is resolved once,
+confirmed, and cached; and the rigor rubric (confound check, stable
+definition, sufficient N) runs before the verdict, not after.
+
+Next: [09 — pre-register a live A/B](09-instrument-the-experiment.md).

examples/09-instrument-the-experiment.md

@@ -0,0 +1,58 @@
+# 09 · Pre-register a live A/B as the pass bar — `/experiment-design` (analytics instrumentation)
+
+A new belief is on the queue: ASM-008, "existing multi-bank ICP owners
+will still convert at £55/mo for multi-account support" — untested, Risk
+68, Metric for truth already countable.
+
+**You type:** "design an experiment for ASM-008"
+
+Rung and feasibility are weighed first, same as any design:
+
+▎ Ideal rung: Paying users (~99) — a live price test, not a survey; we
+▎ already have the payment flow and Mixpanel wired in. Feasibility: High.
+▎ Recommend Paying users directly — no need to step down.
+
+Both bars are pre-registered before anything runs, then step 7's router
+asks the one new question:
+
+▎ Measured live in Mixpanel, or by a hand-run session? — Live routes to
+▎ `analytics-instrumentation.md`.
+
+Resolving the metric hits the glossary from EXP-004 — reused, not
+re-asked:
+
+▎ Glossary has a confirmed mapping: `Subscription Started` (event still
+▎ live, re-checked). Reusing it — same charge event, new segment.
+
+The instrument is pre-registered into the record body before the test
+starts:
+
+```markdown
+## Analytics instrumentation — Do multi-bank ICP owners convert at £55/mo?
+
+### Instrument
+- Platform: Mixpanel · Metric/event: `Subscription Started`, plan=flat_55 (treatment) vs flat_40 (control)
+- Segment: ICP trials with ≥2 connected bank accounts · Denominator: qualified multi-bank trial starts
+- Window: 2026-09-01 → 2026-09-29, fixed · Min qualified N: 30/arm
+
+### Native experiment object
+- Type: A/B (feature flag) · Key: `price-test-multibank-55` · Link: <mixpanel experiment URL>
+
+### Signal → bar
+- We're right if: treatment conversion ≥ control conversion − 5pp
+- We're wrong if: treatment conversion drops ≥10pp vs control
+```
+
+Mixpanel's own pre-launch checks run before the flag goes live — sample
+size and traffic-split sanity, not a hand-rolled one. Two gated writes:
+create EXP-005, then flip ASM-008 `Status → Testing`.
+
+▎ Let it run to 2026-09-29, then close out via `/find-evidence`'s
+▎ analytics flavour — re-run `Subscription Started` segmented by arm,
+▎ don't redefine it.
+
+**What this shows:** the mechanism question (live vs. hand-run) is
+orthogonal to the rung choice already made; the metric glossary is reused
+across skills without re-resolving the same ambiguity twice — proving the
+"resolved once" claim concretely; and the pre-registered instrument is the
+literal pass/kill bar, fixed before the flag goes live.

examples/README.md

@@ -21,6 +21,8 @@ behaviour the scene demonstrates.
 | [05](05-design-the-experiment.md) | Design the test | `/experiment-design` |
 | [06](06-meeting-prep.md) | Prep the interview | `/meeting-prep` |
 | [07](07-log-the-decision.md) | Log the decision | `/decisions` (capture) |
+| [08](08-query-the-metrics.md) | Query live telemetry for evidence | `/find-evidence` (analytics) |
+| [09](09-instrument-the-experiment.md) | Pre-register a live A/B | `/experiment-design` (analytics instrumentation) |
 
 Not shown: the maintenance entry points (assumptions audit / loop / triage,
 decision sweep / audit, terminology) — each skill's SKILL.md mode table

skills/_shared/analytics-metric-resolution.md

@@ -0,0 +1,121 @@
+# Shared helper — analytics metric resolution
+
+Read by `../find-evidence/references/product-analytics.md` (retrospective —
+find a metric that already has an answer) and
+`../experiment-design/references/analytics-instrumentation.md` (prospective
+— pre-register a metric before the test runs). Both need the same thing
+first: turn an assumption's **Metric for truth** into one concrete,
+queryable metric/event on a connected product-analytics platform — without
+this repo ever hardcoding what that platform calls things.
+
+**Delegate, don't reinvent.** validation-os does not ship its own schema
+introspection or query execution for PostHog, Mixpanel, BigQuery, or any
+other analytics platform — those platforms already publish their own agent
+skills that do this well (PostHog: `PostHog/ai-plugin`; Mixpanel:
+`mixpanel/ai-plugins`; others: search [skills.sh](https://skills.sh)). This
+file is the glue between *their* introspection/query tools and *this
+repo's* evidence-ladder discipline — it never contains a platform-specific
+API call, event name, or query syntax. If a step below can't be done
+because no platform skill/MCP tools are connected, stop and say so
+(`connectors/SPEC.md`'s fail-loudly rule) rather than improvising a raw API
+call.
+
+---
+
+## Inputs
+
+```yaml
+metric_for_truth: <string>   # the claim to resolve into a queryable metric
+lens:             <from config vocabulary.lens>
+platform:         <config analytics.platform, free text — e.g. "mixpanel">
+glossary_file:    <config analytics.glossary_file, resolved path>
+```
+
+## Procedure
+
+### 1. Business-context call first
+
+If the connected platform skill exposes a business-context / vocabulary
+tool (e.g. Mixpanel's `Get-Business-Context`), call it before anything
+else — read-only, cheap, and it resolves project nicknames and internal
+acronyms you can't infer from a metric name alone. Not every platform has
+one; skip if none exists.
+
+### 2. Check the glossary for a prior confirmed mapping
+
+Read `glossary_file` (§ format below). A row matching this
+`metric_for_truth` concept → candidate mapping found. **Re-validate before
+trusting it**: one cheap list/describe call to confirm the named
+event/property/metric still exists. Telemetry schemas drift — a renamed or
+removed event silently breaks a stale mapping. Confirmed live → use it,
+skip to §5. Gone or changed → fall through to §3 as if no mapping existed,
+and flag the stale row for cleanup at the write-back step (§6).
+
+### 3. No mapping — resolve via the platform skill's own introspection
+
+Ask the connected platform skill's list/search/describe tools (events,
+properties, metrics — whatever it exposes) for candidates whose name or
+description plausibly matches `metric_for_truth`. **Never assume a name**
+("signup", "activation") exists — every workspace names things
+differently. Gather enough detail per candidate to disambiguate: name,
+description if available, a sample property value or two.
+
+### 4. Ambiguous or empty — gated clarifying question
+
+Zero candidates, or more than one plausible: stop and ask — show what was
+found (name + description + sample), never silently guess or pick the
+first result. On the human's answer, proceed to §5 with the confirmed
+mapping. If genuinely nothing plausible exists on the platform, say so
+plainly; that's a valid outcome, not a failure to keep searching.
+
+### 5. Return the resolved metric
+
+Hand back to the caller: the platform, the resolved
+event/property/metric definition, and whether it came from the glossary
+(§2) or a fresh resolution (§3–4). The caller (`product-analytics.md` or
+`analytics-instrumentation.md`) takes it from here — running the query or
+pre-registering it as the instrument, and applying
+`quant-analytics-rubric.md`.
+
+### 6. Write back to the glossary — gated
+
+Only when §3–4 ran (a fresh resolution, confirmed by a human) or an old
+row was flagged stale in §2. Append or update one row
+(`gated-writes.md` — this is a real file write, just not to the registry).
+Never write speculative or unconfirmed mappings.
+
+---
+
+## The glossary file
+
+Workspace-adjacent, not registry content — resolved like
+`self_review.review_dir` (`skills/self-review/references/storage.md`),
+never routed through the registry connector. Default `analytics-glossary.md`
+next to `validation-os.config.yaml`; not private (no gitignore offer) —
+it's shared team knowledge, not a personal record. Created on first
+write-back, not eagerly scaffolded at setup.
+
+```markdown
+# Analytics glossary — resolved event/metric mappings
+Confirmed once by a human, reused after. Re-validated (existence check,
+not blind trust) before reuse — telemetry schemas drift.
+
+| Concept (as it appears in a Metric for truth) | Platform | Resolved definition | Confirmed | By | Notes |
+|---|---|---|---|---|---|
+| owner completes signup | mixpanel | event `Signup Completed`, `platform=web` | 2026-07-09 | benji | ASM-004 |
+```
+
+One row per concept. A row whose definition no longer resolves (§2) is
+updated in place at the next successful re-resolution — never silently
+left stale.
+
+## Never
+
+- Never hardcode a platform's event/property/metric name in this file, in
+  `product-analytics.md`, or in `analytics-instrumentation.md` — every
+  example above is illustrative, not a default to assume.
+- Never trust a glossary row without a live re-check — schemas drift.
+- Never guess a mapping when candidates are ambiguous — gate it (§4).
+- Never call a platform's raw API directly when no platform skill/MCP
+  tools are connected — say so and point at installing one, don't
+  improvise.

skills/_shared/experiment-guardrails.md

@@ -196,7 +196,10 @@ The checklists below are the **minimum bar**; the full prep skeletons live
 in `/experiment-design`'s `references/` — `interview-guide.md` (guide
 skeleton + how-to-ask rules, also the question discipline `survey.md`
 inherits), `prototype-brief.md` (prototype-needed rule table §3b + brief
-template), `fake-door.md` (stimulus / costly ask / instrumentation spec).
+template), `fake-door.md` (stimulus / costly ask / instrumentation spec),
+`analytics-instrumentation.md` (live-telemetry / native A-B pre-
+registration — a mechanism fork of the Revealed-tier rungs, not a new rung;
+rigor rubric: `quant-analytics-rubric.md`).
 
 ### 🔵 Desk research (🟡 `Desk research` rung)
 

skills/_shared/historic-evidence.md

@@ -50,7 +50,11 @@ sources:    <list>              # WHICH flavour(s) to sweep — see below
 **`sources` is the flavour switch.** Internal values (call transcripts,
 chat, email, notes, CRM — whatever the config's `evidence_sources` declares;
 per-source search guidance lives in `/find-evidence`'s `references/`) sweep
-your own record; `web` sweeps published secondary sources (desk research).
+your own record; `web` sweeps published secondary sources (desk research);
+`analytics` — when configured — searches by discover-then-query against a
+connected product-analytics platform
+(`../find-evidence/references/product-analytics.md`) rather than keyword
+search across text.
 **Default when a caller omits `sources`: the configured internal sources**,
 so desk research is strictly opt-in. **When `web` is in `sources`, the
 search + verification must follow `desk-research-rubric.md`** — source
@@ -117,7 +121,9 @@ its work.
   `Desk research`; unpaid real use → 🟢 `Prototype usage`; payment / signed
   commitment observed → 🟢 `Signed intent` / `Paying users`. A *measured*
   product metric that bears on the Metric for truth is real behaviour —
-  revealed-tier, with the analytics view as the source.
+  revealed-tier, discovered and queried per
+  `../find-evidence/references/product-analytics.md`, verified against
+  `quant-analytics-rubric.md` before writing.
 - **`Source quality`** = High / Medium / Low for the source's seniority /
   decision authority / ICP-fit. Modulates Strength **within** the rung,
   never across it.