diff --git a/README.md b/README.md index ec61bfb..d1da81c 100644 --- a/README.md +++ b/README.md @@ -271,8 +271,8 @@ vocabulary: lens: [Commercial, Consumer, Investor] # your audiences area: [Product, Go-to-market, ...] # your domains audiences: [End user, Investor, Partner, Internal] -evidence_sources: [web] # + fireflies, slack, gmail, attio — whatever - # your harness actually has connected +evidence_sources: [web] # + fireflies, slack, gmail, attio, analytics + # — whatever your harness actually has connected ``` No config at all still works: local files, web-only evidence. Skills diff --git a/docs/writing-connectors.md b/docs/writing-connectors.md index d22ee83..5363a65 100644 --- a/docs/writing-connectors.md +++ b/docs/writing-connectors.md @@ -6,6 +6,18 @@ documents, not code: the **runtime connector** doc translates abstract registry operations into your backend's concrete tool calls, and the **schema guide** doc tells `/setup-validation-os` how to build or validate the backend. +> **Note — this doc is for registry storage backends only.** Product +> analytics tools (PostHog, Mixpanel, BigQuery, warehouse SQL) are +> deliberately **not** modeled as a connector — their event/property +> schemas are workspace-defined and discovered at runtime, not a fixed +> field list a maintainer could map once (the schema-guide contract below +> assumes the latter). They're wired in as an evidence source + +> experiment-design mechanism instead: +> `skills/find-evidence/references/product-analytics.md` and +> `skills/experiment-design/references/analytics-instrumentation.md`, +> sharing `skills/_shared/analytics-metric-resolution.md`. Don't build a +> `connectors/posthog.md` — there's no stable schema to map. + ## Before you start Read, in order: diff --git a/examples/08-query-the-metrics.md b/examples/08-query-the-metrics.md new file mode 100644 index 0000000..4d291f2 --- /dev/null +++ b/examples/08-query-the-metrics.md @@ -0,0 +1,60 @@ +# 08 · Query live telemetry for evidence — `/find-evidence` (analytics) + +Three weeks after DEC-002 launched flat £40/mo, Ledgerly connected +Mixpanel and added `analytics` to `evidence_sources`. ASM-004 ("owners +will pay £40/mo") still sits at Confidence 15 — interview-only, from +EXP-003's Anecdotal rung. Real subscriptions can now settle it directly. + +**You type:** "check Mixpanel for how ASM-004 is actually holding up" + +The skill resolves ASM-004, and this time the flavour is obvious — a +measured metric beats another round of interviews: + +▎ "Owners will pay £40/mo" is now directly measurable — real charges beat +▎ stated intent. Sweeping analytics. + +Business context first, then the glossary — empty, this is the first +analytics sweep — so it falls to introspection, and introspection comes +back ambiguous: + +▎ Mixpanel lists two candidate events for "paying £40/mo": `Subscription +▎ Started` (fires on card charge) and `Plan Selected` (fires on choosing a +▎ tier, before payment succeeds). Which counts as the evidence for +▎ ASM-004 — the charge, or the selection? + +You confirm `Subscription Started` — the costly one. The mapping is +written to `analytics-glossary.md`: + +```markdown +| owner pays £40/mo | mixpanel | event `Subscription Started`, `plan=flat_40` | 2026-08-25 | alex | ASM-004 | +``` + +The query runs — window 2026-08-04→2026-08-25 (launch to today), segment += qualified ICP trial starts, denominator = the same — and the rigor +check runs before anything is written: definition stable (event unchanged +since launch), no concurrent price change or promo in the window, N = 34 +trials, well above the floor. + +▎ 24 of 34 qualified ICP trials converted at flat £40/mo (71%) — no +▎ confound in the window. Clears `We're right if` comfortably. + +One gated write: + +```markdown +## EXP-004: Are ICP owners actually paying £40/mo, post-launch? +- **Assumption**: ASM-004 · **Type**: Paying users +- **Result**: Validated · **Date**: 2026-08-25 +- **Source**: Mixpanel — `Subscription Started`, plan=flat_40, + 2026-08-04→2026-08-25, 24/34 qualified trials +``` + +Confidence on ASM-004: 15 → 99 (the rollup is a `max` over proven +evidence — the strongest concluded experiment wins, weak evidence doesn't +stack). + +**What this shows:** telemetry is read live through Mixpanel's own tools, +never a hardcoded event name; an ambiguous mapping is resolved once, +confirmed, and cached; and the rigor rubric (confound check, stable +definition, sufficient N) runs before the verdict, not after. + +Next: [09 — pre-register a live A/B](09-instrument-the-experiment.md). diff --git a/examples/09-instrument-the-experiment.md b/examples/09-instrument-the-experiment.md new file mode 100644 index 0000000..27b834b --- /dev/null +++ b/examples/09-instrument-the-experiment.md @@ -0,0 +1,58 @@ +# 09 · Pre-register a live A/B as the pass bar — `/experiment-design` (analytics instrumentation) + +A new belief is on the queue: ASM-008, "existing multi-bank ICP owners +will still convert at £55/mo for multi-account support" — untested, Risk +68, Metric for truth already countable. + +**You type:** "design an experiment for ASM-008" + +Rung and feasibility are weighed first, same as any design: + +▎ Ideal rung: Paying users (~99) — a live price test, not a survey; we +▎ already have the payment flow and Mixpanel wired in. Feasibility: High. +▎ Recommend Paying users directly — no need to step down. + +Both bars are pre-registered before anything runs, then step 7's router +asks the one new question: + +▎ Measured live in Mixpanel, or by a hand-run session? — Live routes to +▎ `analytics-instrumentation.md`. + +Resolving the metric hits the glossary from EXP-004 — reused, not +re-asked: + +▎ Glossary has a confirmed mapping: `Subscription Started` (event still +▎ live, re-checked). Reusing it — same charge event, new segment. + +The instrument is pre-registered into the record body before the test +starts: + +```markdown +## Analytics instrumentation — Do multi-bank ICP owners convert at £55/mo? + +### Instrument +- Platform: Mixpanel · Metric/event: `Subscription Started`, plan=flat_55 (treatment) vs flat_40 (control) +- Segment: ICP trials with ≥2 connected bank accounts · Denominator: qualified multi-bank trial starts +- Window: 2026-09-01 → 2026-09-29, fixed · Min qualified N: 30/arm + +### Native experiment object +- Type: A/B (feature flag) · Key: `price-test-multibank-55` · Link: + +### Signal → bar +- We're right if: treatment conversion ≥ control conversion − 5pp +- We're wrong if: treatment conversion drops ≥10pp vs control +``` + +Mixpanel's own pre-launch checks run before the flag goes live — sample +size and traffic-split sanity, not a hand-rolled one. Two gated writes: +create EXP-005, then flip ASM-008 `Status → Testing`. + +▎ Let it run to 2026-09-29, then close out via `/find-evidence`'s +▎ analytics flavour — re-run `Subscription Started` segmented by arm, +▎ don't redefine it. + +**What this shows:** the mechanism question (live vs. hand-run) is +orthogonal to the rung choice already made; the metric glossary is reused +across skills without re-resolving the same ambiguity twice — proving the +"resolved once" claim concretely; and the pre-registered instrument is the +literal pass/kill bar, fixed before the flag goes live. diff --git a/examples/README.md b/examples/README.md index 731c834..9c3fc0c 100644 --- a/examples/README.md +++ b/examples/README.md @@ -21,6 +21,8 @@ behaviour the scene demonstrates. | [05](05-design-the-experiment.md) | Design the test | `/experiment-design` | | [06](06-meeting-prep.md) | Prep the interview | `/meeting-prep` | | [07](07-log-the-decision.md) | Log the decision | `/decisions` (capture) | +| [08](08-query-the-metrics.md) | Query live telemetry for evidence | `/find-evidence` (analytics) | +| [09](09-instrument-the-experiment.md) | Pre-register a live A/B | `/experiment-design` (analytics instrumentation) | Not shown: the maintenance entry points (assumptions audit / loop / triage, decision sweep / audit, terminology) — each skill's SKILL.md mode table diff --git a/skills/_shared/analytics-metric-resolution.md b/skills/_shared/analytics-metric-resolution.md new file mode 100644 index 0000000..3dced20 --- /dev/null +++ b/skills/_shared/analytics-metric-resolution.md @@ -0,0 +1,121 @@ +# Shared helper — analytics metric resolution + +Read by `../find-evidence/references/product-analytics.md` (retrospective — +find a metric that already has an answer) and +`../experiment-design/references/analytics-instrumentation.md` (prospective +— pre-register a metric before the test runs). Both need the same thing +first: turn an assumption's **Metric for truth** into one concrete, +queryable metric/event on a connected product-analytics platform — without +this repo ever hardcoding what that platform calls things. + +**Delegate, don't reinvent.** validation-os does not ship its own schema +introspection or query execution for PostHog, Mixpanel, BigQuery, or any +other analytics platform — those platforms already publish their own agent +skills that do this well (PostHog: `PostHog/ai-plugin`; Mixpanel: +`mixpanel/ai-plugins`; others: search [skills.sh](https://skills.sh)). This +file is the glue between *their* introspection/query tools and *this +repo's* evidence-ladder discipline — it never contains a platform-specific +API call, event name, or query syntax. If a step below can't be done +because no platform skill/MCP tools are connected, stop and say so +(`connectors/SPEC.md`'s fail-loudly rule) rather than improvising a raw API +call. + +--- + +## Inputs + +```yaml +metric_for_truth: # the claim to resolve into a queryable metric +lens: +platform: +glossary_file: +``` + +## Procedure + +### 1. Business-context call first + +If the connected platform skill exposes a business-context / vocabulary +tool (e.g. Mixpanel's `Get-Business-Context`), call it before anything +else — read-only, cheap, and it resolves project nicknames and internal +acronyms you can't infer from a metric name alone. Not every platform has +one; skip if none exists. + +### 2. Check the glossary for a prior confirmed mapping + +Read `glossary_file` (§ format below). A row matching this +`metric_for_truth` concept → candidate mapping found. **Re-validate before +trusting it**: one cheap list/describe call to confirm the named +event/property/metric still exists. Telemetry schemas drift — a renamed or +removed event silently breaks a stale mapping. Confirmed live → use it, +skip to §5. Gone or changed → fall through to §3 as if no mapping existed, +and flag the stale row for cleanup at the write-back step (§6). + +### 3. No mapping — resolve via the platform skill's own introspection + +Ask the connected platform skill's list/search/describe tools (events, +properties, metrics — whatever it exposes) for candidates whose name or +description plausibly matches `metric_for_truth`. **Never assume a name** +("signup", "activation") exists — every workspace names things +differently. Gather enough detail per candidate to disambiguate: name, +description if available, a sample property value or two. + +### 4. Ambiguous or empty — gated clarifying question + +Zero candidates, or more than one plausible: stop and ask — show what was +found (name + description + sample), never silently guess or pick the +first result. On the human's answer, proceed to §5 with the confirmed +mapping. If genuinely nothing plausible exists on the platform, say so +plainly; that's a valid outcome, not a failure to keep searching. + +### 5. Return the resolved metric + +Hand back to the caller: the platform, the resolved +event/property/metric definition, and whether it came from the glossary +(§2) or a fresh resolution (§3–4). The caller (`product-analytics.md` or +`analytics-instrumentation.md`) takes it from here — running the query or +pre-registering it as the instrument, and applying +`quant-analytics-rubric.md`. + +### 6. Write back to the glossary — gated + +Only when §3–4 ran (a fresh resolution, confirmed by a human) or an old +row was flagged stale in §2. Append or update one row +(`gated-writes.md` — this is a real file write, just not to the registry). +Never write speculative or unconfirmed mappings. + +--- + +## The glossary file + +Workspace-adjacent, not registry content — resolved like +`self_review.review_dir` (`skills/self-review/references/storage.md`), +never routed through the registry connector. Default `analytics-glossary.md` +next to `validation-os.config.yaml`; not private (no gitignore offer) — +it's shared team knowledge, not a personal record. Created on first +write-back, not eagerly scaffolded at setup. + +```markdown +# Analytics glossary — resolved event/metric mappings +Confirmed once by a human, reused after. Re-validated (existence check, +not blind trust) before reuse — telemetry schemas drift. + +| Concept (as it appears in a Metric for truth) | Platform | Resolved definition | Confirmed | By | Notes | +|---|---|---|---|---|---| +| owner completes signup | mixpanel | event `Signup Completed`, `platform=web` | 2026-07-09 | benji | ASM-004 | +``` + +One row per concept. A row whose definition no longer resolves (§2) is +updated in place at the next successful re-resolution — never silently +left stale. + +## Never + +- Never hardcode a platform's event/property/metric name in this file, in + `product-analytics.md`, or in `analytics-instrumentation.md` — every + example above is illustrative, not a default to assume. +- Never trust a glossary row without a live re-check — schemas drift. +- Never guess a mapping when candidates are ambiguous — gate it (§4). +- Never call a platform's raw API directly when no platform skill/MCP + tools are connected — say so and point at installing one, don't + improvise. diff --git a/skills/_shared/experiment-guardrails.md b/skills/_shared/experiment-guardrails.md index 148449f..991fa3b 100644 --- a/skills/_shared/experiment-guardrails.md +++ b/skills/_shared/experiment-guardrails.md @@ -196,7 +196,10 @@ The checklists below are the **minimum bar**; the full prep skeletons live in `/experiment-design`'s `references/` — `interview-guide.md` (guide skeleton + how-to-ask rules, also the question discipline `survey.md` inherits), `prototype-brief.md` (prototype-needed rule table §3b + brief -template), `fake-door.md` (stimulus / costly ask / instrumentation spec). +template), `fake-door.md` (stimulus / costly ask / instrumentation spec), +`analytics-instrumentation.md` (live-telemetry / native A-B pre- +registration — a mechanism fork of the Revealed-tier rungs, not a new rung; +rigor rubric: `quant-analytics-rubric.md`). ### 🔵 Desk research (🟡 `Desk research` rung) diff --git a/skills/_shared/historic-evidence.md b/skills/_shared/historic-evidence.md index f4e9f6d..b158703 100644 --- a/skills/_shared/historic-evidence.md +++ b/skills/_shared/historic-evidence.md @@ -50,7 +50,11 @@ sources: # WHICH flavour(s) to sweep — see below **`sources` is the flavour switch.** Internal values (call transcripts, chat, email, notes, CRM — whatever the config's `evidence_sources` declares; per-source search guidance lives in `/find-evidence`'s `references/`) sweep -your own record; `web` sweeps published secondary sources (desk research). +your own record; `web` sweeps published secondary sources (desk research); +`analytics` — when configured — searches by discover-then-query against a +connected product-analytics platform +(`../find-evidence/references/product-analytics.md`) rather than keyword +search across text. **Default when a caller omits `sources`: the configured internal sources**, so desk research is strictly opt-in. **When `web` is in `sources`, the search + verification must follow `desk-research-rubric.md`** — source @@ -117,7 +121,9 @@ its work. `Desk research`; unpaid real use → 🟢 `Prototype usage`; payment / signed commitment observed → 🟢 `Signed intent` / `Paying users`. A *measured* product metric that bears on the Metric for truth is real behaviour — - revealed-tier, with the analytics view as the source. + revealed-tier, discovered and queried per + `../find-evidence/references/product-analytics.md`, verified against + `quant-analytics-rubric.md` before writing. - **`Source quality`** = High / Medium / Low for the source's seniority / decision authority / ICP-fit. Modulates Strength **within** the rung, never across it. diff --git a/skills/_shared/quant-analytics-rubric.md b/skills/_shared/quant-analytics-rubric.md new file mode 100644 index 0000000..5e5708d --- /dev/null +++ b/skills/_shared/quant-analytics-rubric.md @@ -0,0 +1,132 @@ +# Shared helper — quantitative-analytics trust rubric + +Read by `../find-evidence/references/product-analytics.md` (retrospective) +and `../experiment-design/references/analytics-instrumentation.md` +(prospective). +The quantitative-data analogue of `desk-research-rubric.md`: same reason it +exists — a queried number logged against an assumption becomes `Confidence` +the whole team reads off. A sloppy query doesn't just get a wrong answer, it +gets a **confident** wrong answer that retires a live risk on evidence that +never actually settled it. Every load-bearing number in a +`Prototype usage` / `Signed intent` / `Paying users` Experiment record +sourced from telemetry must clear this rubric. + +Where `desk-research-rubric.md` guards against a bad *source*, this guards +against a bad *query* — the failure modes are different because the "source" +here is your own data, queried on demand, which invites a different kind of +self-deception: peeking at the result before fixing the question. + +--- + +## 1. Pre-register the query, don't discover it after peeking + +Prospective use (`analytics-instrumentation.md`) does this naturally — the +instrument is fixed before the test runs, same discipline as any other +pre-registered bar (`experiment-guardrails.md §4`). Retrospective use +(`product-analytics.md`) has to simulate it: derive the metric, segment, +and window from the assumption's `Metric for truth` **wording alone**, run +it once, and treat any adjustment made *after* seeing the number as the +exact failure mode this rubric exists to catch. + +## 2. Definition stability (schema drift) + +Check whether the event/property/metric definition changed mid-series (a +renamed event, a redefined property, an edited metric formula). A series +that crosses a definition change is not one series — split it at the +change point or cap the read at **Inconclusive**. Note the drift in the +metric glossary (`analytics-metric-resolution.md`) so the next sweep +doesn't silently reuse a mapping that no longer means what it used to. + +## 3. Denominator discipline + +Name the exact denominator — the qualified population matching the +assumption's `Lens` — **before** computing any rate. A metric whose +denominator can narrow or widen to hit a target is invalid, same failure +class as desk research's "no source, no claim": no fixed denominator, no +rate. + +## 4. Segment size + Simpson's paradox + +State the minimum qualified N per segment below which the result is +**Inconclusive**, not a shrunken positive — the same floor +`experiment-guardrails.md §2` already sets for sample size generally. When +slicing by segment, check the aggregate trend against each segment's own +trend: a reversal (the aggregate moves one way, every segment moves the +other) must be surfaced, never hidden by reporting only the cut that +favours the bet. + +## 5. Date-range discipline + +No cherry-picked windows. State *why* this window is the natural, complete +one ("the full month following launch," not "the five best days"). For +prospective use the window is literally fixed at pre-registration and +cannot move once the test starts; for retrospective use, picking a window +after seeing which one clears the bar is the same violation as §1. + +## 6. Correlation vs. caused-by-the-tested-change + +Before attributing a metric move to the thing being tested, check the same +window for confounds — other launches, seasonality, channel-mix shifts, +pricing changes. A platform-native A/B with a real control settles this on +its own; a plain pre/post comparison without a control never fully does — +cap the read at **Inconclusive** if a named confound can't be ruled out. + +## 7. Multiple comparisons / p-hacking + +Querying many segments or cuts and reporting only the one that clears the +bar invalidates the read, exactly as `desk-research-rubric.md §6` requires +capturing conflicting sources rather than cherry-picking. If multiple cuts +were examined, report all of them — the ones that missed the bar are part +of the honest record, not a discarded draft. + +## 8. The verdict + +Map the query result onto `We're right if` / `We're wrong if`: + +- **Validated** — the query clears `We're right if`, on a stable + definition (§2), a fixed denominator (§3), sufficient N (§4), a + pre-registered or clearly-justified window (§1, §5), with no unruled-out + confound (§6). +- **Invalidated** — the query clears `We're wrong if` on the same + standards. Log it — a killed assumption from clean telemetry is a cheap, + valuable win. +- **Inconclusive** — any of §2–§7 can't be cleared: schema drifted + mid-series, N too small in the relevant segment, an unruled-out confound, + or — the rule that matters most — **the window or segment was chosen + after seeing the data.** However clean the resulting number looks, a + post-hoc-chosen cut caps the verdict at Inconclusive. + +--- + +## Body template (write into the Experiment record) + +Two variants, same fields — the pre-registration block +(`analytics-instrumentation.md` writes this before the test runs) and the +retrospective finding (`product-analytics.md` writes this when the +evidence already exists). Dates and verdict differ; the instrument fields +don't. + +```markdown +## Analytics — +**Metric for truth:** +**We're right if:** **We're wrong if:** +**Platform:** **Metric/event:** +**Segment:** **Denominator:** +**Window:** **Min qualified N:** +**Queried / measured:** **Verdict:** + +### Result + + +### Rigor check (quant-analytics-rubric.md) +- Definition stability: +- Confounds in window: +- Cuts examined: + +### Caveats + +``` + +Keep it auditable: a teammate should be able to re-run the exact query and +get the same number, or see exactly where they'd disagree with the +verdict. diff --git a/skills/experiment-design/SKILL.md b/skills/experiment-design/SKILL.md index 39509f5..82faae1 100644 --- a/skills/experiment-design/SKILL.md +++ b/skills/experiment-design/SKILL.md @@ -147,6 +147,15 @@ before opening the next; the write is gated at the end. prototype already exists in the config's `prototype_home`, and if absent produce a prototype brief. Then also build the interview guide for the usage sessions. + - `Prototype usage` / `Signed intent` / `Paying users` **measured live** + (telemetry or a platform-native A/B or feature flag) rather than by a + hand-run session → ask once, gated: "measured live in the connected + analytics platform, or hand-run?" If live → + **`references/analytics-instrumentation.md`** (may combine with the + playbook above when a build is also needed) — pre-registers the exact + query/metric definition, segment, and window as the pass/kill bar, and + optionally creates a native experiment/feature-flag object via the + platform's own tools. - `Opinion` / `Anecdotal` → **`references/interview-guide.md`** — the standard guide skeleton into the Experiment body. **Stimulus override** (one gated question): "will you show a prototype in the interview?" — diff --git a/skills/experiment-design/references/analytics-instrumentation.md b/skills/experiment-design/references/analytics-instrumentation.md new file mode 100644 index 0000000..bdaff80 --- /dev/null +++ b/skills/experiment-design/references/analytics-instrumentation.md @@ -0,0 +1,105 @@ +# Prep playbook — analytics instrumentation (live telemetry / native A-B) + +Entered from SKILL.md step 7 when the chosen `Type` is `Prototype usage`, +`Signed intent`, or `Paying users` **and** the mechanism is a live, +instrumented measurement (telemetry, a platform-native A/B or feature +flag) rather than a hand-run/observed session. This is a **mechanism +fork**, orthogonal to the `Type` rung chosen in step 2 — the rung stays a +pure strength × feasibility call; this playbook only changes *how* it gets +measured. Same relationship `interview-guide.md`'s stimulus override has to +its own step. + +**Delegate, don't reinvent.** Schema discovery, querying, and native +experiment/feature-flag creation are the connected platform's own skill's +job (PostHog: `PostHog/ai-plugin`; Mixpanel: `mixpanel/ai-plugins`; others: +search [skills.sh](https://skills.sh)). This playbook only pre-registers +the instrument and, if the platform supports it, invokes that skill's own +creation tools — it never hand-rolls schema introspection, query syntax, or +pre-launch checks the platform skill already ships. + +## 0. The mechanism question (one gated question) + +*"Is this measured live in \, or by an observed/hand-run session +(interview, a manually-run prototype test)?"* Live/instrumented → this +playbook. Hand-run → the existing route (`prototype-brief.md` / +`fake-door.md` / guardrails §3 checklist inline) applies unchanged. Both +may combine when a build is also needed (e.g. a prototype whose usage is +also instrumented) — run `prototype-brief.md`'s build steps first, this +playbook's instrumentation steps second. + +## 1. Resolve the metric + +Hand off to `../../_shared/analytics-metric-resolution.md` — business- +context call → glossary check → the platform skill's own introspection +tools → gated disambiguation if ambiguous. Same procedure +`product-analytics.md` uses; the metric glossary is shared across both +directions, so a mapping confirmed here is reused (re-validated, not +re-asked) when `/find-evidence` closes this record out. + +## 2. Pre-register the exact instrument — this IS the pass/kill bar + +Apply `../../_shared/quant-analytics-rubric.md` at design time, not just +at close-out — the same rigor rubric `product-analytics.md` applies +retrospectively applies here prospectively. Fixed **before** the run +starts, written into the Experiment body: + +- Exact metric/query definition (from step 1) — the identical spec that + will be re-run at close-out. +- Segment/cohort definition, matching the assumption's `Lens`. +- Date range / window — fixed at design time, never extended after + peeking (rubric §1, §5). +- Denominator definition (rubric §3). +- Minimum qualified N below which the result is `Inconclusive` + (rubric §4). + +## 3. Optional — create the native experiment/flag object (gated) + +When the platform supports it and the design calls for a controlled +rollout: use the platform skill's **own** experiment/feature-flag creation +tools (e.g. Mixpanel's `Create-Experiment` / `Create-Feature-Flag`; +PostHog's equivalent), and its own setup-guidance / pre-launch-check tools +first if it has them (e.g. Mixpanel's `Get-Experiment-Setup-Guidance` / +`Run-Experiment-Pre-Launch-Checks`) — don't hand-roll a check the platform +already ships. Record only the resulting object's ID/key and a link in the +Experiment body, same pattern `fake-door.md` uses for its brief link — +never copy the platform object's own config into the registry as new +fields. + +## Render into the Experiment body + +```markdown +## Analytics instrumentation — + +### Instrument +- Platform: <…> · Metric/event: +- Segment: · Denominator: <…> +- Window: · Min qualified N: <…> + +### Native experiment object (if created) +- Type: · ID/key: <…> · Link: <…> + +### Signal → bar +- We're right if: · We're wrong if: +- Rigor: `quant-analytics-rubric.md` §1–§7 applied before close-out +``` + +## Handoff + +Name the next step — *"let it run until \, then close out via +`/find-evidence`'s analytics flavour — re-run this exact query, don't +redefine it"* — and stop. Running the test, and recording `Result` / +findings, stay outside `/experiment-design`. + +**Anti-patterns:** extending the window after peeking at an early trend · +swapping the metric definition mid-run · treating a platform's own +"significant" flag as sufficient without checking `quant-analytics-rubric.md` +§4/§6 (N and confounds) yourself · copying a native object's full config +into the registry instead of just its ID/link. + +**Terminology check:** `../../_shared/ubiquitous-language.md`, Internal +audience. + +**Write gates:** the Experiment-body section, the glossary write-back (if +any, per `analytics-metric-resolution.md §6`), and the native +experiment/flag object (if created) are each gated +(`../../_shared/gated-writes.md`). diff --git a/skills/find-evidence/SKILL.md b/skills/find-evidence/SKILL.md index d6623bb..6435473 100644 --- a/skills/find-evidence/SKILL.md +++ b/skills/find-evidence/SKILL.md @@ -98,11 +98,14 @@ thread. Call transcripts are the primary internal source when available. research can set a *base rate* but can't validate it, and there may be no internal signal yet → say so and point to `/experiment-design`. Log a base rate as context only (`Inconclusive`), never as validation. -- **Your own product metrics** — the raw dashboards live in your analytics - tool, not this skill. **But** a *measured* metric value that bears on an - assumption's Metric for truth is real behaviour → revealed-tier evidence: - log it as an Experiment record (`Prototype usage` / `Paying users` rung, - source = the analytics view). +- **Your own product metrics** — a *measured* metric value that bears on an + assumption's Metric for truth is real behaviour → revealed-tier evidence. + When `analytics` ∈ the config's `evidence_sources`, this is a full + flavour like internal/desk: `references/product-analytics.md` resolves + the right event/metric live (never assumed) via the connected platform's + own tools and queries it under `../_shared/quant-analytics-rubric.md`. + Not configured → the raw dashboards live in your analytics tool, not this + skill. You may run **both** flavours for one assumption when both bear on it — each qualifying piece becomes its own record. diff --git a/skills/find-evidence/references/product-analytics.md b/skills/find-evidence/references/product-analytics.md new file mode 100644 index 0000000..58bf5d3 --- /dev/null +++ b/skills/find-evidence/references/product-analytics.md @@ -0,0 +1,61 @@ +# Evidence source — Product analytics (PostHog / Mixpanel / BigQuery / warehouse SQL) + +Active when `analytics` ∈ the config's `evidence_sources`. Requires a +product-analytics MCP server (or warehouse SQL access) connected in the +harness, ideally via that platform's own official agent skill — PostHog: +`PostHog/ai-plugin`; Mixpanel: `mixpanel/ai-plugins`; others (BigQuery, +warehouse SQL, Amplitude): search [skills.sh](https://skills.sh). Any tool +exposing both introspection (list/describe events, properties, metrics, +tables) and query execution works the same way — this file's patterns +transfer (same generalization `attio.md` already makes for CRMs). +validation-os composes with that skill's own introspection and query +tools; it does not reimplement them. + +**What it holds:** what your users actually did, measured — the strongest +tier of evidence available (🟢 `Prototype usage` / `Paying users`, +Revealed). Unlike every other source in this directory, the schema (event +names, property names, metric definitions) is workspace-defined and +unknowable in advance — there is no fixed "signup" event to assume. This +file is a procedure for discovering it live through the platform's own +tools, not a field-mapping table. + +**How to search:** + +1. Resolve the metric via `../../_shared/analytics-metric-resolution.md` — + business-context call → glossary check → the platform skill's own + introspection tools → gated disambiguation if ambiguous. Never assume an + event/property name; let the platform skill's own tools name it. +2. Query the resolved metric using the platform skill's own query tool, + with an explicit date range, segment (matching the assumption's `Lens`), + and denominator. Pull enough raw detail to sanity-check — per-segment N, + not just one aggregate number. +3. Apply `../../_shared/quant-analytics-rubric.md` before writing + anything — the rubric is what makes a queried number trustworthy enough + for the register, the quantitative analogue of + `../../_shared/desk-research-rubric.md`. +4. No platform skill/MCP tools connected for the named platform → don't + hand-roll an API call. Say so, and point at installing one (search + skills.sh for the platform name). + +**Qualifies as evidence when:** a queried, reproducible metric value bears +on the `Metric for truth`. Log as `Prototype usage` or `Paying users` +(`../../_shared/historic-evidence.md §3`), with the **exact query/metric definition + +segment + date range** as the source — a link to the saved query/dashboard +when the platform supports one, otherwise the full definition written out. +Never a dashboard screenshot with no reproducible spec behind it. + +**Caveats:** +- Schema drift — an event renamed or redefined mid-series invalidates + reuse of an old glossary mapping (`quant-analytics-rubric.md §2`); + re-resolve, don't extrapolate from a stale mapping. +- Small-N segments read `Inconclusive`, not a weak positive + (`quant-analytics-rubric.md §4`). +- Correlation ≠ caused-by-the-tested-change — check for confounds in the + same window before attributing a move to the thing being tested + (`quant-analytics-rubric.md §6`). +- Never let the read pick its own window or segment after seeing the + data — pre-registration discipline applies retrospectively too + (`quant-analytics-rubric.md §1, §7`). +- A base rate from a platform's aggregate/benchmark feature (if it has one) + is still a base rate, not validation, for a your-user behavioural claim — + same rule as desk research (`../../_shared/desk-research-rubric.md §7`). diff --git a/skills/setup-validation-os/SKILL.md b/skills/setup-validation-os/SKILL.md index c55dd88..aff6e1c 100644 --- a/skills/setup-validation-os/SKILL.md +++ b/skills/setup-validation-os/SKILL.md @@ -78,8 +78,15 @@ Worked example: `../../examples/01-setup.md`. 5. **Declare evidence sources.** Ask which of these the user's agent can actually reach, and list only those under `evidence_sources:`: `web` (default — desk research), `fireflies` (call transcripts), `slack`, - `gmail`, `attio` (CRM). An empty list is fine — `/find-evidence` and - `/meeting-prep` fall back to web research and pasted notes. + `gmail`, `attio` (CRM), `analytics` (product analytics — PostHog, + Mixpanel, BigQuery, warehouse SQL). An empty list is fine — + `/find-evidence` and `/meeting-prep` fall back to web research and + pasted notes. For `analytics`: if no matching platform skill/MCP tools + are connected yet, say so and point at installing one (PostHog's + `ai-plugin`, Mixpanel's `ai-plugins`, or search skills.sh for the + platform) rather than declaring the source without it — same + fail-loudly rule as a missing MCP server for any other source; this + skill doesn't install it for you. 6. **Set the vocabulary.** The schema leaves two lists to the user (`skills/_shared/registry-schema.md`): diff --git a/templates/validation-os.config.yaml b/templates/validation-os.config.yaml index f8e94ed..c23a282 100644 --- a/templates/validation-os.config.yaml +++ b/templates/validation-os.config.yaml @@ -65,6 +65,20 @@ vocabulary: # Evidence sources available to /find-evidence and /meeting-prep. # Skills only sweep what is listed here; with nothing listed they still work # via web/desk research and by asking you to paste notes or transcripts. -# Known sources: web, fireflies, slack, gmail, attio +# Known sources: web, fireflies, slack, gmail, attio, analytics evidence_sources: - web + +# Optional: product-analytics evidence + instrumentation (PostHog, Mixpanel, +# BigQuery, warehouse SQL, or any platform with its own agent skill/MCP +# tools). Active when `analytics` is in evidence_sources above. The platform +# skill/MCP tools are a harness-side prerequisite you install separately +# (e.g. via skills.sh) — this config doesn't wire them up, and no field +# mapping is stored here: schemas are workspace-defined and discovered live +# by the platform skill (skills/_shared/analytics-metric-resolution.md). +# `platform` is a free-text label used only in prose and the glossary, never +# to select code paths. `glossary_file` caches human-confirmed event/metric +# mappings so the same ambiguity isn't re-resolved every sweep. +# analytics: +# platform: "" # e.g. "mixpanel", "posthog", "bigquery" +# glossary_file: analytics-glossary.md # relative to this config file