From c240d008f492aaf38eccbe6b7de8854b73df3c65 Mon Sep 17 00:00:00 2001 From: Joshua Ordehi <45109738+ordehi@users.noreply.github.com> Date: Fri, 19 Jun 2026 14:24:22 -0400 Subject: [PATCH 1/3] feat(audit): consolidate FDE audit knowledge into context-mill skills Adds the marketing-attribution audit dimension (a documented gap), folds eight production-validated identity / flag failure patterns into the existing audit skills, and formalizes the FDE investigation methodology as a cross-cutting reference. New skill: audit-attribution - wizard audit attribution (role: command, parent: audit) - 5 fix-side checks: utm-survives-landing, survives-auth-redirect, first-touch-set-once, custom-click-ids, on-conversion-events - 3 config-side checks: cross-subdomain-cookie, cookieless-mode-impact, consent-integration - Pins shared_docs to utm-segmentation, libraries/js/config, privacy/data-collection, getting-started/identify-users - Bundles engagement-provenance.md mapping each check to its source customer engagement and observed scale audit-identify - New step 5 (server-SDK) with server-process-person-profile, server-sdk-flush-on-exit, server-set-without-identify - New identify-sequential-calls check (Task E in step 2) - Tightened identify-alias-usage polarity: error on backend-aliases- browser-uuid even when alias is on the server side - Old 5-report.md renamed to 6-report.md; report area expanded to include Server SDK; shared_docs picks up anonymous-vs-identified-events.md audit - New init-not-duplicated check in step 2 (dual-init set_once race) audit-feature-flags - New ff-bootstrap-distinct-id-mismatch check (bootstrap overriding identity) - New ff-identified-only-pre-auth-targeting check (silent flag default on anonymous users when person_profiles is identified_only) Investigation methodology - New posthog-best-practices/references/investigation-standards.md - Every audit report step now renders an Assumptions and blind spots subsection per area (placeholder count in the audit notebook-upload skeleton bumped 3 to 5 per area) Companion work (not in this PR): - Wizard repo: seed updates so new check ids appear in the audit ledger TUI - posthog.com docs PRs: callouts for net-new patterns Tests: 92 passing. Build emits audit-attribution.zip and registers wizard audit attribution under cliEntries. --- context/skills/audit-attribution/config.yaml | 20 ++ .../skills/audit-attribution/description.md | 70 +++++ .../references/1-presence.md | 37 +++ .../references/2-attribution-fix.md | 243 ++++++++++++++++++ .../references/3-attribution-config.md | 152 +++++++++++ .../audit-attribution/references/4-report.md | 126 +++++++++ .../references/engagement-provenance.md | 70 +++++ .../audit-autocapture/references/4-report.md | 10 + .../audit-events/references/4-report.md | 10 + .../references/2-feature-flags-fix.md | 97 ++++++- .../references/4-report.md | 10 + context/skills/audit-identify/config.yaml | 1 + .../references/2-identify-fix.md | 49 +++- .../references/3-identify-lifecycle.md | 20 +- .../references/4-identify-optimize.md | 4 +- .../audit-identify/references/5-server-sdk.md | 159 ++++++++++++ .../references/{5-report.md => 6-report.md} | 32 ++- .../references/4-report.md | 10 + context/skills/audit/references/2-init.md | 18 +- context/skills/audit/references/5-report.md | 16 +- .../posthog-best-practices/description.md | 1 + .../references/investigation-standards.md | 65 +++++ 22 files changed, 1188 insertions(+), 32 deletions(-) create mode 100644 context/skills/audit-attribution/config.yaml create mode 100644 context/skills/audit-attribution/description.md create mode 100644 context/skills/audit-attribution/references/1-presence.md create mode 100644 context/skills/audit-attribution/references/2-attribution-fix.md create mode 100644 context/skills/audit-attribution/references/3-attribution-config.md create mode 100644 context/skills/audit-attribution/references/4-report.md create mode 100644 context/skills/audit-attribution/references/engagement-provenance.md create mode 100644 context/skills/audit-identify/references/5-server-sdk.md rename context/skills/audit-identify/references/{5-report.md => 6-report.md} (66%) create mode 100644 context/skills/posthog-best-practices/references/investigation-standards.md diff --git a/context/skills/audit-attribution/config.yaml b/context/skills/audit-attribution/config.yaml new file mode 100644 index 00000000..6fcb05d1 --- /dev/null +++ b/context/skills/audit-attribution/config.yaml @@ -0,0 +1,20 @@ +type: skill +template: description.md +description: Audit a PostHog integration's marketing attribution capture for UTM survival, click-id coverage, cross-subdomain identity, and consent/cookieless interactions +tags: [best-practices] +cli: + role: command + parentCommand: audit + command: attribution +references: + preamble: "**Read ONLY this file.** Do not read any other reference file until this one tells you to." +shared_docs: + - https://posthog.com/docs/data/utm-segmentation.md + - https://posthog.com/docs/libraries/js/config.md + - https://posthog.com/docs/privacy/data-collection.md + - https://posthog.com/docs/getting-started/identify-users.md +variants: + - id: all + display_name: PostHog audit — attribution + tags: [best-practices] + docs_urls: [] diff --git a/context/skills/audit-attribution/description.md b/context/skills/audit-attribution/description.md new file mode 100644 index 00000000..09b3588b --- /dev/null +++ b/context/skills/audit-attribution/description.md @@ -0,0 +1,70 @@ +# PostHog Audit — Attribution + +This skill audits a PostHog integration's **marketing attribution capture** for data integrity. **Read-only** — the only file you create is the final audit report. + +The check space here covers patterns the other audit skills don't: UTM survival across client-side routing and OAuth redirects, custom click-id capture (`gclid`, `fbclid`, `msclkid`), cross-subdomain identity, cookieless-mode tradeoffs against attribution measurement, and consent-banner load ordering. Failures here look like "ad-spend money disappears between ad click and signup event" — silent, expensive, and hard to debug after the fact. + +## Workflow + +The audit runs as a 4-step chain: Presence → Attribution capture (fix) → Attribution configuration → Report. Each step file ends with a pointer to the next. Follow them in order. Resolve each in order before any source-tree exploration. + +**Start by reading the path relative to this file at `references/1-presence.md`.** Do not Glob, ls, or find the skill directory. Do not preload future steps. Do not re-read a step file once you've moved past it. Do not re-read SKILL.md. + +`ToolSearch` is only for loading a tool by exact name when the SDK has it deferred (e.g. `select:Grep`). Do **not** use it to browse for other tools — every tool the audit needs (`Glob`, `Grep`, `Read`, `Write`, `Bash`, and the named `mcp__wizard-tools__audit_*` tools) is already named in this skill. + +**Do not call `TaskCreate` / `TaskUpdate` / `TaskGet` / `TaskList`.** The audit doesn't track its own task list — progress comes from the audit ledger plus `[STATUS]` lines. + +## Live activity — `[STATUS]` + +The "Working on …" banner reads from `[STATUS]` lines you emit in plain text. Whenever you start a new sub-step, write a line like: + +``` +[STATUS] Detecting attribution surfaces +``` + +The wizard intercepts these and updates the spinner. Use them freely — they are cheap. Each step file lists the exact `[STATUS]` strings to emit at each sub-step. + +## Audit checks ledger + +The ledger lives at `.posthog-audit-checks.json` and is rendered live in the "Audit plan" tab. It is owned by MCP tools — **never `Write` this file directly**: + +- `mcp__wizard-tools__audit_resolve_checks({ updates })` — patch one or more checks by `id`. Each `update` is `{ id, status, file?, details? }`. Batch updates from the same step into a single call. + +All audit ledger calls are atomic and serialize internally — **concurrent calls from parallel subagents cannot lose updates**, so feel free to fan out runtime checks across `Agent` subagents when a step says so. + +### Check entry shape + +- `id` — stable kebab-case slug. Reuse the existing seeded ids exactly when calling `audit_resolve_checks`. +- `area` — short group name. This skill uses `Attribution` (fix) and `Attribution — Configuration` (config). +- `label` — short human name. +- `status` — `pending` | `pass` | `error` | `warning` | `suggestion`. +- `file` — optional `path:line` for findings tied to a location. +- `details` — optional one-line explanation (or compact JSON for structured findings). + +After the report is written (Step 4), delete `.posthog-audit-checks.json`. + +## Severity levels + +- `error`: Must fix. Broken attribution or guaranteed data corruption. +- `warning`: Should fix. Pattern that causes silent attribution loss or measurement degradation. +- `suggestion`: Nice to have. Best-practice improvement or coverage gap on a non-critical attribution surface. + +## Investigation standards + +Every finding produced by this skill must meet the standards in [posthog-best-practices/references/investigation-standards.md]: provenance on every claim, verification evidence inline, and adversarial self-review per area in the report. The skill's grep patterns and rule prose enforce provenance and evidence; the report step renders the per-area "Assumptions and blind spots" subsection. + +## Key principles + +- **Read-only**: Do not edit project source files. The only file you create is the audit report. +- **Evidence-based**: Reference specific `file:line` for every non-pass finding. +- **Actionable**: Every finding states what to fix and how. +- **Conservative on tenant-shape inference**: when there's no static signal that a project runs paid ad campaigns (no ad-platform pixels, no campaign-tracking parameters anywhere in the code, no marketing-site routes), resolve campaign-only checks with `pass` and `details: "skip: no paid-acquisition signal detected"` rather than warning on absence. + +## Abort statuses + +Report abort states with `[ABORT]` prefixed messages. The wizard catches these and terminates the run — do not halt yourself. +- No PostHog SDK initialization found + +## Framework guidelines + +{commandments} diff --git a/context/skills/audit-attribution/references/1-presence.md b/context/skills/audit-attribution/references/1-presence.md new file mode 100644 index 00000000..a7f601ee --- /dev/null +++ b/context/skills/audit-attribution/references/1-presence.md @@ -0,0 +1,37 @@ +--- +next_step: 2-attribution-fix.md +--- + +# Step 1 — Presence detector + +This step decides whether the rest of the audit has anything to look at, and records signals later steps need. Run it **before** any other work. Resolve zero ledger checks here — this step is gating only. + +## Status + +Emit: + +``` +[STATUS] Detecting PostHog and attribution surfaces +``` + +## Action + +Run **two `Grep` calls in parallel**, both with `output_mode: "files_with_matches"`: + +1. PostHog init surface — any of: + `posthog\.init\(|new PostHog\(|posthog\.Posthog\(|Posthog\(` +2. Attribution / acquisition signals — any of: + `utm_source|utm_medium|utm_campaign|gclid|fbclid|msclkid|msfclkid|li_fat_id|ttclid|twclid|partner_id|referrer_id` + +## Decision + +- **Init grep returns zero hits anywhere in the project:** emit `[ABORT] No PostHog SDK initialization found` and stop. The wizard catches `[ABORT]` and terminates the run. +- **Init found:** continue, regardless of whether attribution signals were detected. Even projects with no explicit click-id capture rely on PostHog's built-in UTM auto-capture; each step's individual rules decide whether to skip or warn based on the kind of evidence present. + +## Record acquisition signal for later steps + +Keep the acquisition-signal grep result in working memory. Step 2's `attribution-custom-click-ids` check uses it to decide whether to warn on absence (project clearly runs paid acquisition but doesn't capture click ids) vs. skip (no paid-acquisition signal anywhere — silence is fine). + +Do not read any files in this step. Do not call `audit_resolve_checks`. Do not preload future steps. + +Continue to **`2-attribution-fix.md`**. diff --git a/context/skills/audit-attribution/references/2-attribution-fix.md b/context/skills/audit-attribution/references/2-attribution-fix.md new file mode 100644 index 00000000..9ad1af0d --- /dev/null +++ b/context/skills/audit-attribution/references/2-attribution-fix.md @@ -0,0 +1,243 @@ +--- +next_step: 3-attribution-config.md +--- + +# Step 2 — Attribution capture (fix) + +This step resolves five attribution-capture checks **in parallel**, one subagent per check: + +- `attribution-utm-survives-landing` +- `attribution-survives-auth-redirect` +- `attribution-first-touch-set-once` +- `attribution-custom-click-ids` +- `attribution-on-conversion-events` + +## Status + +Emit before dispatching: + +``` +[STATUS] Auditing attribution capture +``` + +## Action — dispatch five subagents in one message + +Make **five `Agent` tool calls in a single message** so they run concurrently. Wait for all five to return, then continue to `3-attribution-config.md`. Do not run any other tools between dispatch and the next step. + +The bundled `utm-segmentation.md` reference holds PostHog's authoritative guidance on UTM auto-capture and campaign properties. It's typically at `.claude/skills/audit-attribution/references/utm-segmentation.md`; if that path doesn't exist, discover it with `Glob` `**/skills/audit-attribution/references/utm-segmentation.md`. The bundled `identify-users.md` reference covers first-touch person properties. Each subagent reads only the references relevant to its check. + +### Task A — `attribution-utm-survives-landing` + +`description`: `Audit attribution-utm-survives-landing` + +`prompt`: +``` +You are an audit subagent. Resolve exactly one rule and return: attribution-utm-survives-landing. + +Read this skill's bundled `utm-segmentation.md` reference once (typically `.claude/skills/audit-attribution/references/utm-segmentation.md`). + +Background: PostHog auto-captures the standard `utm_source`, `utm_medium`, `utm_campaign`, `utm_content`, `utm_term` URL parameters from the page URL at PostHog init time. Several common patterns silently drop those params before PostHog gets to read them: +- Client-side router redirects (Next.js middleware redirects, locale rewrites, `www` → root redirects) that drop the query string. +- A landing page that strips its query string in a useEffect / mounted hook (often "for cleanliness") before `posthog.init` runs. +- Service workers or edge functions rewriting the request URL. +- PostHog init that runs after a route change — by then the `?utm_*` is gone from `window.location.search`. + +Run **three** Greps in parallel: +- `posthog\.init\(|new PostHog\(|posthog\.Posthog\(|Posthog\(` — init sites. +- `searchParams|URLSearchParams|window\.location\.search|router\.replace|router\.push|history\.replaceState|history\.pushState|setQuery|\.replace\(\{[^}]*query` — URL/query-string manipulation sites. +- `middleware\.(ts|js)|next\.config\.(js|ts|mjs)|redirects\s*:|rewrites\s*:|locale.*redirect|i18n` — server-side routing config that might strip query params. + +Read each file that contains a query-manipulation hit, once. For each manipulation site, determine whether it removes `utm_*` parameters before the PostHog init has had a chance to fire (look for evidence the manipulation runs in the same component / module that mounts before the PostHog provider, or in middleware that runs server-side before the page renders). + +Rule: +- pass: no obvious UTM-stripping patterns detected, OR PostHog init clearly runs before any detected query manipulation. +- suggestion: the project has middleware / redirects that COULD strip query params but no direct evidence they touch `utm_*`. Recommend the operator add UTM passthrough to the redirect chain. +- warning: a code path explicitly strips query params before PostHog init runs (e.g. a top-level `useEffect` calling `router.replace(pathname)` in the same provider tree as ``). + +Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for id `attribution-utm-survives-landing`, including `file` (path:line of the most relevant manipulation site) and `details` as compact JSON: + +``` +{ + "init_runs_before_query_manipulation": , + "manipulation_sites": ["", ...], + "recommendation": "keep | preserve-utms-in-redirects | reorder-init-before-strip" +} +``` + +Return when the call completes. Do not write the audit report. +``` + +### Task B — `attribution-survives-auth-redirect` + +`description`: `Audit attribution-survives-auth-redirect` + +`prompt`: +``` +You are an audit subagent. Resolve exactly one rule and return: attribution-survives-auth-redirect. + +Read this skill's bundled `utm-segmentation.md` reference once (typically `.claude/skills/audit-attribution/references/utm-segmentation.md`). + +Background: OAuth / SSO authentication redirects the user away from the site (to Auth0 / Clerk / WorkOS / Supabase / Firebase / Google / GitHub / etc.) and back via a callback URL. UTM parameters present on the click that started the signup flow do NOT survive this redirect unless the application deliberately preserves them. Three preservation patterns are valid: +1. **sessionStorage / localStorage stash**: read `utm_*` before redirecting, store, restore on callback before firing `posthog.identify()` or the signup event. +2. **OAuth `state` param**: encode UTM values into the `state` string the auth provider echoes back on callback. +3. **Cookie stash**: set a short-lived first-party cookie with UTM values before the redirect. + +Without preservation, PostHog's first-touch person properties (`$initial_utm_*`) may already have been set from the landing pageview, but the signup conversion event itself fires without UTMs — making it impossible to correlate ad spend with funnel completion. + +Run **three** Greps in parallel: +- `(?i)(authorize|signin|signup|signIn|signUp|loginWith|signInWith|redirectToSignIn|workos|auth0|clerk|supabase\.auth|firebase\.auth|next-auth|auth-js|@oauth|oauth)` — auth redirect surfaces. +- `(?i)(utm_source|utm_campaign).{0,80}(sessionStorage|localStorage|setCookie|document\.cookie|state\s*:|state\s*=)` — UTM stash patterns. +- `(?i)(callback|/api/auth/callback|/auth/callback|onAuthStateChange)` — callback handlers. + +Read files matching any of these once. Determine: +- Does the project have an OAuth / SSO redirect flow at all? +- If yes, is there logic that reads `utm_*` from the URL before the redirect AND restores them after the callback? + +Rule: +- pass with details "skip: no OAuth/SSO redirect flow detected" — project has no detected auth redirect. +- pass: project has an auth-redirect flow AND a detectable UTM preservation pattern (stash + restore) is present. +- warning: project has an auth-redirect flow AND NO detected UTM preservation. The signup conversion event will not carry the original UTMs, breaking ad-attribution correlation. Recommend adding a sessionStorage stash before the redirect and a restore in the callback handler. + +Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for id `attribution-survives-auth-redirect`, including `file` (path:line of the most relevant auth-redirect site or callback handler) and `details` as compact JSON: + +``` +{ + "auth_redirect_detected": , + "preservation_pattern_detected": "session-storage | local-storage | oauth-state | cookie | none", + "callback_handler_site": "", + "recommendation": "keep | add-stash-and-restore" +} +``` + +Return when the call completes. Do not write the audit report. +``` + +### Task C — `attribution-first-touch-set-once` + +`description`: `Audit attribution-first-touch-set-once` + +`prompt`: +``` +You are an audit subagent. Resolve exactly one rule and return: attribution-first-touch-set-once. + +Read this skill's bundled `identify-users.md` reference once (typically `.claude/skills/audit-attribution/references/identify-users.md`). + +Background: first-touch attribution properties — `$initial_referrer`, `$initial_referring_domain`, `$initial_utm_*`, `signup_date`, `first_seen_*`, `original_*` — should be written with `$set_once`, not `$set`. PostHog's own `$initial_*` properties are managed by the SDK and use `$set_once` automatically; the risk is project-defined first-touch attrs (e.g. `original_partner_id`, `signup_landing_page`, `first_seen_plan_intent`) being written with `$set`, which overwrites the first-touch value on every subsequent identify and corrupts attribution analytics. + +Run **two** Greps in parallel: +- `posthog\.identify\(|setPersonProperties\(` — identify and person-property-setter call sites. +- `\$set\b|\$set_once\b|setOnce\(|setPersonPropertiesOnce` — every $set / $set_once usage in the project. + +Read each file that contains a `$set` or `$set_once` hit, once. For each usage, classify the property keys being passed: +- Project-defined first-touch property — name starts with `initial_`, `first_`, `original_`, or contains `signup_date`, `landing_`, `first_seen_`, `acquisition_`. +- General mutable property — `plan`, `email`, `last_seen_*`, `account_status`, etc. + +Flag any project-defined first-touch property written with `$set` (not `$set_once`). + +Rule: +- pass: no project-defined first-touch properties detected, OR every project-defined first-touch property is written with `$set_once`. +- warning: 1+ project-defined first-touch properties written with `$set` — every subsequent identify() overwrites the original-touch value. + +Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for id `attribution-first-touch-set-once`, including `file` (path:line of the most representative offending $set) and `details` as compact JSON: + +``` +{ + "first_touch_using_set_count": , + "examples": [ + {"file": "", "property": "", "issue": "first-touch-using-set"} + ] +} +``` + +Return when the call completes. Do not write the audit report. +``` + +### Task D — `attribution-custom-click-ids` + +`description`: `Audit attribution-custom-click-ids` + +`prompt`: +``` +You are an audit subagent. Resolve exactly one rule and return: attribution-custom-click-ids. + +Read this skill's bundled `utm-segmentation.md` reference once (typically `.claude/skills/audit-attribution/references/utm-segmentation.md`). + +Background: PostHog auto-captures the standard UTM parameters but does NOT auto-capture ad-platform click identifiers: `gclid` (Google Ads), `fbclid` (Facebook / Meta), `msclkid` (Microsoft / Bing), `ttclid` (TikTok), `twclid` (Twitter / X), `li_fat_id` (LinkedIn), `pinterest_click_id`, etc. These click ids are what enable downstream conversion APIs (Google Ads "Enhanced Conversions", Meta CAPI, etc.) and revenue-attribution platforms to credit specific ad clicks for specific conversions. Without them, paid acquisition campaigns can't measure ROAS accurately. + +Run **two** Greps in parallel: +- `gclid|fbclid|msclkid|msfclkid|ttclid|twclid|li_fat_id|wbraid|gbraid|pinterest_click_id|sccid` — explicit click-id handling in the codebase. +- `gtag\(|fbq\(|window\.dataLayer|googletagmanager|google.ads|facebook.com/tr|googleads|meta-pixel|, + "captures_without_attribution_count": , + "examples": [ + {"file": "", "event": "", "issue": "missing-utm | missing-click-id | missing-referrer"} + ] +} +``` + +Return when the call completes. Do not write the audit report. +``` + +## After all five return + +Continue to **`3-attribution-config.md`**. diff --git a/context/skills/audit-attribution/references/3-attribution-config.md b/context/skills/audit-attribution/references/3-attribution-config.md new file mode 100644 index 00000000..76ef6ff6 --- /dev/null +++ b/context/skills/audit-attribution/references/3-attribution-config.md @@ -0,0 +1,152 @@ +--- +next_step: 4-report.md +--- + +# Step 3 — Attribution configuration + +This step resolves three configuration-side attribution checks **in parallel**, one subagent per check: + +- `attribution-cross-subdomain-cookie` +- `attribution-cookieless-mode-impact` +- `attribution-consent-integration` + +## Status + +Emit before dispatching: + +``` +[STATUS] Auditing attribution configuration +``` + +## Action — dispatch three subagents in one message + +Make **three `Agent` tool calls in a single message** so they run concurrently. Wait for all three to return, then continue to `4-report.md`. Do not run any other tools between dispatch and the next step. + +The bundled `config.md` (posthog-js config reference) and `data-collection.md` (privacy / cookieless) references hold the canonical settings. Both are typically at `.claude/skills/audit-attribution/references/`. Each subagent reads only what its check needs. + +### Task A — `attribution-cross-subdomain-cookie` + +`description`: `Audit attribution-cross-subdomain-cookie` + +`prompt`: +``` +You are an audit subagent. Resolve exactly one rule and return: attribution-cross-subdomain-cookie. + +Read this skill's bundled `config.md` reference once (typically `.claude/skills/audit-attribution/references/config.md`). + +Background: when a project spans subdomains (e.g. `example.com` marketing site, `app.example.com` product, `dashboard.example.com` admin), PostHog's anonymous distinct_id by default lives in a cookie scoped to the exact subdomain. A user visiting `example.com?utm_source=google`, then clicking through to `app.example.com/signup`, gets two separate anonymous person profiles — the UTM-tagged one on the marketing host, and a fresh one on the app host. The cross-subdomain conversion isn't visible until they identify, and the initial UTMs never propagate to the app-side anonymous profile. The fix is `cross_subdomain_cookie: true` in the init config, which scopes the cookie to the registrable parent domain. + +Run **two** Greps in parallel: +- `posthog\.init\(|new PostHog\(|posthog\.Posthog\(|Posthog\(` — every init site. +- `cross_subdomain_cookie|crossSubdomainCookie` — explicit setting. + +Read each init file once. Determine whether `cross_subdomain_cookie` is set. Also scan the project for subdomain signals to decide whether the project actually spans subdomains: +- README / package.json mentions of multiple subdomains. +- `next.config`, `vite.config`, `astro.config` with custom domains. +- Hard-coded URLs in source pointing at multiple subdomains of the same parent (e.g. `dashboard.example.com` and `example.com` both referenced). + +Rule: +- pass: `cross_subdomain_cookie: true` is set, OR the project shows no signal of spanning subdomains (single-host project). +- suggestion: the project shows signals of spanning subdomains AND `cross_subdomain_cookie` is unset — recommend adding it on the browser init. Also verify the cookie `domain` value (when set explicitly) matches the parent (e.g. `.example.com`). +- warning: `cross_subdomain_cookie: true` is set but with a `cookie_domain` that doesn't match the parent (e.g. `.app.example.com` instead of `.example.com`) — the cookie is still scoped too narrowly. + +Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for id `attribution-cross-subdomain-cookie`, including `file` (path:line of the init that sets or should set the option) and `details` as compact JSON: + +``` +{ + "cross_subdomain_cookie_setting": "true | false | unset", + "cookie_domain_setting": "", + "subdomain_signals_detected": , + "recommendation": "keep | enable-cross-subdomain | fix-cookie-domain" +} +``` + +Return when the call completes. Do not write the audit report. +``` + +### Task B — `attribution-cookieless-mode-impact` + +`description`: `Audit attribution-cookieless-mode-impact` + +`prompt`: +``` +You are an audit subagent. Resolve exactly one rule and return: attribution-cookieless-mode-impact. + +Read this skill's bundled `data-collection.md` reference once (typically `.claude/skills/audit-attribution/references/data-collection.md`). + +Background: `cookieless_mode` in posthog-js gives pseudonymous tracking without persistent identifiers. Two modes: +- `cookieless_mode: 'on_reject'` — falls back to cookieless tracking only for users who reject the consent banner. +- `cookieless_mode: 'always'` — cookieless for every user. + +In cookieless mode, the SDK derives the anonymous identifier from a daily-rotating hash of user-agent + IP + day. This means **cross-session attribution is lost** — a user clicking an ad on Monday and signing up on Thursday cannot be linked. For sites running paid acquisition campaigns, this directly impacts the ability to measure ROAS. This is a legitimate tradeoff (privacy vs. measurement) but should be a conscious one, not silent. + +Run **two** Greps in parallel: +- `cookieless_mode|cookielessMode` — explicit setting. +- `gtag\(|fbq\(|googletagmanager|google.ads|facebook.com/tr|googleads|meta-pixel|, + "recommendation": "keep | review-attribution-tradeoff" +} +``` + +Return when the call completes. Do not write the audit report. +``` + +### Task C — `attribution-consent-integration` + +`description`: `Audit attribution-consent-integration` + +`prompt`: +``` +You are an audit subagent. Resolve exactly one rule and return: attribution-consent-integration. + +Read this skill's bundled `data-collection.md` reference once (typically `.claude/skills/audit-attribution/references/data-collection.md`). + +Background: cookie / tracking consent banners interact with PostHog in two attribution-relevant ways: +1. **PostHog loads BEFORE consent is granted** — landing pageview fires (and possibly captures UTMs) before the user has the chance to opt in. Depending on jurisdiction and consent posture, this may violate GDPR / ePrivacy. +2. **PostHog respects consent state via `opt_in_capturing()` / `opt_out_capturing()`** — when consent is granted, the SDK starts capturing; when revoked, it stops. If the integration is missing or the consent state isn't checked at init time, either no events are captured (silent attribution loss when consent IS granted but the check never fires) or every event is captured regardless of consent (legal risk). + +Run **two** Greps in parallel: +- `opt_in_capturing|opt_out_capturing|optInCapturing|optOutCapturing|hasOptedIn|hasOptedOut` — explicit consent-API usage. +- `(?i)(cookie.?banner|cookie.?consent|consent.?banner|cookiebot|onetrust|osano|usercentrics|@iubenda|trackingConsent|gdprConsent|granted.*consent|consentGiven)` — consent surfaces. + +Read each consent-surface file, once. Determine: +- Does the project have a detected consent surface at all? (If no, resolve `pass with details "skip: no consent surface detected"`.) +- Does the project call `posthog.opt_in_capturing()` / `opt_out_capturing()` when consent state changes? +- Is `posthog.init` called BEFORE or AFTER the consent state is known? (Look for init-after-consent patterns: init inside the consent banner's `onAccept` callback, or init gated on `consentGiven` state.) + +Rule: +- pass with details "skip: no consent surface detected" — no consent banner detected. +- pass: consent surface exists AND `opt_in_capturing` / `opt_out_capturing` is integrated AND PostHog init either runs after consent OR explicitly starts in opt-out state. +- suggestion: consent surface exists but no PostHog opt-in/opt-out integration detected — recommend wiring the consent banner to PostHog's API so attribution capture respects user choice. +- warning: PostHog init runs before the consent surface mounts AND no `opt_out_capturing()` is called at init time — events fire before the user has chosen, with legal and attribution implications. + +Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for id `attribution-consent-integration`, including `file` (path:line of the most relevant consent surface or init) and `details` as compact JSON: + +``` +{ + "consent_surface_detected": , + "opt_in_api_integrated": , + "init_runs_before_consent": , + "recommendation": "keep | wire-opt-in-api | gate-init-on-consent" +} +``` + +Return when the call completes. Do not write the audit report. +``` + +## After all three return + +Continue to **`4-report.md`**. diff --git a/context/skills/audit-attribution/references/4-report.md b/context/skills/audit-attribution/references/4-report.md new file mode 100644 index 00000000..f4599e6b --- /dev/null +++ b/context/skills/audit-attribution/references/4-report.md @@ -0,0 +1,126 @@ +--- +next_step: null +--- + +# Step 4 — Generate the audit report + +The audit report is rendered **directly from `.posthog-audit-checks.json`** — that file is the source of truth. Every check the wizard seeded for this skill ends up in the report, even passes; nothing is invented. + +## Status + +Emit: + +``` +[STATUS] Writing attribution audit report +``` + +## Action + +`Read` the ledger once, then transform every entry into the report below. Use `area`, `label`, `status`, `file`, and `details` from each entry verbatim where the report calls for them. + +`Write` `posthog-audit-attribution-report.md` at the project root with the structure shown below. After the report is written, delete `.posthog-audit-checks.json`. + +The report has four sections in this order: + +1. **Summary** — one-paragraph overview, severity counts, and a problematic-items table. +2. **Recommended actions** — prioritized fixes with `file:line` where applicable. +3. **Full audit** — every check the wizard ran, grouped by `area`, including passes. +4. **About this audit** — short closing block explaining what this audit covered. + +For the Full audit section, group rows by each distinct `area` value in the ledger, preserving first-seen area order from the JSON. This skill produces two areas: **Attribution** (fix) and **Attribution — Configuration** (config). Render whatever areas the ledger actually contains. + +For each area, write a one-paragraph framing immediately under the area heading, then the table. + +## Report template + + +# PostHog Attribution Audit Report + +## Summary + +[1–2 sentence overview: which surfaces are covered (landing, auth, conversion), overall attribution health, and which lens — fix or configuration — surfaced issues.] + +**Counts** + +- **Errors**: [N] (must fix) +- **Warnings**: [N] (should fix) +- **Suggestions**: [N] (nice to have) +- **Passes**: [N] + +**Problematic items** _(only `error`, `warning`, `suggestion` — no passes)_ + +| Severity | Area | Check | File | Details | +|----------|------|-------|------|---------| +| `error` | Attribution | [label] | [file:line] | [details] | + +If there are no problematic items, write `_No issues found — your attribution setup looks healthy._` instead of the table. + +## Recommended actions + +Numbered list, ordered by severity (errors → warnings → suggestions), then by area within a severity (Attribution → Attribution — Configuration). Each item is **three sentences**, in this order: + +1. **What's wrong** — the finding, written as a one-sentence diagnosis derived from `details`. +2. **Why it matters** — one sentence on the attribution-data-quality consequence (lost UTM trace, untrackable cross-session signup, broken cross-subdomain identity, missing click-id for conversion API). +3. **How to fix** — one short imperative sentence pointing at `file:line` and the concrete change. End with a docs link. + +Format: + +1. **[Area] · [label]** — [what's wrong]. _Why it matters:_ [why-it-matters]. _Fix:_ [how-to-fix at `file:line`]. See [docs]([area docs url]). + +Suggested docs URLs: +- `attribution-utm-survives-landing`, `attribution-survives-auth-redirect`, `attribution-on-conversion-events`, `attribution-custom-click-ids` → https://posthog.com/docs/data/utm-segmentation +- `attribution-first-touch-set-once` → https://posthog.com/docs/getting-started/identify-users +- `attribution-cross-subdomain-cookie` → https://posthog.com/docs/libraries/js/config +- `attribution-cookieless-mode-impact`, `attribution-consent-integration` → https://posthog.com/docs/privacy/data-collection + +If there are no actions, write `_Nothing to fix._`. + +## Full audit + +### Attribution + +This area covers attribution capture quality: whether UTM parameters survive client-side routing and OAuth redirects to reach the signup event, whether project-defined first-touch properties use `$set_once` (not `$set`) so the original-touch value persists, whether ad-platform click ids (`gclid`, `fbclid`, `msclkid`, etc.) are captured for downstream conversion APIs, and whether conversion events are enriched with attribution context. + +| Check | Status | File | Details | +|-------|--------|------|---------| +| [label] | [status] | [file] | [details] | + +### Attribution — Configuration + +This area covers configuration-side attribution health: `cross_subdomain_cookie` on multi-subdomain projects (so marketing-host UTMs follow the user to the app host), `cookieless_mode` and its tradeoff against cross-session attribution, and consent-banner integration with PostHog's opt-in / opt-out APIs. + +| Check | Status | File | Details | +|-------|--------|------|---------| +| [label] | [status] | [file] | [details] | + +[Repeat the heading + paragraph + table for each area in ledger order, in case future versions of this skill add new areas.] + +### Assumptions and blind spots + +Under each area's table above, render a `### Assumptions and blind spots` subsection per the investigation standards in `posthog-best-practices/references/investigation-standards.md` (standard 3). Answer the four questions in plain prose, ≤4 sentences total: +- Which code paths or files this area did NOT check that could change the findings. +- Which runtime assumptions are unproven by the static code (mount order, async timing, route gating). +- Alternative explanations for the patterns the checks flagged. +- What you would verify in the live PostHog project (event volumes, property fill rates, dashboard usage) to confirm or refute the most important findings. + +When an area produced only `pass` rows, write `_No findings to qualify; the standard checks for this area passed cleanly._` and skip the four-question rundown. + +## About this audit + +This audit ran the PostHog `audit-attribution` skill — a focused, read-only check of attribution capture health across two lenses: **fix** (capture quality on landing, redirects, conversion events) and **configuration** (cross-subdomain identity, cookieless / consent interactions). All checks scan the project source; none require PostHog MCP access. Attribution checks include conservative skip conditions (`no auth redirect`, `no consent surface`, `no paid-acquisition signal`) so single-host product-only projects don't get noisy warnings. + +- `error` items break attribution now (broken cross-subdomain identity, guaranteed UTM loss). Fix first. +- `warning` items work today but cause silent attribution loss (un-preserved UTMs through auth, missing click ids, cookieless × paid-acquisition mismatch). Fix when convenient. +- `suggestion` items are best-practice improvements with measurable upside on ad-spend attribution. + +Pair with `posthog-wizard audit identify` for the identity-resolution side (which is where many attribution failures actually manifest as blocked merges), and with `posthog-wizard audit events` for the conversion-event taxonomy side. + +Re-run `posthog-wizard audit attribution` after applying fixes to refresh the ledger. + + + +After the report is written, emit a final line so the wizard can surface the path to the user: + +``` +Created audit report: +``` diff --git a/context/skills/audit-attribution/references/engagement-provenance.md b/context/skills/audit-attribution/references/engagement-provenance.md new file mode 100644 index 00000000..f7067336 --- /dev/null +++ b/context/skills/audit-attribution/references/engagement-provenance.md @@ -0,0 +1,70 @@ +# Engagement provenance — why each check exists + +Reference for skill maintainers. Each check in this skill traces back to a production failure observed during a PostHog FDE engagement or support escalation. This file captures that pedigree so the next maintainer who asks "why do we check for `attribution-survives-auth-redirect`?" sees the customer impact, not just the rule prose. + +This file is NOT loaded by the audit at runtime — it's bundled with the skill so it ships alongside the checks but stays out of the subagent context. Treat it as institutional memory for code review and check-rule iteration. + +## Source engagements (anonymized) + +### Multi-SDK SaaS A — identity merges at scale + +- **Dual `posthog.init()` race corrupting `$set_once` attribution** (~165k blocked merges/day). Two browser inits on the signup page raced to stamp `$initial_pathname` and `$initial_referrer`, producing contradictory `$set_once` values stamped at the same millisecond. + - Now lives in **`audit/references/2-init.md`** (`init-not-duplicated`). +- **Sequential identify calls** (auth provider UID → internal user ID, ~30–35k blocked merges/day). The first identify stamped the auth-provider UID as device identity; the second was blocked from merging because the device was already "identified." + - Now lives in **`audit-identify/references/2-identify-fix.md`** (`identify-sequential-calls`). +- **Backend `posthog.alias()` making UUIDs identified**, blocking web SDK merges (~80–100k blocked merges/day). The server aliased a browser anonymous UUID to a user id; the next client identify() with the same UUID as `$anon_distinct_id` was silently blocked. + - Now lives in **`audit-identify/references/3-identify-lifecycle.md`** — `identify-alias-usage` (polarity tightened to error on this case). +- **Missing `posthog.reset()` on workspace / account switch** (~3–9k blocked merges/day). + - Already covered by `audit-identify` → `identify-reset-on-logout`. +- **`bootstrap.distinctID` overriding identity chain.** + - Now lives in **`audit-feature-flags/references/2-feature-flags-fix.md`** (`ff-bootstrap-distinct-id-mismatch`). + +### Consumer app B — backend SDK property corruption + event loss + +- **posthog-python `$os` auto-attach** corrupting person properties (24% of profiles with `$os = "Linux"` on a macOS-heavy product). + - Now lives in **`audit-identify/references/5-server-sdk.md`** (`server-process-person-profile`). +- **Missing alias on identity transitions** — solved by tightened `identify-alias-usage`. +- **Logout race conditions** — covered by `identify-reset-on-logout`. +- **Integer user_id as distinct_id** instead of stable UUID — covered by `identify-stable-distinct-id`. +- **Redundant backend identify calls** — covered by `identify-set-discipline` + `identify-alias-usage`. +- **SDK event loss on Celery worker termination** (background workers exiting before buffer flush). + - Now lives in **`audit-identify/references/5-server-sdk.md`** (`server-sdk-flush-on-exit`). +- **Backend local-eval missing person properties** — not directly covered (flag-eval, not setup); `audit-feature-flags` → `ff-identified-only-pre-auth-targeting` flags the same root-cause shape. + +### B2B SaaS C — data gaps and integrations + +- **Missing `groupIdentify` at org creation** — covered by `audit-identify` → `identify-groupidentify-correctness`. +- **Internal event suppression** — covered by `audit-events` → `events-env-pollution`. +- **`$set`-without-`$identify` anti-pattern** — now lives in **`audit-identify/references/5-server-sdk.md`** (`server-set-without-identify`). +- **Stripe webhook handler not syncing billing state to PostHog** — out of scope for static codebase audit. + +### Multi-surface app D — flags, consent, attribution + +- **`person_profiles: identified_only` silently breaking property-targeted flags for anonymous users** on pre-auth surfaces. + - Now lives in **`audit-feature-flags/references/2-feature-flags-fix.md`** (`ff-identified-only-pre-auth-targeting`). +- **`cookieless_mode` interaction with attribution for ad campaigns.** + - Now lives in **`audit-attribution/references/3-attribution-config.md`** (`attribution-cookieless-mode-impact`). +- **Consent posture affecting cross-session attribution measurement.** + - Now lives in **`audit-attribution/references/3-attribution-config.md`** (`attribution-consent-integration`). + +### Pre-launch SaaS E — UTM loss through OAuth + +- **UTM params lost through OAuth redirects.** A user clicked an ad with `?utm_source=google`, started signup, got redirected to the auth provider, came back via callback URL with no UTMs, and the signup event fired without any campaign context. The fix is a sessionStorage / state-param / cookie stash before the redirect. + - Now lives in **`audit-attribution/references/2-attribution-fix.md`** (`attribution-survives-auth-redirect`). +- Highest priority for this skill because there was no canonical PostHog docs page covering the pattern — a parallel docs PR proposing one is the long-term remediation home. + +### Cross-engagement patterns absorbed elsewhere + +- **Event naming drift between autocapture and custom events** — `audit-events` → `event-naming-standardization`, `event-duplicates-and-bloat`. +- **SDK distribution and version inventory** — `audit` → `sdk-installed`, `sdk-up-to-date`. + +## How to use this when adding new checks + +When a new failure pattern surfaces in an FDE engagement or support escalation: + +1. Decide whether it fits an existing check (patch) or warrants a new check (action A from the consolidation plan). +2. If a new check, decide whether it belongs in an existing skill (`audit-identify`, `audit-attribution`, etc.) or warrants a whole new skill (action B). Action B has a very high bar — Area 2 (attribution) clearing it is the rare case. +3. Add a one-paragraph entry here citing the engagement (anonymized), the customer-scale impact (blocked merges/day, event-loss share, attribution-coverage drop), and the file:line of the new check. +4. Cross-link from the check's `details` JSON to this provenance file when a one-line citation helps the operator (e.g. `"source_pattern_scale": "~80k blocked merges/day at multi-SDK SaaS A"`). + +The goal is that six months from now, "why does this check exist?" has a one-paragraph answer with real engagement scale, not a guess. diff --git a/context/skills/audit-autocapture/references/4-report.md b/context/skills/audit-autocapture/references/4-report.md index 83943ce5..68a1dec7 100644 --- a/context/skills/audit-autocapture/references/4-report.md +++ b/context/skills/audit-autocapture/references/4-report.md @@ -93,6 +93,16 @@ This area covers cost-side autocapture health: whether `$autocapture` dominates [Repeat the heading + paragraph + table for each area in ledger order, in case future versions of this skill add new areas.] +### Assumptions and blind spots + +Under each area's table above, render a `### Assumptions and blind spots` subsection per the investigation standards in `posthog-best-practices/references/investigation-standards.md` (standard 3). Answer the four questions in plain prose, ≤4 sentences total: +- Which code paths or files this area did NOT check that could change the findings. +- Which runtime assumptions are unproven by the static code (mount order, async timing, route gating). +- Alternative explanations for the patterns the checks flagged. +- What you would verify in the live PostHog project (event volumes, property fill rates, dashboard usage) to confirm or refute the most important findings. + +When an area produced only `pass` rows, write `_No findings to qualify; the standard checks for this area passed cleanly._` and skip the four-question rundown. + ## About this audit This audit ran the PostHog `audit-autocapture` skill — a focused, read-only check of autocapture health across two lenses: **fix** (correctness) and **optimize** (cost). Fix checks scan the project source; optimize checks additionally query the PostHog project via MCP in read-only mode (and gracefully skip when MCP is unavailable). diff --git a/context/skills/audit-events/references/4-report.md b/context/skills/audit-events/references/4-report.md index 86a76877..5afb981e 100644 --- a/context/skills/audit-events/references/4-report.md +++ b/context/skills/audit-events/references/4-report.md @@ -93,6 +93,16 @@ This area covers cost-side event capture health: whether captured events are act [Repeat the heading + paragraph + table for each area in ledger order, in case future versions of this skill add new areas.] +### Assumptions and blind spots + +Under each area's table above, render a `### Assumptions and blind spots` subsection per the investigation standards in `posthog-best-practices/references/investigation-standards.md` (standard 3). Answer the four questions in plain prose, ≤4 sentences total: +- Which code paths or files this area did NOT check that could change the findings. +- Which runtime assumptions are unproven by the static code (mount order, async timing, route gating). +- Alternative explanations for the patterns the checks flagged. +- What you would verify in the live PostHog project (event volumes, property fill rates, dashboard usage) to confirm or refute the most important findings. + +When an area produced only `pass` rows, write `_No findings to qualify; the standard checks for this area passed cleanly._` and skip the four-question rundown. + ## About this audit This audit ran the PostHog `audit-events` skill — a focused, read-only check of event capture health across two lenses: **fix** (correctness and quality) and **optimize** (cost). Fix checks scan the project source; optimize checks additionally query the PostHog project via MCP in read-only mode (and gracefully skip when MCP is unavailable). diff --git a/context/skills/audit-feature-flags/references/2-feature-flags-fix.md b/context/skills/audit-feature-flags/references/2-feature-flags-fix.md index fba6a845..b6b5e807 100644 --- a/context/skills/audit-feature-flags/references/2-feature-flags-fix.md +++ b/context/skills/audit-feature-flags/references/2-feature-flags-fix.md @@ -4,11 +4,13 @@ next_step: 3-feature-flags-optimize.md # Step 2 — Feature flags (fix) -This step resolves three correctness checks **in parallel**, one subagent per check: +This step resolves five correctness checks **in parallel**, one subagent per check: - `ff-bootstrap-when-known-set` - `ff-await-readiness` - `ff-default-values` +- `ff-bootstrap-distinct-id-mismatch` +- `ff-identified-only-pre-auth-targeting` ## Status @@ -18,9 +20,9 @@ Emit before dispatching: [STATUS] Auditing feature flag correctness ``` -## Action — dispatch three subagents in one message +## Action — dispatch five subagents in one message -Make **three `Agent` tool calls in a single message** so they run concurrently. Wait for all three to return, then continue to `3-feature-flags-optimize.md`. Do not run any other tools between dispatch and the next step. +Make **five `Agent` tool calls in a single message** so they run concurrently. Wait for all five to return, then continue to `3-feature-flags-optimize.md`. Do not run any other tools between dispatch and the next step. The bundled `best-practices.md` reference holds PostHog's authoritative guidance on flag bootstrapping, readiness, and default values. It's typically at `.claude/skills/audit-feature-flags/references/best-practices.md`; if that path doesn't exist, discover it with `Glob` `**/skills/audit-feature-flags/references/best-practices.md`. Each subagent reads it once before judging. @@ -144,6 +146,93 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for Return when the call completes. Do not write the audit report. ``` -## After all three return +### Task D — `ff-bootstrap-distinct-id-mismatch` + +`description`: `Audit ff-bootstrap-distinct-id-mismatch` + +`prompt`: +``` +You are an audit subagent. Resolve exactly one rule and return: ff-bootstrap-distinct-id-mismatch. + +Read this skill's bundled `bootstrapping.md` reference once (typically `.claude/skills/audit-feature-flags/references/bootstrapping.md`; otherwise discover with `Glob` `**/skills/audit-feature-flags/references/bootstrapping.md`). + +Background: `bootstrap.distinctID` (or `bootstrap: { distinctID: ... }`) lets the host application seed the SDK's distinct_id at init time — usually for SSR/SSG scenarios where the server already knows the user. But if the value passed doesn't match either the user's eventual stable id (after `identify()`) or the SDK's natural anonymous id, it overrides the identity chain in ways that break later merges. Two failure modes: +1. `distinctID` set to a per-request random / session UUID — the SDK considers itself "already identified" with that UUID; the next `identify(realUserId)` is blocked from merging anonymous activity. +2. `distinctID` set to a known user id but the project ALSO calls `identify(differentId)` shortly after — the two ids race; whichever loses creates an orphan profile. + +Run **two** Greps in parallel: +- `bootstrap[\s\S]{0,40}distinctID|bootstrap[\s\S]{0,40}distinct_id|distinctID\s*:` — bootstrap-with-distinctID sites. +- `posthog\.identify\(` — every identify call (so the subagent can cross-reference). + +Read each file that contains a bootstrap.distinctID hit, once. For each site, determine: +- What value is being passed (literal, variable, request-scoped, randomly generated)? +- Is the same value later passed to `posthog.identify()`? If yes, that's the safe pattern (matching SSR hydration). +- Is the value request-scoped / per-render (e.g. `crypto.randomUUID()`, `Math.random()`, a Next.js per-request id)? If yes, this is the failure mode. + +Rule: +- pass: no `bootstrap.distinctID` usage detected, OR the bootstrapped value is stable across requests and matches the value passed to a later identify() call. +- warning: `bootstrap.distinctID` is set to a value that appears request-scoped, randomly generated, or otherwise volatile — the next identify() call will be blocked from merging anonymous activity. +- error: `bootstrap.distinctID` is set to one value and `posthog.identify()` is called immediately after with a DIFFERENT value on the same code path — orphan profile guaranteed. + +Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for id `ff-bootstrap-distinct-id-mismatch`, including `file` (path:line of the bootstrap site) and `details` as compact JSON: + +``` +{ + "bootstrap_distinct_id_site_count": , + "examples": [ + {"file": "", "issue": "volatile-bootstrap-id | bootstrap-identify-mismatch | safe-ssr-hydration"} + ] +} +``` + +Return when the call completes. Do not write the audit report. +``` + +### Task E — `ff-identified-only-pre-auth-targeting` + +`description`: `Audit ff-identified-only-pre-auth-targeting` + +`prompt`: +``` +You are an audit subagent. Resolve exactly one rule and return: ff-identified-only-pre-auth-targeting. + +Read this skill's bundled `best-practices.md` reference once (typically `.claude/skills/audit-feature-flags/references/best-practices.md`; otherwise discover with `Glob` `**/skills/audit-feature-flags/references/best-practices.md`). + +Background: when `person_profiles: 'identified_only'` is set (the recommended default for most B2B SaaS), anonymous visitors don't create person profiles. If a feature flag targets users by person properties AND that flag is evaluated on a pre-auth surface (landing page, pricing page, signup form), the anonymous user has no person profile for the flag to evaluate against, so the flag silently returns its default value. The variant the operator intended to ship to "users in the EU" / "users on the Pro plan" never reaches anyone visiting before login. This is a silent failure — the flag appears to work for identified users but the anonymous-traffic branch quietly never fires. + +Run **three** Greps in parallel: +- `person_profiles\s*:|personProfiles\s*:` — locate the person_profiles setting. +- `getFeatureFlag\(|isFeatureEnabled\(|useFeatureFlag\(|getFeatureFlagPayload\(` — every flag-eval call site. +- `posthog\.identify\(` — every identify call (used to classify a surface as pre-auth or post-auth). + +Step 1 — read the file(s) containing `person_profiles` hits to determine the configured value. If unset, the posthog-js default is `'identified_only'`. Record `mode` as `identified_only`, `always`, `never`, or `unset (defaults to identified_only)`. + +Step 2 — if mode is NOT `identified_only` (or unset), resolve `pass` with `details: "skip: person_profiles is not identified_only"` and return. + +Step 3 — for each flag-eval call site, read the surrounding file once. Classify it as **pre-auth** if it lives in: landing pages, marketing routes, pricing pages, signup/login UI components that render before the user authenticates, public homepage components, or any route gated to anonymous-only access. Classify as **post-auth** if the file also calls `posthog.identify()` in the same flow, requires authenticated session via middleware, or lives under a `/(app)/`, `/dashboard/`, `/(authenticated)/` style route. + +Step 4 — for each pre-auth flag-eval site, attempt to determine whether the flag's targeting condition references person properties. The skill can't read PostHog flag definitions; instead, flag any pre-auth eval whose flag key suggests person-property targeting (variants gated on plan, country, persona, role, signup_method, etc.) — name patterns like `eu-banner`, `pro-only-cta`, `enterprise-pricing-variant`. When ambiguous, default to warning and let the operator confirm. + +Rule: +- pass: mode is not identified_only, OR no flag-eval call sites run on pre-auth surfaces, OR all pre-auth flag evals pass property overrides at eval time (`getFeatureFlag(key, { personProperties: {...} })` or equivalent). +- suggestion: 1–2 pre-auth flag-eval call sites exist but flag names don't strongly suggest person-property targeting — recommend the operator confirm flag definitions in PostHog. +- warning: 3+ pre-auth flag-eval call sites OR any pre-auth flag-eval whose flag name strongly suggests person-property targeting — anonymous users silently get default values. Recommend either passing property overrides at eval time, switching to `posthog.bootstrap.featureFlags` with server-computed values, or moving the eval behind authentication. + +Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for id `ff-identified-only-pre-auth-targeting`, including `file` (path:line of the most representative pre-auth flag-eval) and `details` as compact JSON: + +``` +{ + "person_profiles_mode": "identified_only | always | never | unset", + "pre_auth_flag_eval_count": , + "examples": [ + {"file": "", "flag_key": "", "suspected_property_targeting": } + ] +} +``` + +Return when the call completes. Do not write the audit report. +``` + +## After all five return Continue to **`3-feature-flags-optimize.md`**. diff --git a/context/skills/audit-feature-flags/references/4-report.md b/context/skills/audit-feature-flags/references/4-report.md index e4d4158f..b1ad9122 100644 --- a/context/skills/audit-feature-flags/references/4-report.md +++ b/context/skills/audit-feature-flags/references/4-report.md @@ -96,6 +96,16 @@ This area covers cost-side `/flags` health: unreferenced-but-active flags that c [Repeat the heading + paragraph + table for each area in ledger order, in case future versions of this skill add new areas.] +### Assumptions and blind spots + +Under each area's table above, render a `### Assumptions and blind spots` subsection per the investigation standards in `posthog-best-practices/references/investigation-standards.md` (standard 3). Answer the four questions in plain prose, ≤4 sentences total: +- Which code paths or files this area did NOT check that could change the findings. +- Which runtime assumptions are unproven by the static code (mount order, async timing, route gating). +- Alternative explanations for the patterns the checks flagged. +- What you would verify in the live PostHog project (event volumes, property fill rates, dashboard usage) to confirm or refute the most important findings. + +When an area produced only `pass` rows, write `_No findings to qualify; the standard checks for this area passed cleanly._` and skip the four-question rundown. + ## About this audit This audit ran the PostHog `audit-feature-flags` skill — a focused, read-only check of feature flag health across two lenses: **fix** (correctness) and **optimize** (cost). Fix checks scan the project source; optimize checks additionally query the PostHog project via MCP in read-only mode (and gracefully skip when MCP is unavailable). The billed endpoint is `/flags` (the renamed `/decide`). diff --git a/context/skills/audit-identify/config.yaml b/context/skills/audit-identify/config.yaml index 66c30f9d..f99c87e9 100644 --- a/context/skills/audit-identify/config.yaml +++ b/context/skills/audit-identify/config.yaml @@ -12,6 +12,7 @@ shared_docs: - https://posthog.com/docs/getting-started/identify-users.md - https://posthog.com/docs/product-analytics/cutting-costs.md - https://posthog.com/docs/libraries/js/config.md + - https://posthog.com/docs/data/anonymous-vs-identified-events.md variants: - id: all display_name: PostHog audit — identify diff --git a/context/skills/audit-identify/references/2-identify-fix.md b/context/skills/audit-identify/references/2-identify-fix.md index 2b982095..a5e4bd3d 100644 --- a/context/skills/audit-identify/references/2-identify-fix.md +++ b/context/skills/audit-identify/references/2-identify-fix.md @@ -4,16 +4,17 @@ next_step: 3-identify-lifecycle.md # Step 2 — Identify (fix) -This step resolves four correctness checks **in parallel**, one subagent per check. These are the same ids the broader PostHog audit seeds — `audit-identify` reuses them so a fix once made here is observable from either entry point. +This step resolves five correctness checks **in parallel**, one subagent per check. The first four ids are reused by the broader PostHog audit so a fix once made here is observable from either entry point. The fifth (`identify-sequential-calls`) is audit-identify-specific. - `identify-stable-distinct-id` - `identify-not-late` - `cross-runtime-distinct-id` - `identify-reset-on-logout` +- `identify-sequential-calls` ## Skip case — no `posthog.identify` calls found -If Step 1's identify grep returned **zero** hits, resolve all four checks in a single `audit_resolve_checks` call with `status: "pass"` and `details: "skip: no posthog.identify call sites detected"`. Then continue to **`3-identify-optimize.md`**. Do not dispatch subagents. +If Step 1's identify grep returned **zero** hits, resolve all five checks in a single `audit_resolve_checks` call with `status: "pass"` and `details: "skip: no posthog.identify call sites detected"`. Then continue to **`3-identify-optimize.md`**. Do not dispatch subagents. ## Status @@ -23,9 +24,9 @@ Emit before dispatching: [STATUS] Auditing identify correctness ``` -## Action — dispatch four subagents in one message +## Action — dispatch five subagents in one message -Make **four `Agent` tool calls in a single message** so they run concurrently. Wait for all four to return, then continue to `3-identify-optimize.md`. Do not run any other tools between dispatch and the next step. +Make **five `Agent` tool calls in a single message** so they run concurrently. Wait for all five to return, then continue to `3-identify-optimize.md`. Do not run any other tools between dispatch and the next step. The bundled `identify-users.md` reference holds PostHog's authoritative guidance on `distinct_id`, `identify()` ordering, and cross-runtime identity. It's typically at `.claude/skills/audit-identify/references/identify-users.md`; if that path doesn't exist, discover it with `Glob` `**/skills/audit-identify/references/identify-users.md`. Each subagent reads it once before judging. @@ -117,6 +118,44 @@ Rule: Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for id `identify-reset-on-logout`, including `file` (path:line of the most relevant logout or reset site) and `details` (one-line explanation). Return when the call completes. Do not write the audit report. ``` -## After all four return +### Task E — `identify-sequential-calls` + +`description`: `Audit identify-sequential-calls` + +`prompt`: +``` +You are an audit subagent. Resolve exactly one rule and return: identify-sequential-calls. + +Read this skill's bundled `identify-users.md` reference once (typically `.claude/skills/audit-identify/references/identify-users.md`; otherwise discover with `Glob` `**/skills/audit-identify/references/identify-users.md`). + +Background: a common identity-merge failure is two `posthog.identify()` calls firing back-to-back in the same auth flow with different first arguments — for example, `posthog.identify(authProviderUid)` followed by `posthog.identify(internalUserId)`. The first call stamps the auth provider id as the device identity. By the time the second call runs, the SDK considers the device already-identified, so the merge from the *anonymous* distinct id to `internalUserId` is blocked. The user is now split across two profiles: the anon profile correctly merged to `authProviderUid`, and `internalUserId` as a brand-new profile with no anonymous activity. This pattern has caused 30k+ blocked merges/day at multi-SDK SaaS customers. + +Run **one** Grep: `posthog\.identify\(`. Read each file that contains a hit, once. + +For each file, find pairs of `posthog.identify()` calls that: +1. Live in the same enclosing function, hook, effect, or auth-callback handler (close enough that they reliably execute in one flow), AND +2. Pass **different** first arguments (different variable names, different property accesses, or one literal and one variable). + +Pay special attention to auth-callback handlers, login success handlers, and middleware/route guards. Treat one identify() call wrapped in an `if`/`else` branching on the same condition as a SINGLE logical call site (it can't fire twice in one flow); only flag pairs that are actually sequential on the same code path. + +Rule: +- pass: every flow has at most one identify() call, OR all identify() calls in the same flow pass the same first argument. +- error: one or more flows have two or more identify() calls with different first arguments executing in sequence. List the file:line of each offender and the names of the two distinct ids. + +Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for id `identify-sequential-calls`, including `file` (path:line of the most representative offending pair) and `details` as compact JSON: + +``` +{ + "sequential_pair_count": , + "examples": [ + {"file": "", "first_id": "", "second_id": "", "flow": ""} + ] +} +``` + +Return when the call completes. Do not write the audit report. +``` + +## After all five return Continue to **`3-identify-lifecycle.md`**. Do not write the report yet — that's Step 4's job after Step 3 finishes. diff --git a/context/skills/audit-identify/references/3-identify-lifecycle.md b/context/skills/audit-identify/references/3-identify-lifecycle.md index c87d8ffa..90d9e248 100644 --- a/context/skills/audit-identify/references/3-identify-lifecycle.md +++ b/context/skills/audit-identify/references/3-identify-lifecycle.md @@ -80,19 +80,24 @@ Return when the call completes. Do not write the audit report. ``` You are an audit subagent. Resolve exactly one rule and return: identify-alias-usage. -Read this skill's bundled `identify-users.md` reference once (typically `.claude/skills/audit-identify/references/identify-users.md`; otherwise discover with `Glob` `**/skills/audit-identify/references/identify-users.md`). Focus on the `alias()` guidance: modern PostHog SDKs handle the anonymous → identified merge automatically when `identify()` is called for the first time. `alias()` is mostly legacy and is only needed in narrow cases (e.g. linking a server-side distinct_id to a known account_id when the client never called identify). +Read this skill's bundled `identify-users.md` reference once (typically `.claude/skills/audit-identify/references/identify-users.md`; otherwise discover with `Glob` `**/skills/audit-identify/references/identify-users.md`). Focus on the `alias()` guidance: modern PostHog SDKs handle the anonymous → identified merge automatically when `identify()` is called for the first time. `alias()` is mostly legacy. **Backend `alias()` is particularly hazardous**: when a server aliases a browser's anonymous UUID to a user id, that UUID becomes "identified" in PostHog. When the web SDK later tries to `identify()` with the same UUID as `$anon_distinct_id`, the merge is blocked. This pattern has caused 80–100k blocked merges/day at multi-SDK SaaS customers and is almost never what the operator intended. Run **one** Grep: `posthog\.alias\(|posthog\.createAlias\(`. If zero matches, resolve `pass` with `details: "no alias() usage detected"` and return — this is the healthy default. -Otherwise read each file that contains a hit, once. For each alias() call, determine: -- **Is there also a posthog.identify() call for the same user in the same flow?** If yes, the alias() call is likely redundant — identify() already handles merging. +Otherwise read each file that contains a hit, once. Also run a second Grep `posthog\.identify\(` to know whether the project ever calls `identify()` anywhere (this gates one of the rules below). + +For each alias() call, determine: +- **Where it runs** — client-side (browser bundle) or server-side (Node, Python, edge handler, background worker). +- **Is there a `posthog.identify()` call for the same user nearby?** If yes, the alias() call is likely redundant — identify() already handles merging. - **Is alias() being called with the same id as the current distinct_id?** That's a no-op that still emits a `$create_alias` event. -- **Is alias() being used in a server-side context to link an anonymous client distinct_id to a server-known user id?** That's a legitimate use case — pass. +- **For server-side alias:** is the first argument a value sourced from a request-scoped browser identifier (cookie, request body, header passing the client distinct_id)? If so, this is the backend-aliases-browser-UUID anti-pattern — flag it. The narrow legitimate case is server-only projects that genuinely have no client SDK and need to link an internal id to a previously-used anonymous server id; this is rare. Rule: -- pass: no alias() calls, OR alias() is used only for legitimate server-side identity linking. +- pass: no alias() calls, OR alias() is server-side AND the project has zero `posthog.identify()` call sites anywhere AND the alias arguments don't come from a request-scoped browser identifier (the narrow legitimate "no client SDK" case). - suggestion: alias() is called alongside identify() for the same user — likely redundant. Recommend removal. -- warning: alias() is called with the same id as the current distinct_id, OR alias() is the only identity API used (no identify() anywhere) — the project should switch to identify(). +- warning: alias() is called with the same id as the current distinct_id. +- warning: alias() is the only identity API used (no identify() anywhere on the client) — the project should switch to identify(). +- **error: server-side `alias()` is called with a value sourced from a request-scoped browser identifier (cookie, request body, header) while the project ALSO has a client-side `posthog.identify()`.** This is the backend-aliases-browser-UUID anti-pattern; the first server alias() call identifies the UUID and every later client identify() with the same UUID as `$anon_distinct_id` is silently blocked. Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for id `identify-alias-usage`, with `file` set to the most representative alias() path:line if any, and `details` as compact JSON: @@ -100,8 +105,9 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for { "alias_call_count": , "redundant_with_identify_count": , + "backend_alias_blocks_merges": , "examples": [ - {"file": "", "issue": "redundant-with-identify | self-alias | only-identity-api"} + {"file": "", "issue": "redundant-with-identify | self-alias | only-identity-api | backend-aliases-browser-uuid"} ] } ``` diff --git a/context/skills/audit-identify/references/4-identify-optimize.md b/context/skills/audit-identify/references/4-identify-optimize.md index 3427aba9..8cf72019 100644 --- a/context/skills/audit-identify/references/4-identify-optimize.md +++ b/context/skills/audit-identify/references/4-identify-optimize.md @@ -1,5 +1,5 @@ --- -next_step: 5-report.md +next_step: 5-server-sdk.md --- # Step 4 — Identify (optimize) @@ -239,4 +239,4 @@ Return when the call completes. Do not write the audit report. ## After all four return -Continue to **`5-report.md`**. +Continue to **`5-server-sdk.md`**. diff --git a/context/skills/audit-identify/references/5-server-sdk.md b/context/skills/audit-identify/references/5-server-sdk.md new file mode 100644 index 00000000..9dc8fd45 --- /dev/null +++ b/context/skills/audit-identify/references/5-server-sdk.md @@ -0,0 +1,159 @@ +--- +next_step: 6-report.md +--- + +# Step 5 — Server SDK identity hygiene + +This step resolves three server-side identity checks **in parallel**, one subagent per check: + +- `server-process-person-profile` +- `server-sdk-flush-on-exit` +- `server-set-without-identify` + +These cover failure patterns that are invisible in client-only audits but cause silent data corruption and event loss at scale: backend events overwriting client-set person properties (e.g. `$os` → `"Linux"` on 24% of profiles), serverless / edge / worker processes terminating before the SDK flushes, and `$set` properties firing from server captures with no canonical `identify()` to anchor them. + +## Skip case — no server SDK init detected + +If Step 1's init grep returned hits only for browser-runtime locations (no `posthog-node`, `posthog-python`, `posthog-ruby`, `posthog-go`, `posthog-php`, `posthog-java`, etc. imports; no server-only init), resolve all three checks in a single `audit_resolve_checks` call with `status: "pass"` and `details: "skip: no server-side SDK init detected"`. Then continue to **`6-report.md`**. Do not dispatch subagents. + +## Status + +Emit before dispatching: + +``` +[STATUS] Auditing server-side identity hygiene +``` + +## Action — dispatch three subagents in one message + +Make **three `Agent` tool calls in a single message** so they run concurrently. Wait for all three to return, then continue to `6-report.md`. Do not run any other tools between dispatch and the next step. + +The bundled `anonymous-vs-identified-events.md` reference holds PostHog's authoritative guidance on `$process_person_profile`, person-mode selection, and when person processing should be disabled. It's typically at `.claude/skills/audit-identify/references/anonymous-vs-identified-events.md`; if that path doesn't exist, discover it with `Glob` `**/skills/audit-identify/references/anonymous-vs-identified-events.md`. Each subagent reads it once before judging. + +### Task A — `server-process-person-profile` + +`description`: `Audit server-process-person-profile` + +`prompt`: +``` +You are an audit subagent. Resolve exactly one rule and return: server-process-person-profile. + +Read this skill's bundled `anonymous-vs-identified-events.md` reference once (typically `.claude/skills/audit-identify/references/anonymous-vs-identified-events.md`; otherwise discover with `Glob` `**/skills/audit-identify/references/anonymous-vs-identified-events.md`). + +Background: posthog-node, posthog-python, and other backend SDKs default to processing person profiles on every `capture()` call. The SDK auto-attaches runtime metadata (`$os`, `$lib`, `$lib_version`, etc.) that overwrites whatever the client SDK previously set on the same person. The canonical failure pattern: a backend captures `subscription_upgraded` from a Linux server; the user's profile gets `$os = "Linux"` even though they're a macOS user, because the server SDK's `$os` overrode the client's. The fix is to set `$process_person_profile: false` on backend events that aren't supposed to update person properties (most server-side capture calls fit this). + +Run **two** Greps in parallel: +- `posthog\.capture\(|posthog\.Capture\(|new PostHog\(.+\.capture` — every server-side capture call site. Filter to files that are server-side (Node/Python/Go/Java/Ruby/PHP) — exclude browser files. +- `\$process_person_profile|process_person_profile|processPersonProfile` — explicit usage of the property anywhere. + +Read each file that contains a server-side `capture(` hit, once. For each server-side capture, determine whether: +- The capture is intended to update person properties (it should follow a corresponding `identify()` and pass `$set` / `$set_once`), OR +- The capture is a transactional/business event that shouldn't touch person properties (most cases — `subscription_upgraded`, `webhook_received`, `cron_job_completed`). + +For the second category, check whether `$process_person_profile: false` is set in the properties object. + +Rule: +- pass: every server-side capture either passes `$process_person_profile: false` OR is paired with an explicit identify()/$set in the same flow (intentional person-property update). +- suggestion: 1–3 server-side captures lack `$process_person_profile: false` and don't appear to update person properties intentionally — recommend adding the flag to prevent silent property corruption. +- warning: 4+ server-side captures or any high-frequency server-side capture (cron, webhook, polling loop) without `$process_person_profile: false` — high blast radius for property corruption. + +Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for id `server-process-person-profile`, including `file` (path:line of the most representative offending capture) and `details` as compact JSON: + +``` +{ + "server_capture_call_count": , + "captures_without_flag_count": , + "examples": [ + {"file": "", "event": "", "issue": "missing-flag-on-business-event | missing-flag-on-hot-path"} + ] +} +``` + +Return when the call completes. Do not write the audit report. +``` + +### Task B — `server-sdk-flush-on-exit` + +`description`: `Audit server-sdk-flush-on-exit` + +`prompt`: +``` +You are an audit subagent. Resolve exactly one rule and return: server-sdk-flush-on-exit. + +Background: posthog-node, posthog-python, and posthog-ruby buffer events in memory and flush asynchronously. In long-running servers this is fine — the buffer drains during operation. In serverless functions (AWS Lambda, Vercel Functions, Cloudflare Workers), edge handlers, and background workers (Celery, Sidekiq, BullMQ, RQ), the process can terminate before the buffer drains. Events captured in the last few milliseconds before exit are silently lost. The fix is to call `posthog.shutdown()` (Node, Python) or `await posthog.flush()` before the handler returns or the worker exits. + +Run **three** Greps in parallel: +- `posthog\.shutdown\(|posthog\.Shutdown\(|posthog\.flush\(|await posthog\.flush|posthog\.disable\(` — explicit flush/shutdown calls. +- `posthog\.capture\(|posthog\.Capture\(` — every server-side capture call site. +- `exports\.handler|export\s+(default\s+)?(async\s+)?function\s+handler|app/api/.*/route\.(ts|js)|runtime\s*=\s*['"]edge['"]|celery|sidekiq|bullmq|defineEventHandler|onRequest|export\s+(const|let|var)\s+config\s*=\s*\{[^}]*runtime` — short-lived runtime signals. + +Read each file that contains both a short-lived signal AND a server-side capture, once. Determine whether the handler / worker function calls `posthog.shutdown()` or awaits `posthog.flush()` before returning. + +Rule: +- pass: no short-lived server-side capture sites detected, OR every short-lived capture site flushes/shutdowns before exit. +- pass with details "skip: long-running server only": only persistent-server captures detected (Express middleware, Django views, etc.). +- warning: 1–2 short-lived capture sites without flush/shutdown — events captured in the last milliseconds before exit will be silently lost. +- error: 3+ short-lived capture sites without flush/shutdown, OR any capture inside a background worker (Celery, Sidekiq, BullMQ, RQ) without flush — high event-loss risk. One customer lost a measurable share of subscription events to this pattern in Celery workers. + +Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for id `server-sdk-flush-on-exit`, including `file` (path:line of the most representative un-flushed handler) and `details` as compact JSON: + +``` +{ + "short_lived_capture_site_count": , + "unflushed_site_count": , + "examples": [ + {"file": "", "runtime": "lambda | vercel-function | edge | worker | cron", "issue": "missing-shutdown | missing-flush-await"} + ] +} +``` + +Return when the call completes. Do not write the audit report. +``` + +### Task C — `server-set-without-identify` + +`description`: `Audit server-set-without-identify` + +`prompt`: +``` +You are an audit subagent. Resolve exactly one rule and return: server-set-without-identify. + +Read this skill's bundled `identify-users.md` reference once (typically `.claude/skills/audit-identify/references/identify-users.md`; otherwise discover with `Glob` `**/skills/audit-identify/references/identify-users.md`). + +Background: a backend `posthog.capture()` with `$set` / `$set_once` properties on a user-scoped distinct_id makes that person *appear* identified in PostHog — the profile shows the properties — but no canonical `$identify` event fired and no anonymous → identified merge happened. Anonymous activity from that user's earlier sessions stays orphaned. The fix is either to call `posthog.identify(userId, properties)` (which fires the canonical merge event) instead of `posthog.capture(event, { $set: ... })`, or to ensure a prior `posthog.identify()` call from the client SDK preceded this capture for the same distinct_id. + +Run **two** Greps in parallel: +- `posthog\.capture\(|posthog\.Capture\(` — every server-side capture call site. +- `posthog\.identify\(|posthog\.Identify\(` — every server-side identify call site. + +Read each file that contains a server-side capture, once. For each capture whose properties include `$set`, `$set_once`, `setPersonProperties`, or `setOnce`: +- Is there a `posthog.identify()` call in the same file (or imported module) using the same distinct_id? +- Is the capture's distinct_id sourced from a server-known user record (a stable user id) rather than a request-scoped anonymous id? + +If the project uses ONLY server-side captures (no client SDK detected in Step 1), the canonical identify must come from the server — flag every server $set-without-identify as a problem. + +If the project also has a client SDK, the canonical identify can come from the client; the server-side $set is fine as long as the same distinct_id is used and the client has identified that user at least once. + +Rule: +- pass: no server-side captures include `$set` / `$set_once`, OR every server-side `$set` capture is preceded by a server-side `identify()` for the same distinct_id, OR the client SDK identifies the same distinct_id elsewhere in the codebase. +- warning: 1–2 server-side `$set` captures with no server `identify()` AND no obvious client-side identify covering the same distinct_id. Person properties will be set but no merge happens. +- error: 3+ server-side `$set` captures with no `identify()` anywhere in the project — every affected user has an orphan profile. + +Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for id `server-set-without-identify`, including `file` (path:line of the most representative offending capture) and `details` as compact JSON: + +``` +{ + "server_set_capture_count": , + "uncovered_count": , + "examples": [ + {"file": "", "event": "", "issue": "set-without-server-identify | set-without-any-identify"} + ] +} +``` + +Return when the call completes. Do not write the audit report. +``` + +## After all three return + +Continue to **`6-report.md`**. diff --git a/context/skills/audit-identify/references/5-report.md b/context/skills/audit-identify/references/6-report.md similarity index 66% rename from context/skills/audit-identify/references/5-report.md rename to context/skills/audit-identify/references/6-report.md index 0901a2f8..0449fb24 100644 --- a/context/skills/audit-identify/references/5-report.md +++ b/context/skills/audit-identify/references/6-report.md @@ -2,7 +2,7 @@ next_step: null --- -# Step 5 — Generate the audit report +# Step 6 — Generate the audit report The audit report is rendered **directly from `.posthog-audit-checks.json`** — that file is the source of truth. Every check the wizard seeded for this skill ends up in the report, even passes; nothing is invented. @@ -27,7 +27,7 @@ The report has four sections in this order: 3. **Full audit** — every check the wizard ran, grouped by `area`, including passes. 4. **About this audit** — short closing block explaining what this audit covered. -For the Full audit section, group rows by each distinct `area` value in the ledger, preserving first-seen area order from the JSON. This skill produces three areas: **Identification** (fix), **Identification — Lifecycle** (quality), and **Identification — Optimize** (cost). Render whatever areas the ledger actually contains. +For the Full audit section, group rows by each distinct `area` value in the ledger, preserving first-seen area order from the JSON. This skill produces four areas: **Identification** (fix), **Identification — Lifecycle** (quality), **Identification — Optimize** (cost), and **Identification — Server SDK** (server-side hygiene). Render whatever areas the ledger actually contains. For each area, write a one-paragraph framing immediately under the area heading, then the table. @@ -57,7 +57,7 @@ If there are no problematic items, write `_No issues found — your $identify se ## Recommended actions -Numbered list, ordered by severity (errors → warnings → suggestions), then by area within a severity (Identification → Identification — Lifecycle → Identification — Optimize). Each item is **three sentences**, in this order: +Numbered list, ordered by severity (errors → warnings → suggestions), then by area within a severity (Identification → Identification — Lifecycle → Identification — Optimize → Identification — Server SDK). Each item is **three sentences**, in this order: 1. **What's wrong** — the finding, written as a one-sentence diagnosis derived from `details`. 2. **Why it matters** — one sentence on the data-quality or cost consequence. For fix-side checks: which downstream artifact (funnels, retention, person count, experiments) this finding contaminates. For optimize-side checks: the billing or volume impact, quoting the ratio or count from `details`. @@ -68,11 +68,13 @@ Format: 1. **[Area] · [label]** — [what's wrong]. _Why it matters:_ [why-it-matters]. _Fix:_ [how-to-fix at `file:line`]. See [docs]([area docs url]). Suggested docs URLs: -- Identification fix checks → https://posthog.com/docs/getting-started/identify-users +- Identification fix checks (including `identify-sequential-calls`) → https://posthog.com/docs/getting-started/identify-users - `identify-set-discipline`, `identify-alias-usage` → https://posthog.com/docs/getting-started/identify-users - `identify-groupidentify-correctness` → https://posthog.com/docs/product-analytics/group-analytics -- `identify-person-profiles-mode` → https://posthog.com/docs/data/anonymous-vs-identified-events +- `identify-person-profiles-mode`, `server-process-person-profile` → https://posthog.com/docs/data/anonymous-vs-identified-events - `identify-isidentified-guard`, `identify-duplicate-identify-per-session`, `identify-duplicate-groupidentify-per-session` → https://posthog.com/docs/product-analytics/cutting-costs +- `server-sdk-flush-on-exit` → https://posthog.com/docs/libraries/node (and language equivalents — `python`, `ruby`, `go`, `cloudflare-workers`) +- `server-set-without-identify` → https://posthog.com/docs/getting-started/identify-users If there are no actions, write `_Nothing to fix._`. @@ -102,11 +104,29 @@ This area covers cost-side `$identify` health: whether the `person_profiles` set |-------|--------|------|---------| | [label] | [status] | [file] | [details] | +### Identification — Server SDK + +This area covers server-side identity hygiene: whether backend `capture()` calls disable person processing where appropriate (preventing silent property corruption like `$os = "Linux"` on macOS users' profiles), whether serverless / edge / worker processes flush the SDK buffer before exit (preventing event loss), and whether server-side `$set` properties are paired with a canonical `identify()` (preventing orphan profiles). Only runs when a server-side SDK is detected. + +| Check | Status | File | Details | +|-------|--------|------|---------| +| [label] | [status] | [file] | [details] | + [Repeat the heading + paragraph + table for each area in ledger order, in case future versions of this skill add new areas.] +### Assumptions and blind spots + +Under each area's table above, render a `### Assumptions and blind spots` subsection per the investigation standards in `posthog-best-practices/references/investigation-standards.md` (standard 3). Answer the four questions in plain prose, ≤4 sentences total: +- Which code paths or files this area did NOT check that could change the findings. +- Which runtime assumptions are unproven by the static code (mount order, async timing, route gating). +- Alternative explanations for the patterns the checks flagged. +- What you would verify in the live PostHog project (event volumes, property fill rates, dashboard usage) to confirm or refute the most important findings. + +When an area produced only `pass` rows, write `_No findings to qualify; the standard checks for this area passed cleanly._` and skip the four-question rundown. + ## About this audit -This audit ran the PostHog `audit-identify` skill — a focused, read-only check of $identify health across two lenses: **fix** (correctness) and **optimize** (cost). Fix checks scan the project source; optimize checks additionally query the PostHog project via MCP in read-only mode (and gracefully skip when MCP is unavailable). +This audit ran the PostHog `audit-identify` skill — a focused, read-only check of $identify health across three lenses: **fix** (correctness), **optimize** (cost), and **server SDK** (backend identity hygiene). Fix and server-SDK checks scan the project source; optimize checks additionally query the PostHog project via MCP in read-only mode (and gracefully skip when MCP is unavailable). Server-SDK checks only run when a server-side PostHog SDK is detected. - `error` items break correctness now (identity broken, captures merged across users). Fix first. - `warning` items work today but cause subtle bugs or noticeably elevated cost. Fix when convenient. diff --git a/context/skills/audit-session-replay/references/4-report.md b/context/skills/audit-session-replay/references/4-report.md index b330c616..c63802c0 100644 --- a/context/skills/audit-session-replay/references/4-report.md +++ b/context/skills/audit-session-replay/references/4-report.md @@ -95,6 +95,16 @@ This area covers cost-side replay health: whether the project's sample rate matc [Repeat the heading + paragraph + table for each area in ledger order, in case future versions of this skill add new areas.] +### Assumptions and blind spots + +Under each area's table above, render a `### Assumptions and blind spots` subsection per the investigation standards in `posthog-best-practices/references/investigation-standards.md` (standard 3). Answer the four questions in plain prose, ≤4 sentences total: +- Which code paths or files this area did NOT check that could change the findings. +- Which runtime assumptions are unproven by the static code (mount order, async timing, route gating). +- Alternative explanations for the patterns the checks flagged. +- What you would verify in the live PostHog project (event volumes, property fill rates, dashboard usage) to confirm or refute the most important findings. + +When an area produced only `pass` rows, write `_No findings to qualify; the standard checks for this area passed cleanly._` and skip the four-question rundown. + ## About this audit This audit ran the PostHog `audit-session-replay` skill — a focused, read-only check of session replay health across two lenses: **fix** (correctness) and **optimize** (cost). Fix checks scan the project source; optimize checks additionally query the PostHog project via MCP in read-only mode (and gracefully skip when MCP is unavailable). diff --git a/context/skills/audit/references/2-init.md b/context/skills/audit/references/2-init.md index 8d6a14a3..4bd493da 100644 --- a/context/skills/audit/references/2-init.md +++ b/context/skills/audit/references/2-init.md @@ -4,7 +4,7 @@ next_step: 3-identification.md # Step 2 — Init correctness -This step resolves exactly one check: `init-correct`. Manifests and SDK versions are already resolved (Step 1). Identification call sites belong to Step 3 and event-capture call sites to Step 4 — do not scan for them here. +This step resolves two checks: `init-correct` and `init-not-duplicated`. Manifests and SDK versions are already resolved (Step 1). Identification call sites belong to Step 3 and event-capture call sites to Step 4 — do not scan for them here. ## Status @@ -16,27 +16,35 @@ Emit: ## Action -Locate the project's PostHog init by issuing whatever `Grep` and `Read` calls are needed in parallel. Confirm the init exists, runs in the right runtime for the detected SDK + framework, and sources its token from an env variable (not hardcoded). Also check `.env*` files to confirm the token env var is actually set. Reverse-proxy / `api_host` configuration belongs to Step 4 — don't evaluate it here. +Locate every PostHog init site by issuing one `Grep` for `posthog\.init\(|new PostHog\(|posthog\.Posthog\(|Posthog\(` plus whatever `Read` calls are needed. Confirm at least one init exists, runs in the right runtime for the detected SDK + framework, and sources its token from an env variable (not hardcoded). Also check `.env*` files to confirm the token env var is actually set. Reverse-proxy / `api_host` configuration belongs to Step 4 — don't evaluate it here. Use the detected SDK + framework from Step 1 to know what to look for: the canonical init filename, runtime, and shape vary by framework. If the host project already ships a PostHog integration skill, use that as the source of truth. Skills are typically under `.claude/skills/`; if that directory doesn't exist (some projects keep skills under `agents/skills/`, plain `skills/`, etc.), discover any candidates with one `Glob` pattern: `**/skills/**/SKILL.md`. Read the matching skill before judging. When no integration skill is available, rely on general framework knowledge — and stay conservative on `init-correct` (prefer `warning` over `error` when the convention is unclear). +For `init-not-duplicated`, count the init sites the grep found and group them by runtime (browser vs. server vs. mobile). The check fires when more than one init site exists in the same runtime — the browser bundle running `posthog.init()` twice will race to stamp `$set_once` properties like `$initial_pathname` and `$initial_referrer`, corrupting attribution at scale. A browser init plus a server init is fine; two browser inits is not. Treat conditional inits inside an `if (typeof window === 'undefined')` / `if (typeof window !== 'undefined')` branch on the same code path as ONE logical site per runtime (only one branch executes). + ## Resolution rules `init-correct`: -- `pass`: init present, env-sourced token, runtime-appropriate location. +- `pass`: at least one init present, env-sourced token, runtime-appropriate location. - `error`: init missing, hardcoded token, or wrong runtime (e.g. server-only init for a browser-side framework). - `warning`: init present but in a non-canonical location for the framework. +`init-not-duplicated`: +- `pass`: at most one init site per runtime. +- `warning`: two or more init sites in the same runtime that could plausibly both execute. Report the file:line of each and which runtime. The dual-init `$set_once` race corrupted attribution at scale at one multi-SDK SaaS customer (~165k blocked merges/day). +- `error`: two or more browser-runtime init sites confirmed to run on the same page (e.g. one in the root layout / provider, another in a child component that always mounts). + ## Resolve -Single call to `mcp__wizard-tools__audit_resolve_checks` with one update: +Single call to `mcp__wizard-tools__audit_resolve_checks` with two updates: ``` { "updates": [ - { "id": "init-correct", "status": "pass|error|warning", "file": "", "details": "..." } + { "id": "init-correct", "status": "pass|error|warning", "file": "", "details": "..." }, + { "id": "init-not-duplicated", "status": "pass|warning|error", "file": "", "details": "..." } ] } ``` diff --git a/context/skills/audit/references/5-report.md b/context/skills/audit/references/5-report.md index 48667db8..a0c0f922 100644 --- a/context/skills/audit/references/5-report.md +++ b/context/skills/audit/references/5-report.md @@ -98,7 +98,7 @@ One `Edit`: One `Edit`: - `old_string`: `` -- `new_string`: the per-area headings + paragraphs + tables, in ledger order. +- `new_string`: the per-area headings + paragraphs + tables + per-area `#### Assumptions and blind spots` subsection, in ledger order. The blind-spots subsection lives directly under each area's table, following the per-area body template below. If the Full audit section is large (many areas, many checks), you may split it across multiple Edits by including per-area placeholders in the original skeleton and filling each with one Edit. Most audits fit in one Edit. @@ -155,6 +155,10 @@ For each `area` from the ledger, in first-seen order: | Check | Status | File | Details | |-------|--------|------|---------| | [label] | [status] | [file] | [details] | + +#### Assumptions and blind spots + +[Per the investigation standards in `posthog-best-practices/references/investigation-standards.md`, standard 3. ≤4 sentences answering: which code paths were not checked, which runtime assumptions are unproven by static code, what alternative explanations exist for the patterns found, and what to verify in the live PostHog project to confirm the most important findings. When the area produced only `pass` rows, write `_No findings to qualify; the standard checks for this area passed cleanly._` instead.] ``` After the report is written, emit a line so the wizard can surface the path to the user: @@ -235,7 +239,7 @@ For a typical doctor deliverable the placeholder set is: |---|---|---| | Summary | `__SUMMARY_OVERVIEW__` (one paragraph), `__SUMMARY_COUNTS__` (one bulletList), `__SUMMARY_PROBLEMATIC__` (one table or fallback paragraph) | 3 | | Recommended actions | `__RECOMMENDED_ACTIONS__` (one orderedList, or fallback paragraph if none) | 1 | -| Full audit per area | `__FULL_AUDIT__HEADING__` (level-3 heading), `__FULL_AUDIT__PARAGRAPH__` (framing paragraph), `__FULL_AUDIT__TABLE__` (per-area table) | 3 × N (one set per distinct area in the ledger) | +| Full audit per area | `__FULL_AUDIT__HEADING__` (level-3 heading), `__FULL_AUDIT__PARAGRAPH__` (framing paragraph), `__FULL_AUDIT__TABLE__` (per-area table), `__FULL_AUDIT__BLIND_SPOTS_HEADING__` (level-4 heading "Assumptions and blind spots"), `__FULL_AUDIT__BLIND_SPOTS__` (paragraph) | 5 × N (one set per distinct area in the ledger) | | About this audit | `__ABOUT_PARAGRAPH__`, `__ABOUT_BULLETS__`, `__ABOUT_CLOSING__` | 3 | Use uppercased, underscored area names (e.g. `INSTALLATION`, `IDENTIFICATION`, `EVENT_CAPTURE`). Build the skeleton with that count and call `notebooks-create`: @@ -266,13 +270,19 @@ Use uppercased, underscored area names (e.g. `INSTALLATION`, `IDENTIFICATION`, ` {"type":"paragraph","content":[{"type":"text","text":"__FULL_AUDIT_INSTALLATION_HEADING__"}]}, {"type":"paragraph","content":[{"type":"text","text":"__FULL_AUDIT_INSTALLATION_PARAGRAPH__"}]}, {"type":"paragraph","content":[{"type":"text","text":"__FULL_AUDIT_INSTALLATION_TABLE__"}]}, + {"type":"paragraph","content":[{"type":"text","text":"__FULL_AUDIT_INSTALLATION_BLIND_SPOTS_HEADING__"}]}, + {"type":"paragraph","content":[{"type":"text","text":"__FULL_AUDIT_INSTALLATION_BLIND_SPOTS__"}]}, {"type":"paragraph","content":[{"type":"text","text":"__FULL_AUDIT_IDENTIFICATION_HEADING__"}]}, {"type":"paragraph","content":[{"type":"text","text":"__FULL_AUDIT_IDENTIFICATION_PARAGRAPH__"}]}, {"type":"paragraph","content":[{"type":"text","text":"__FULL_AUDIT_IDENTIFICATION_TABLE__"}]}, + {"type":"paragraph","content":[{"type":"text","text":"__FULL_AUDIT_IDENTIFICATION_BLIND_SPOTS_HEADING__"}]}, + {"type":"paragraph","content":[{"type":"text","text":"__FULL_AUDIT_IDENTIFICATION_BLIND_SPOTS__"}]}, {"type":"paragraph","content":[{"type":"text","text":"__FULL_AUDIT_EVENT_CAPTURE_HEADING__"}]}, {"type":"paragraph","content":[{"type":"text","text":"__FULL_AUDIT_EVENT_CAPTURE_PARAGRAPH__"}]}, {"type":"paragraph","content":[{"type":"text","text":"__FULL_AUDIT_EVENT_CAPTURE_TABLE__"}]}, - // … add three placeholders per additional area in the ledger … + {"type":"paragraph","content":[{"type":"text","text":"__FULL_AUDIT_EVENT_CAPTURE_BLIND_SPOTS_HEADING__"}]}, + {"type":"paragraph","content":[{"type":"text","text":"__FULL_AUDIT_EVENT_CAPTURE_BLIND_SPOTS__"}]}, + // … add five placeholders per additional area in the ledger … {"type":"heading","attrs":{"level":2},"content":[{"type":"text","text":"About this audit"}]}, {"type":"paragraph","content":[{"type":"text","text":"__ABOUT_PARAGRAPH__"}]}, diff --git a/context/skills/posthog-best-practices/description.md b/context/skills/posthog-best-practices/description.md index 3bba4249..1705a980 100644 --- a/context/skills/posthog-best-practices/description.md +++ b/context/skills/posthog-best-practices/description.md @@ -10,6 +10,7 @@ If the project uses a specific PostHog product mentioned below, consult the rele - If the project uses Experiments, also consult `references/experiments.md`. - If the project uses Feature Flags, also consult `references/feature-flags.md`. - If the project uses Session Replay, also consult `references/session-replay.md`. +- If you are running an audit skill (anything that produces a findings report), also consult `references/investigation-standards.md` for the provenance, evidence, and adversarial-review standards that apply to every audit finding. ## Reference files diff --git a/context/skills/posthog-best-practices/references/investigation-standards.md b/context/skills/posthog-best-practices/references/investigation-standards.md new file mode 100644 index 00000000..c5fbaf8b --- /dev/null +++ b/context/skills/posthog-best-practices/references/investigation-standards.md @@ -0,0 +1,65 @@ +# Investigation standards for PostHog audits + +Use this reference when an audit skill produces a findings report. These standards exist to make audit findings trustworthy and prevent the failure mode where a plausible-sounding finding ("PostHog is initialized twice") turns out to be wrong because the investigator inferred behavior from documentation rather than locating the actual code. + +Wrong findings in customer-facing audits erode trust and cause rework. The standards below are cheap to apply during the audit and expensive to skip. + +## The three standards + +### 1. Provenance on every claim + +Every non-`pass` finding must cite the exact file path, line number(s), and a short code snippet (or equivalent search-result evidence). This is enforced by the ledger schema — `audit_resolve_checks` requires a `file` field for non-skip findings — but the standard goes further than the schema. + +**What counts as provenance:** +- File path + line number + code snippet showing the behavior. +- Grep / search results showing presence or absence of a pattern. +- Package.json / lock-file entries for version claims. +- Config-file values for configuration claims. + +**What does NOT count:** +- "PostHog is likely configured with…" (inference from docs). +- "Based on the framework, it probably…" (assumption from conventions). +- "The codebase appears to…" (hedge language without evidence). + +**When you cannot find the code that would prove a claim:** resolve with `pass` and `details: "skip: …"` listing what you searched for and where. An honest skip is more useful than a plausible guess. The operator can point you at the right place. + +### 2. Verification evidence inline with each finding + +The evidence IS the finding. Showing the search result, file path, or config value that proves the behavior is not a separate step; it's part of stating the finding. This is what the `details` JSON field is for — populate it with the actual values, counts, and examples that justify the verdict, not a paraphrase of the rule. + +For **presence findings** (something exists that shouldn't), evidence is the code snippet or matched line. For **absence findings** (something is missing that should exist), evidence is the search you ran and the directories you checked. Absence findings are the most dangerous to get wrong — only resolve "missing reset() on logout" after checking every plausible logout / signout code path, not just the first one you found. + +### 3. Adversarial self-review per area + +After every check in an area is resolved, the report step's "Assumptions and blind spots" subsection answers four questions for that area: + +1. **What code paths did I NOT check** that could change these findings? (e.g. "I checked the main auth callback but not the magic-link flow.") +2. **What runtime assumptions am I making** that the static code doesn't prove? (e.g. "I'm assuming these two components mount on the same page, but they might be route-gated.") +3. **Are there alternative explanations** for the patterns I found? (e.g. "the dual init might be intentional for different PostHog projects in dev vs prod.") +4. **What would I verify in the live PostHog project** to confirm or refute the most important findings? (e.g. "query `$feature_flag_called` events to see if both inits are actually firing.") + +This subsection is not about being uncertain. It's about making the boundaries of the investigation explicit so the operator knows where to probe further. Run the self-review per area, not at the end of the whole report — five areas of context at once is too much to hold. + +## Ordering: investigate, document, review + +For each area: + +1. **Investigate** — search the codebase, read the relevant code, form claims via the subagents the skill dispatches. +2. **Document findings** — each finding gets provenance + verification evidence inline. Resolutions go to the ledger as the subagents finish. +3. **Adversarial self-review** — after all the area's checks are resolved, the report step renders the "Assumptions and blind spots" subsection. + +Do not defer the adversarial review to the end of the full report. It belongs at the bottom of each area, while that area's findings are still in working memory. + +## Tests vs. evidence + +**Evidence is sufficient for the audit** when the code that creates the behavior is visible. A finding of "dual `posthog.init()` on the signup page" supported by two file paths and two init() calls is actionable without a test. + +**Runnable tests belong to the fix phase**, not the audit. When the operator implements the recommended fix, they should write a test confirming the fix works. The audit's job is to identify and prove the problem, not to build the test harness. + +**Exception:** when a finding depends on runtime behavior that static code can't prove (async timing, race conditions, cross-component mount order), note this explicitly in the adversarial review. The recommended next step is a live verification (PostHog data, temporary logging), not a test in the audit. + +## Common failure modes + +- **Scope creep through decomposition.** An investigation that starts as "check if identify() is called correctly" expands into a full identity resolution architecture review, then person-processing deep-dive, then flag evaluation analysis. Each expansion feels justified, but the audit skill's seeded checks define the scope — resist drift unless the operator explicitly asks for depth. +- **Polishing findings instead of shipping.** Reorganizing report structure, adding context sections, improving formatting on findings that are already clear. The operator wants the findings, not the report. Ship rough findings; polish only if asked. +- **Comprehensive when MVP was the ask.** When the operator asked one question, deliver the minimum answer first, then offer to go deeper. From 4d037e1839d88d37e6c19cf7f67130cac072c3cf Mon Sep 17 00:00:00 2001 From: Joshua Ordehi <45109738+ordehi@users.noreply.github.com> Date: Fri, 19 Jun 2026 14:43:54 -0400 Subject: [PATCH 2/3] fix(audit-attribution): self-seed the ledger in step 1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The runtime resolution path (wizard audit attribution -> dispatchFamily -> agentSkillConfig) does not pre-seed the audit ledger; audit_resolve_checks would error 'unknown check id' on every check otherwise. Follow the events-audit pattern: skill calls audit_seed_checks directly in step 1 with the full 8-check + write-report payload. Note: the existing narrow audit leaves (audit-identify, audit-events, audit-feature-flags, audit-autocapture, audit-session-replay) appear to have the same latent issue — their description.md says 'seeded by the wizard' but the wizard's agentSkillConfig does not seed. Out of scope for this PR; flagged for a follow-up. --- .../references/1-presence.md | 43 +++++++++++++++---- .../audit-attribution/references/4-report.md | 2 +- 2 files changed, 36 insertions(+), 9 deletions(-) diff --git a/context/skills/audit-attribution/references/1-presence.md b/context/skills/audit-attribution/references/1-presence.md index a7f601ee..d1a0ee6c 100644 --- a/context/skills/audit-attribution/references/1-presence.md +++ b/context/skills/audit-attribution/references/1-presence.md @@ -2,20 +2,27 @@ next_step: 2-attribution-fix.md --- -# Step 1 — Presence detector +# Step 1 — Presence detector + seed the ledger -This step decides whether the rest of the audit has anything to look at, and records signals later steps need. Run it **before** any other work. Resolve zero ledger checks here — this step is gating only. +This step decides whether the rest of the audit has anything to look at, seeds the audit ledger, and records signals later steps need. Run it **before** any other work. + +## Tools + +Load via `ToolSearch select:Grep,mcp__wizard-tools__audit_seed_checks,mcp__wizard-tools__audit_resolve_checks` once at the start of this step. Subsequent steps reuse `audit_resolve_checks` to patch each check as it resolves, so it stays loaded. ## Status -Emit: +Emit, in order: ``` [STATUS] Detecting PostHog and attribution surfaces +[STATUS] Seeding audit checklist ``` ## Action +### a. Detect presence + Run **two `Grep` calls in parallel**, both with `output_mode: "files_with_matches"`: 1. PostHog init surface — any of: @@ -23,15 +30,35 @@ Run **two `Grep` calls in parallel**, both with `output_mode: "files_with_matche 2. Attribution / acquisition signals — any of: `utm_source|utm_medium|utm_campaign|gclid|fbclid|msclkid|msfclkid|li_fat_id|ttclid|twclid|partner_id|referrer_id` -## Decision +### b. Abort or continue + +- **Init grep returns zero hits anywhere in the project:** emit `[ABORT] No PostHog SDK initialization found` and stop. The wizard catches `[ABORT]` and terminates the run. Do NOT seed the ledger in this case. +- **Init found:** continue to (c). Even projects with no explicit click-id capture rely on PostHog's built-in UTM auto-capture; each step's individual rules decide whether to skip or warn based on the kind of evidence present. -- **Init grep returns zero hits anywhere in the project:** emit `[ABORT] No PostHog SDK initialization found` and stop. The wizard catches `[ABORT]` and terminates the run. -- **Init found:** continue, regardless of whether attribution signals were detected. Even projects with no explicit click-id capture rely on PostHog's built-in UTM auto-capture; each step's individual rules decide whether to skip or warn based on the kind of evidence present. +### c. Seed the audit ledger + +The ledger lives at `.posthog-audit-checks.json` and renders live in the wizard sidebar / "Audit plan" tab. **The runtime does not pre-seed this skill's ledger** — call `mcp__wizard-tools__audit_seed_checks` directly here with the exact payload below. The tool replaces the file atomically, so calling it once at the start of every run is safe. + +```json +{ + "checks": [ + { "id": "attribution-utm-survives-landing", "area": "Attribution", "label": "UTM params survive landing-page routing", "status": "pending" }, + { "id": "attribution-survives-auth-redirect", "area": "Attribution", "label": "UTM params survive auth/OAuth redirects", "status": "pending" }, + { "id": "attribution-first-touch-set-once", "area": "Attribution", "label": "First-touch attribution properties use $set_once", "status": "pending" }, + { "id": "attribution-custom-click-ids", "area": "Attribution", "label": "Ad-platform click ids captured (gclid, fbclid, msclkid)", "status": "pending" }, + { "id": "attribution-on-conversion-events", "area": "Attribution", "label": "Conversion events carry attribution context", "status": "pending" }, + { "id": "attribution-cross-subdomain-cookie", "area": "Attribution — Configuration", "label": "cross_subdomain_cookie configured for multi-subdomain projects", "status": "pending" }, + { "id": "attribution-cookieless-mode-impact", "area": "Attribution — Configuration", "label": "cookieless_mode tradeoff acknowledged", "status": "pending" }, + { "id": "attribution-consent-integration", "area": "Attribution — Configuration", "label": "Consent banner wired to PostHog opt_in/opt_out", "status": "pending" }, + { "id": "write-report", "area": "Write report", "label": "Render posthog-audit-attribution-report.md", "status": "pending" } + ] +} +``` ## Record acquisition signal for later steps -Keep the acquisition-signal grep result in working memory. Step 2's `attribution-custom-click-ids` check uses it to decide whether to warn on absence (project clearly runs paid acquisition but doesn't capture click ids) vs. skip (no paid-acquisition signal anywhere — silence is fine). +Keep the acquisition-signal grep result in working memory. Step 2's `attribution-custom-click-ids` check uses it to decide whether to warn on absence (project clearly runs paid acquisition but doesn't capture click ids) vs. skip (no paid-acquisition signal anywhere — silence is fine). Step 3's `attribution-cookieless-mode-impact` reuses the same signal. -Do not read any files in this step. Do not call `audit_resolve_checks`. Do not preload future steps. +Do not read any project files in this step. Do not call `audit_resolve_checks`. Do not preload future steps. Continue to **`2-attribution-fix.md`**. diff --git a/context/skills/audit-attribution/references/4-report.md b/context/skills/audit-attribution/references/4-report.md index f4599e6b..bb9e1aa7 100644 --- a/context/skills/audit-attribution/references/4-report.md +++ b/context/skills/audit-attribution/references/4-report.md @@ -18,7 +18,7 @@ Emit: `Read` the ledger once, then transform every entry into the report below. Use `area`, `label`, `status`, `file`, and `details` from each entry verbatim where the report calls for them. -`Write` `posthog-audit-attribution-report.md` at the project root with the structure shown below. After the report is written, delete `.posthog-audit-checks.json`. +`Write` `posthog-audit-attribution-report.md` at the project root with the structure shown below. After the markdown lands on disk, resolve the `write-report` ledger row to `pass` so the wizard sidebar advances. Then delete `.posthog-audit-checks.json`. The report has four sections in this order: From b297a3b4fcede7e680d22d1261156e3807434888 Mon Sep 17 00:00:00 2001 From: Joshua Ordehi <45109738+ordehi@users.noreply.github.com> Date: Fri, 19 Jun 2026 15:22:40 -0400 Subject: [PATCH 3/3] fix(audit-skills): escape nested code fences inside prompt blocks Subagent prompts in audit skills wrap their text in a triple-backtick fence but contain a triple-backtick JSON example inside. Per CommonMark, the inner fence closes the outer block, breaking rendering. Bump the outer fence to four backticks so the inner three-backtick JSON blocks nest cleanly. Affects 24 prompt blocks across 7 reference files in audit-attribution, audit-feature-flags, and audit-identify. --- .../references/2-attribution-fix.md | 20 +++++++++---------- .../references/3-attribution-config.md | 12 +++++------ .../references/2-feature-flags-fix.md | 20 +++++++++---------- .../references/2-identify-fix.md | 4 ++-- .../references/3-identify-lifecycle.md | 12 +++++------ .../references/4-identify-optimize.md | 16 +++++++-------- .../audit-identify/references/5-server-sdk.md | 12 +++++------ 7 files changed, 48 insertions(+), 48 deletions(-) diff --git a/context/skills/audit-attribution/references/2-attribution-fix.md b/context/skills/audit-attribution/references/2-attribution-fix.md index 9ad1af0d..7b43b91a 100644 --- a/context/skills/audit-attribution/references/2-attribution-fix.md +++ b/context/skills/audit-attribution/references/2-attribution-fix.md @@ -31,7 +31,7 @@ The bundled `utm-segmentation.md` reference holds PostHog's authoritative guidan `description`: `Audit attribution-utm-survives-landing` `prompt`: -``` +```` You are an audit subagent. Resolve exactly one rule and return: attribution-utm-survives-landing. Read this skill's bundled `utm-segmentation.md` reference once (typically `.claude/skills/audit-attribution/references/utm-segmentation.md`). @@ -65,14 +65,14 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for ``` Return when the call completes. Do not write the audit report. -``` +```` ### Task B — `attribution-survives-auth-redirect` `description`: `Audit attribution-survives-auth-redirect` `prompt`: -``` +```` You are an audit subagent. Resolve exactly one rule and return: attribution-survives-auth-redirect. Read this skill's bundled `utm-segmentation.md` reference once (typically `.claude/skills/audit-attribution/references/utm-segmentation.md`). @@ -110,14 +110,14 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for ``` Return when the call completes. Do not write the audit report. -``` +```` ### Task C — `attribution-first-touch-set-once` `description`: `Audit attribution-first-touch-set-once` `prompt`: -``` +```` You are an audit subagent. Resolve exactly one rule and return: attribution-first-touch-set-once. Read this skill's bundled `identify-users.md` reference once (typically `.claude/skills/audit-attribution/references/identify-users.md`). @@ -150,14 +150,14 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for ``` Return when the call completes. Do not write the audit report. -``` +```` ### Task D — `attribution-custom-click-ids` `description`: `Audit attribution-custom-click-ids` `prompt`: -``` +```` You are an audit subagent. Resolve exactly one rule and return: attribution-custom-click-ids. Read this skill's bundled `utm-segmentation.md` reference once (typically `.claude/skills/audit-attribution/references/utm-segmentation.md`). @@ -191,14 +191,14 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for ``` Return when the call completes. Do not write the audit report. -``` +```` ### Task E — `attribution-on-conversion-events` `description`: `Audit attribution-on-conversion-events` `prompt`: -``` +```` You are an audit subagent. Resolve exactly one rule and return: attribution-on-conversion-events. Read this skill's bundled `utm-segmentation.md` reference once (typically `.claude/skills/audit-attribution/references/utm-segmentation.md`). @@ -236,7 +236,7 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for ``` Return when the call completes. Do not write the audit report. -``` +```` ## After all five return diff --git a/context/skills/audit-attribution/references/3-attribution-config.md b/context/skills/audit-attribution/references/3-attribution-config.md index 76ef6ff6..cb92d57b 100644 --- a/context/skills/audit-attribution/references/3-attribution-config.md +++ b/context/skills/audit-attribution/references/3-attribution-config.md @@ -29,7 +29,7 @@ The bundled `config.md` (posthog-js config reference) and `data-collection.md` ( `description`: `Audit attribution-cross-subdomain-cookie` `prompt`: -``` +```` You are an audit subagent. Resolve exactly one rule and return: attribution-cross-subdomain-cookie. Read this skill's bundled `config.md` reference once (typically `.claude/skills/audit-attribution/references/config.md`). @@ -62,14 +62,14 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for ``` Return when the call completes. Do not write the audit report. -``` +```` ### Task B — `attribution-cookieless-mode-impact` `description`: `Audit attribution-cookieless-mode-impact` `prompt`: -``` +```` You are an audit subagent. Resolve exactly one rule and return: attribution-cookieless-mode-impact. Read this skill's bundled `data-collection.md` reference once (typically `.claude/skills/audit-attribution/references/data-collection.md`). @@ -102,14 +102,14 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for ``` Return when the call completes. Do not write the audit report. -``` +```` ### Task C — `attribution-consent-integration` `description`: `Audit attribution-consent-integration` `prompt`: -``` +```` You are an audit subagent. Resolve exactly one rule and return: attribution-consent-integration. Read this skill's bundled `data-collection.md` reference once (typically `.claude/skills/audit-attribution/references/data-collection.md`). @@ -145,7 +145,7 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for ``` Return when the call completes. Do not write the audit report. -``` +```` ## After all three return diff --git a/context/skills/audit-feature-flags/references/2-feature-flags-fix.md b/context/skills/audit-feature-flags/references/2-feature-flags-fix.md index b6b5e807..9448f263 100644 --- a/context/skills/audit-feature-flags/references/2-feature-flags-fix.md +++ b/context/skills/audit-feature-flags/references/2-feature-flags-fix.md @@ -31,7 +31,7 @@ The bundled `best-practices.md` reference holds PostHog's authoritative guidance `description`: `Audit ff-bootstrap-when-known-set` `prompt`: -``` +```` You are an audit subagent. Resolve exactly one rule and return: ff-bootstrap-when-known-set. Read this skill's bundled `best-practices.md` reference once (typically `.claude/skills/audit-feature-flags/references/best-practices.md`; otherwise discover with `Glob` `**/skills/audit-feature-flags/references/best-practices.md`). Focus on the bootstrapping guidance — when an initial flag set is already known at app start (e.g. computed server-side, persisted in a cookie, or passed through SSR props), client-side `posthog.init` should set `bootstrap.featureFlags` so the first render has the right values without a `/flags` round trip. @@ -67,14 +67,14 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for ``` Return when the call completes. Do not write the audit report. -``` +```` ### Task B — `ff-await-readiness` `description`: `Audit ff-await-readiness` `prompt`: -``` +```` You are an audit subagent. Resolve exactly one rule and return: ff-await-readiness. Read this skill's bundled `best-practices.md` reference once (typically `.claude/skills/audit-feature-flags/references/best-practices.md`; otherwise discover with `Glob` `**/skills/audit-feature-flags/references/best-practices.md`). Focus on the readiness / "have the value before you need it" section — client-side flag evaluation is async, so any flag-eval before `onFeatureFlags` fires (or before the `loaded` callback runs, or before `bootstrap.featureFlags` is set) returns `undefined`, which is **not** `false`. Misreading the loading gap is one of the most common flag bugs. @@ -106,14 +106,14 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for ``` Return when the call completes. Do not write the audit report. -``` +```` ### Task C — `ff-default-values` `description`: `Audit ff-default-values` `prompt`: -``` +```` You are an audit subagent. Resolve exactly one rule and return: ff-default-values. Read this skill's bundled `best-practices.md` reference once (typically `.claude/skills/audit-feature-flags/references/best-practices.md`; otherwise discover with `Glob` `**/skills/audit-feature-flags/references/best-practices.md`). Focus on the "undefined is not false" and per-flag default guidance — `getFeatureFlag('key')` returns `undefined` during the loading window and may also return `undefined` when PostHog is unreachable or quota-limited. A per-flag default (via `?? 'control'`, a wrapper helper, or the SDK's `default_value`/`defaultValue` option when supported) controls what users see during these windows. @@ -144,14 +144,14 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for ``` Return when the call completes. Do not write the audit report. -``` +```` ### Task D — `ff-bootstrap-distinct-id-mismatch` `description`: `Audit ff-bootstrap-distinct-id-mismatch` `prompt`: -``` +```` You are an audit subagent. Resolve exactly one rule and return: ff-bootstrap-distinct-id-mismatch. Read this skill's bundled `bootstrapping.md` reference once (typically `.claude/skills/audit-feature-flags/references/bootstrapping.md`; otherwise discover with `Glob` `**/skills/audit-feature-flags/references/bootstrapping.md`). @@ -186,14 +186,14 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for ``` Return when the call completes. Do not write the audit report. -``` +```` ### Task E — `ff-identified-only-pre-auth-targeting` `description`: `Audit ff-identified-only-pre-auth-targeting` `prompt`: -``` +```` You are an audit subagent. Resolve exactly one rule and return: ff-identified-only-pre-auth-targeting. Read this skill's bundled `best-practices.md` reference once (typically `.claude/skills/audit-feature-flags/references/best-practices.md`; otherwise discover with `Glob` `**/skills/audit-feature-flags/references/best-practices.md`). @@ -231,7 +231,7 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for ``` Return when the call completes. Do not write the audit report. -``` +```` ## After all five return diff --git a/context/skills/audit-identify/references/2-identify-fix.md b/context/skills/audit-identify/references/2-identify-fix.md index a5e4bd3d..673faef4 100644 --- a/context/skills/audit-identify/references/2-identify-fix.md +++ b/context/skills/audit-identify/references/2-identify-fix.md @@ -123,7 +123,7 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for `description`: `Audit identify-sequential-calls` `prompt`: -``` +```` You are an audit subagent. Resolve exactly one rule and return: identify-sequential-calls. Read this skill's bundled `identify-users.md` reference once (typically `.claude/skills/audit-identify/references/identify-users.md`; otherwise discover with `Glob` `**/skills/audit-identify/references/identify-users.md`). @@ -154,7 +154,7 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for ``` Return when the call completes. Do not write the audit report. -``` +```` ## After all five return diff --git a/context/skills/audit-identify/references/3-identify-lifecycle.md b/context/skills/audit-identify/references/3-identify-lifecycle.md index 90d9e248..c56e089f 100644 --- a/context/skills/audit-identify/references/3-identify-lifecycle.md +++ b/context/skills/audit-identify/references/3-identify-lifecycle.md @@ -33,7 +33,7 @@ The bundled `identify-users.md` reference holds PostHog's authoritative guidance `description`: `Audit identify-set-discipline` `prompt`: -``` +```` You are an audit subagent. Resolve exactly one rule and return: identify-set-discipline. Read this skill's bundled `identify-users.md` reference once (typically `.claude/skills/audit-identify/references/identify-users.md`; otherwise discover with `Glob` `**/skills/audit-identify/references/identify-users.md`). Focus on the `$set` vs `$set_once` guidance: `$set` overwrites every time and should be reserved for properties that genuinely change (plan tier, last_seen_app_version). `$set_once` is for first-touch attributes that should never be overwritten (initial_referrer, signup_date, first_seen_country). Calling `$set` on every `capture()` inflates person-property version count and is the most common form of "person property bloat". @@ -70,14 +70,14 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for ``` Return when the call completes. Do not write the audit report. -``` +```` ### Task B — `identify-alias-usage` `description`: `Audit identify-alias-usage` `prompt`: -``` +```` You are an audit subagent. Resolve exactly one rule and return: identify-alias-usage. Read this skill's bundled `identify-users.md` reference once (typically `.claude/skills/audit-identify/references/identify-users.md`; otherwise discover with `Glob` `**/skills/audit-identify/references/identify-users.md`). Focus on the `alias()` guidance: modern PostHog SDKs handle the anonymous → identified merge automatically when `identify()` is called for the first time. `alias()` is mostly legacy. **Backend `alias()` is particularly hazardous**: when a server aliases a browser's anonymous UUID to a user id, that UUID becomes "identified" in PostHog. When the web SDK later tries to `identify()` with the same UUID as `$anon_distinct_id`, the merge is blocked. This pattern has caused 80–100k blocked merges/day at multi-SDK SaaS customers and is almost never what the operator intended. @@ -113,14 +113,14 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for ``` Return when the call completes. Do not write the audit report. -``` +```` ### Task C — `identify-groupidentify-correctness` `description`: `Audit identify-groupidentify-correctness` `prompt`: -``` +```` You are an audit subagent. Resolve exactly one rule and return: identify-groupidentify-correctness. Read this skill's bundled `identify-users.md` reference once (typically `.claude/skills/audit-identify/references/identify-users.md`; otherwise discover with `Glob` `**/skills/audit-identify/references/identify-users.md`). Focus on the group analytics section: `posthog.group(type, key, properties?)` sets the active group context (so subsequent events are attributed to that group) and may call `groupIdentify` internally to set group properties. `posthog.groupIdentify(type, key, properties)` is the lower-level API that emits a `$groupidentify` event to set group properties without changing the active context. @@ -154,7 +154,7 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for ``` Return when the call completes. Do not write the audit report. -``` +```` ## After all three return diff --git a/context/skills/audit-identify/references/4-identify-optimize.md b/context/skills/audit-identify/references/4-identify-optimize.md index 8cf72019..66aefa49 100644 --- a/context/skills/audit-identify/references/4-identify-optimize.md +++ b/context/skills/audit-identify/references/4-identify-optimize.md @@ -32,7 +32,7 @@ The bundled `cutting-costs.md` reference holds PostHog's authoritative cost-redu `description`: `Audit identify-person-profiles-mode` `prompt`: -``` +```` You are an audit subagent. Resolve exactly one rule and return: identify-person-profiles-mode. Read this skill's bundled `cutting-costs.md` reference once (typically `.claude/skills/audit-identify/references/cutting-costs.md`; otherwise discover with `Glob` `**/skills/audit-identify/references/cutting-costs.md`). Focus on the `person_profiles` section — anonymous events are roughly 4x cheaper than identified events, so `'identified_only'` is the right default when most traffic is anonymous, and `'always'` is only justified when nearly all traffic is identified. @@ -77,14 +77,14 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for ``` Return when the call completes. Do not write the audit report. -``` +```` ### Task B — `identify-isidentified-guard` `description`: `Audit identify-isidentified-guard` `prompt`: -``` +```` You are an audit subagent. Resolve exactly one rule and return: identify-isidentified-guard. Read this skill's bundled `cutting-costs.md` reference once (typically `.claude/skills/audit-identify/references/cutting-costs.md`; otherwise discover with `Glob` `**/skills/audit-identify/references/cutting-costs.md`). Focus on the section that recommends guarding `identify()` with `_isIdentified()` to avoid emitting a `$identify` event on every page load / re-render. @@ -113,14 +113,14 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for ``` Return when the call completes. Do not write the audit report. -``` +```` ### Task C — `identify-duplicate-identify-per-session` `description`: `Audit identify-duplicate-identify-per-session` `prompt`: -``` +```` You are an audit subagent. Resolve exactly one rule and return: identify-duplicate-identify-per-session. This check requires PostHog MCP access. If the MCP server is unavailable, auth fails, or any call errors after one retry: resolve with `suggestion` and `details: "PostHog MCP unavailable — could not measure duplicate $identify events"`. Do not block the audit. @@ -179,14 +179,14 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for ``` Return when the call completes. Do not write the audit report. -``` +```` ### Task D — `identify-duplicate-groupidentify-per-session` `description`: `Audit identify-duplicate-groupidentify-per-session` `prompt`: -``` +```` You are an audit subagent. Resolve exactly one rule and return: identify-duplicate-groupidentify-per-session. This check requires PostHog MCP access. If the MCP server is unavailable, auth fails, or any call errors after one retry: resolve with `suggestion` and `details: "PostHog MCP unavailable — could not measure duplicate $groupidentify events"`. Do not block the audit. @@ -235,7 +235,7 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for ``` Return when the call completes. Do not write the audit report. -``` +```` ## After all four return diff --git a/context/skills/audit-identify/references/5-server-sdk.md b/context/skills/audit-identify/references/5-server-sdk.md index 9dc8fd45..aacd1117 100644 --- a/context/skills/audit-identify/references/5-server-sdk.md +++ b/context/skills/audit-identify/references/5-server-sdk.md @@ -35,7 +35,7 @@ The bundled `anonymous-vs-identified-events.md` reference holds PostHog's author `description`: `Audit server-process-person-profile` `prompt`: -``` +```` You are an audit subagent. Resolve exactly one rule and return: server-process-person-profile. Read this skill's bundled `anonymous-vs-identified-events.md` reference once (typically `.claude/skills/audit-identify/references/anonymous-vs-identified-events.md`; otherwise discover with `Glob` `**/skills/audit-identify/references/anonymous-vs-identified-events.md`). @@ -70,14 +70,14 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for ``` Return when the call completes. Do not write the audit report. -``` +```` ### Task B — `server-sdk-flush-on-exit` `description`: `Audit server-sdk-flush-on-exit` `prompt`: -``` +```` You are an audit subagent. Resolve exactly one rule and return: server-sdk-flush-on-exit. Background: posthog-node, posthog-python, and posthog-ruby buffer events in memory and flush asynchronously. In long-running servers this is fine — the buffer drains during operation. In serverless functions (AWS Lambda, Vercel Functions, Cloudflare Workers), edge handlers, and background workers (Celery, Sidekiq, BullMQ, RQ), the process can terminate before the buffer drains. Events captured in the last few milliseconds before exit are silently lost. The fix is to call `posthog.shutdown()` (Node, Python) or `await posthog.flush()` before the handler returns or the worker exits. @@ -108,14 +108,14 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for ``` Return when the call completes. Do not write the audit report. -``` +```` ### Task C — `server-set-without-identify` `description`: `Audit server-set-without-identify` `prompt`: -``` +```` You are an audit subagent. Resolve exactly one rule and return: server-set-without-identify. Read this skill's bundled `identify-users.md` reference once (typically `.claude/skills/audit-identify/references/identify-users.md`; otherwise discover with `Glob` `**/skills/audit-identify/references/identify-users.md`). @@ -152,7 +152,7 @@ Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for ``` Return when the call completes. Do not write the audit report. -``` +```` ## After all three return