feat(audit): consolidate FDE audit knowledge into context-mill skills#193
Merged
Conversation
Adds the marketing-attribution audit dimension (a documented gap), folds eight production-validated identity / flag failure patterns into the existing audit skills, and formalizes the FDE investigation methodology as a cross-cutting reference. New skill: audit-attribution - wizard audit attribution (role: command, parent: audit) - 5 fix-side checks: utm-survives-landing, survives-auth-redirect, first-touch-set-once, custom-click-ids, on-conversion-events - 3 config-side checks: cross-subdomain-cookie, cookieless-mode-impact, consent-integration - Pins shared_docs to utm-segmentation, libraries/js/config, privacy/data-collection, getting-started/identify-users - Bundles engagement-provenance.md mapping each check to its source customer engagement and observed scale audit-identify - New step 5 (server-SDK) with server-process-person-profile, server-sdk-flush-on-exit, server-set-without-identify - New identify-sequential-calls check (Task E in step 2) - Tightened identify-alias-usage polarity: error on backend-aliases- browser-uuid even when alias is on the server side - Old 5-report.md renamed to 6-report.md; report area expanded to include Server SDK; shared_docs picks up anonymous-vs-identified-events.md audit - New init-not-duplicated check in step 2 (dual-init set_once race) audit-feature-flags - New ff-bootstrap-distinct-id-mismatch check (bootstrap overriding identity) - New ff-identified-only-pre-auth-targeting check (silent flag default on anonymous users when person_profiles is identified_only) Investigation methodology - New posthog-best-practices/references/investigation-standards.md - Every audit report step now renders an Assumptions and blind spots subsection per area (placeholder count in the audit notebook-upload skeleton bumped 3 to 5 per area) Companion work (not in this PR): - Wizard repo: seed updates so new check ids appear in the audit ledger TUI - posthog.com docs PRs: callouts for net-new patterns Tests: 92 passing. Build emits audit-attribution.zip and registers wizard audit attribution under cliEntries.
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Results will be posted here when complete. |
Contributor
Author
|
/wizard-ci basic-integration Smoke test for manifest integrity. This PR doesn't touch the integration skills, but it adds a new skill ( Note for reviewers: the new audit logic itself ( |
🧙 Wizard CI ResultsTrigger ID:
Configuration
|
The runtime resolution path (wizard audit attribution -> dispatchFamily -> agentSkillConfig) does not pre-seed the audit ledger; audit_resolve_checks would error 'unknown check id' on every check otherwise. Follow the events-audit pattern: skill calls audit_seed_checks directly in step 1 with the full 8-check + write-report payload. Note: the existing narrow audit leaves (audit-identify, audit-events, audit-feature-flags, audit-autocapture, audit-session-replay) appear to have the same latent issue — their description.md says 'seeded by the wizard' but the wizard's agentSkillConfig does not seed. Out of scope for this PR; flagged for a follow-up.
gewenyu99
approved these changes
Jun 19, 2026
gewenyu99
left a comment
gewenyu99
left a comment
Collaborator
There was a problem hiding this comment.
Nice, there are some places where ``` are malformed because the prompt: points to ``` with nested ```.
Otherwise good
|
|
||
| Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for id `attribution-cookieless-mode-impact`, including `file` (path:line of the cookieless_mode setting, or the init if unset) and `details` as compact JSON: | ||
|
|
||
| ``` |
Collaborator
There was a problem hiding this comment.
I think this breaks the ``` wrap here. You may need to do the hack where you use ``````
Collaborator
|
Next time you can do |
Subagent prompts in audit skills wrap their text in a triple-backtick fence but contain a triple-backtick JSON example inside. Per CommonMark, the inner fence closes the outer block, breaking rendering. Bump the outer fence to four backticks so the inner three-backtick JSON blocks nest cleanly. Affects 24 prompt blocks across 7 reference files in audit-attribution, audit-feature-flags, and audit-identify.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Consolidates eight production-validated identity / flag failure patterns and the marketing-attribution audit dimension (a documented gap in our coverage) into the context-mill audit suite. Also formalizes the FDE investigation methodology as a cross-cutting reference.
Background, cross-reference, and per-gap recommendations:
vault/posthog/fde/skills/dev/ph/consolidation-plan.md(FDE team only).What's in this PR
New skill:
audit-attribution→wizard audit attribution5 fix-side checks (UTM survival on landing, UTM survival through OAuth, first-touch
$set_oncediscipline, custom click-id capture, attribution on conversion events) + 3 config-side checks (cross_subdomain_cookie,cookieless_mode× attribution, consent integration). Step 1 self-seeds the ledger (see "Heads-up" below). Includesengagement-provenance.mddocumenting which customer engagement each check traces back to and at what scale.shared_docspins todata/utm-segmentation.md,libraries/js/config.md,privacy/data-collection.md,getting-started/identify-users.md. Patterns with no canonical doc are inlined — see "Companion docs PRs" below.Patches to existing skills
audit/references/2-init.md— newinit-not-duplicatedcheck (dual-init$set_oncerace that's caused ~165k blocked merges/day at one multi-SDK customer).audit-identify/references/2-identify-fix.md— newidentify-sequential-callscheck (auth-provider-UID → internal-UID pattern, ~30k blocked merges/day at the same customer).audit-identify/references/3-identify-lifecycle.md— tightenedidentify-alias-usagepolarity toerroron backend-aliases-browser-UUID (~80k blocked merges/day). The existing rule's "legitimate server-side alias" allowance covered exactly the failure pattern.audit-identify/references/5-server-sdk.md(new step) — three checks:server-process-person-profile,server-sdk-flush-on-exit,server-set-without-identify. Old5-report.mdrenamed to6-report.md.anonymous-vs-identified-events.mdadded toshared_docs.audit-feature-flags/references/2-feature-flags-fix.md— newff-bootstrap-distinct-id-mismatchandff-identified-only-pre-auth-targetingchecks.Investigation methodology
posthog-best-practices/references/investigation-standards.md— the FDE provenance + verification + adversarial-review standards as a cross-cutting reference.### Assumptions and blind spotssubsection per area.audit/references/5-report.mdnotebook-upload placeholder count bumped 3 → 5 per area.Why now
This work folds knowledge that has been living in customer-specific deliverables (and FDE vault docs) into the audit suite where it's discoverable and runnable by anyone. After this lands, a Convex-style audit is
wizard audit && wizard audit identify && wizard audit attribution && wizard audit events && wizard audit feature-flags— five reports with MCP-backed data-side cross-checks the static prompt couldn't do — instead of a 400-line prompt that has to be tuned per customer.Wizard side: nothing required
Earlier I planned a companion wizard PR for program scaffolding and seed updates. That's not needed — the CLI overhaul (wizard #617 / context-mill #178, merged 2026-06-08) made command registration purely runtime.
dispatchFamilyresolvesaudit attributionagainstcliEntriesinskill-menu.jsonon every wizard invocation, then runs it through the genericagentSkillConfig. As soon as a context-mill release with this PR is cut,wizard audit attributionworks on the next wizard invocation. No wizard release required.Heads-up: latent issue in existing narrow audit leaves (out of scope here)
While verifying that
audit-attributionwould seed correctly, I found that the existing narrow audit leaves —audit-identify,audit-events,audit-feature-flags,audit-autocapture,audit-session-replay— say in theirdescription.md"The audit ledger is seeded by the wizard" but the wizard'sagentSkillConfig(which all of them route through after the CLI overhaul) does not seed. The top-levelauditandevents-auditboth have wizard-sideseedAuditLedger()calls in theirindex.ts; the narrow leaves don't, and their skill markdown doesn't callaudit_seed_checkseither. So either they're producing empty reports today or the agent is improvising whenaudit_resolve_checksreturns "unknown check id".The fix is the same pattern this PR uses for
audit-attribution: add anaudit_seed_checkscall to step 1 of each affected skill. Out of scope for this PR; happy to do as a follow-up.audit-attributionitself is fine — see commit4d037e1.Companion PR(s): posthog.com docs
Inlined rule prose in this PR for patterns with no canonical doc. Planned parallel docs work to land the durable explanations:
contents/docs/data/utm-through-auth-redirects.mdxor section inutm-segmentation.mdx. Zero existing coverage; highest leverage.posthog.init()race (N1)product-analytics/identity-resolution.mdxidentity-resolution.mdxposthog.alias()(N3)identity-resolution.mdx\$set-without-\$identify(N4)product-analytics/person-properties.mdxbootstrap.distinctIDmismatch (N7)feature-flags/bootstrapping.mdxidentified_only+ flag targeting (N8)feature-flags/best-practices.mdxcookieless_mode× attribution (N9)product-analytics/privacy.mdxWhen each docs PR merges, the corresponding inline prose in the skill markdown collapses to a thin rule wrapper that defers to the doc via
shared_docs:.Role-change ack
audit-attributionships asrole: command.CONTRIBUTING.mdcriterion 5 asks for wizard-maintainer review on command-surface additions. Flagging for ack on this PR; if the team prefersrole: skillfirst, happy to downgrade and promote later.Test plan
npm test— 92/92 passingnpm run build— clean;audit-attribution.zip(38.5 KB) emitted;audit-identify.zipcarries the new5-server-sdk.md+ bundledanonymous-vs-identified-events.mddist/skills/skill-menu.jsonlistsaudit-attributionundercliEntrieswithrole: command, parentCommand: audit, command: attributionutm-segmentation.md,config.md,data-collection.md,identify-users.md,anonymous-vs-identified-events.md)audit-attributionself-seeds its ledger in step 1 (parity withevents-audit)/wizard-ci basic-integration— see PR comment; smoke test for manifest integritypnpm dev+ wizard local-mcp) — recommended before release-cut to confirm the agent reads the new step files end-to-end against a real codebase