Align benchmark prompts with PolicyEngine target-year input values

[MaxGhenis]

PolicyBench error audit found prompt/reference mismatches that can make model errors look larger or qualitatively different than they are.\n\nIssues to fix before the next paid run:\n\n- UK prompts currently show raw transfer-dataset input amounts, while the 2026-27 PolicyEngine reference simulation uprates inputs before calculating outputs. In the 100-household run, all 69 UK scenarios with positive employment income differed between prompt and PE calculation, with median PE/prompt ratio about 1.034. Capital gains, savings interest, dividends, and private pensions were also uprated before reference calculation.\n- UK pension contribution wording is ambiguous. Models interpreted employee pension contributions as reducing taxable pay, while the PE reference did not always reduce taxable income that way. Prompts should either expose target-year tax variables in the same semantics PE uses or specify the pension contribution treatment precisely.\n- ACA PTC prompts are not clean no-web household calculations when local SLCSP/benchmark premium and Marketplace enrollment assumptions are implicit. Either supply the benchmark/selected plan facts, label PTC as a no-web local-market stress test, or exclude/stratify those cases.\n- SSDI/disability plus Medicare inference is a prompt-contract risk: some models inferred Medicare from SSDI even though unlisted statuses are false and PE had Medicare eligibility false. Clarify or avoid cases where waiting-period/enrollment facts are not supplied.\n\nSuggested acceptance criteria:\n\n- Scenario prompt values are generated from the same target-year PolicyEngine values used to produce reference outputs, or the prompt explicitly says the displayed values are base-year values and gives the uprating rule. Prefer target-year values.\n- Add a test that UK prompt-visible income/gain/pension inputs reconcile to PE target-year input variables for sampled scenarios.\n- Add prompt-contract tests or snapshot examples for PTC and SSDI/Medicare edge cases.\n\nContext: this came from the post-run error audit for results/local/full_run_20260513_policyengine_4_4_4_nested_outputs.

[MaxGhenis]

Follow-up from the broader deviation audit while the refreshed US data build is running.

Additional prompt/scenario-contract risks to fix before the next paid run:

• US Medicaid person-level outputs: several adult-dependent cases are all- or near-all-model false negatives, e.g. dependent1_medicaid_eligible in NC/WA. The models consistently used full household/tax-unit income, while PolicyEngine applies person-specific Medicaid household/non-filer rules for a 19-year-old tax dependent not required to file. The current preface says everyone is in one household group for tax and benefit calculations, which may push models toward the wrong unit. Fix by explicitly saying person-level eligibility outputs should use the program's own person-specific unit/household rules, not necessarily total household income, or expose the relevant Medicaid household size/income facts.
• ACA PTC: all 6 positive PTC reference cases had most/all models return zero. Explanations repeatedly say no Marketplace enrollment/plan was listed, or one person's ESI disqualifies the household. The benchmark currently assumes tax filing/program take-up and says to estimate a benchmark plan if selected plan info is absent, but that is not explicit enough for PTC. Fix by either supplying selected Marketplace plan/benchmark-premium facts, or stating directly that eligible members should be treated as taking up Marketplace coverage for PTC unless person-specific ESI/Medicare/Medicaid facts make them ineligible.
• UK PIP: the current 100-household UK snapshot has zero positive PIP reference cases (pip total is 0). Most PIP deviations are missing parsed outputs or models inferring a PIP award from generic disability facts. Drop PIP from the headline run unless the sample includes modeled positive awards with prompt-visible award/eligibility facts.
• US local income tax: the current 100-household US snapshot has zero positive local_income_tax cases. Either stratify the sample to include local-income-tax jurisdictions, or drop it from the headline run and keep it as a targeted diagnostic.
• Output omissions are concentrated by model/provider rather than policy: grok-4.1-fast has 787 missing US rows and 588 missing UK rows; Gemini Flash-Lite also has 273 UK missing rows. That is a response-compliance/provider reliability issue, not a policy-understanding finding. For headline runs, consider excluding high-omission models, using smaller chunks for them, or reporting a separate parse/compliance metric prominently.

Things that did not look like prompt/reference defects in this pass:

• US payroll tax errors mostly reflect FICA cap/additional Medicare/state paid-leave contribution mistakes or mixing in self-employment tax despite the prompt excluding it.
• US SNAP errors mostly reflect simplified gross-income/FPL tests, deduction mistakes, and asset/vehicle assumptions; the known PE-US SNAP reference bug is being handled through the data refresh.
• UK Child Benefit, NI, Income Tax, CGT, Universal Credit, and Pension Credit errors mostly look like stale rates, annualization mistakes, capital/tariff-income mistakes, or simplified means-test calculations. The separate target-year prompt/reference alignment issue above still applies.

[MaxGhenis]

Correction on the PIP/local-tax bullets above: do not drop outputs solely because they are all-zero in the current 100-household sample. A zero-positive count at n=100 is a sample-coverage limitation, not by itself a benchmark-scope defect. Better acceptance criterion: report positive-case counts by output in the run metadata/dashboard, and consider a larger or stratified sample later if we need more coverage for sparse programs or local taxes.

[MaxGhenis]

Small A/B follow-up on the softened wording, using two models (gpt-5.4-mini, claude-haiku-4.5) and paired old/new prompts. This is intentionally small and stochastic; it is not a replacement for a full ablation.

Sample:

• Positive PTC references: scenario_025, scenario_066, scenario_089.
• Zero PTC controls: scenario_005, scenario_055, scenario_071.
• Adult-dependent Medicaid positives: scenario_002, scenario_010, scenario_033.

Result summary:

• Old PTC prompts produced almost all zero PTC values. Across positive PTC cases, old prompt nonzero predictions were 1/6; new prompt nonzero predictions were 3/6.
• The new wording did not give away the answer: PTC scores remained low because benchmark premiums, FPL thresholds, Medicaid interaction, and ESI/family-affordability logic were still hard.
• The new wording introduced a risk on controls for gpt-5.4-mini: it false-positived 2/3 zero-PTC controls. claude-haiku-4.5 did not false-positive those controls.
• Adult-dependent Medicaid results were mixed: claude-haiku-4.5 improved from 0/3 to 1/3; gpt-5.4-mini worsened from 1/3 to 0/3 in this small stochastic rerun.

Interpretation:

• The person-specific-unit clarification is fair and still worth keeping.
• The PTC clarification is not an answer leak, but it changes behavior: it reduces the old prompt's tendency to treat missing Marketplace enrollment as automatic zero, while exposing the model's weaker ACA eligibility interactions on zero controls.
• For a serious next run, report PTC separately or include a prompt-mode sensitivity table for PTC. If PTC remains headline, add zero-control monitoring for false positives.

Local diagnostic output: results/local/ptc_medicaid_prompt_ab_results.csv (ignored/local only).