Skip to content

feat(migrate): Clarify assumption-sheet wizard — confirm defaults, ask only essential questions#127

Merged
icarthick merged 8 commits into
awslabs:mainfrom
herosjourney:feat/clarify-assumption-sheet-wizard
Jul 9, 2026
Merged

feat(migrate): Clarify assumption-sheet wizard — confirm defaults, ask only essential questions#127
icarthick merged 8 commits into
awslabs:mainfrom
herosjourney:feat/clarify-assumption-sheet-wizard

Conversation

@herosjourney

@herosjourney herosjourney commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Summary

Make the Clarify phase default to a confirm-or-edit wizard instead of a question-by-question interview. Everything the plugin can extract from discovery or safely default is presented up front as one assumption sheet — each row with its one-line design consequence — and only the questions with no safe default are asked directly.

User-visible result: a typical Terraform-discovered stack goes from ~12–18 questions across 2–3 batches to 1 sheet + 2–7 essential questions, with zero loss of information the user actually needed to provide. The full interview remains one reply away ("ask me everything").

Problem

Evidence from a real GCP-to-AWS run (small dev-tier stack: Cloud Run + Cloud SQL + Gemini):

  • The plugin already extracted 8 answers (region, availability, DB size, DB traffic/IO, model, modalities, framework) and defaulted 9 more — yet the flow still walked the user through multi-question batches for values that were about to be defaulted anyway.
  • Defaults are invisible until the report: the user never sees "I assumed balanced AI priority → Sonnet-class model" at a moment where correcting it is cheap.
  • The existing Step 2.5 Confirm Detected Settings gate shows extracted values only. Defaulted values — the majority of the assumption surface — bypass user review entirely.
  • The Defaults Table in clarify.md is incomplete: no rows for Q3.5, Q14, Q19, Q23–Q27, or the Category B billing prompts, so "use defaults for the rest" is under-specified for exactly the questions agents are most likely to improvise on.

Solution

1. Step 2.5 becomes the Assumption Sheet (extracted + defaulted)

Two sections, one gate:

  • Detected — extracted values with source (terraform:availability_type=ZONAL, billing:, code:) — unchanged behavior, now merged into the sheet.
  • Assumed — documented defaults with a one-line "consequence if left as-is" (e.g., "Assuming 24/7 Cloud Run traffic → conservative (higher) AWS estimate").

User replies "looks good", corrects inline (availability: mission-critical), converts a row to a full question (ask me about ai priority), or opts out entirely (ask me everything).

2. Question Disposition Catalog (new Step 3)

Every question in an active category gets exactly one disposition — DETECTED / PROPOSED / ESSENTIAL / N/A — from a single table in the orchestrator. Essential (never assumed silently):

Question Why no safe default
Q2 compliance Gates security baseline, regions, service catalog
Q7 maintenance window Selects DMS vs pg_dump/pgcopydb and the runbook shape
Q1 region Only when multi-region inventory makes extraction ambiguous
Q3 GCP spend Only when no billing data
Q3.5 CUDs Only fires when billing proves commitments exist
Q15 AI spend Anchors Bedrock savings + credits tier; no discovery signal
Q23–Q25 agentic Routes the entire agentic design path
Conflict rows Multi-instance Cloud SQL disagreements

Category question files (clarify-global.md, clarify-compute.md, clarify-database.md, clarify-ai.md) are not modified — dispositions, defaults, and consequence lines are centralized in the orchestrator, keeping this PR to effectively one file. A maintenance note at the top of the catalog requires same-commit updates when category-file defaults change; co-locating consequence strings in the category files is a possible follow-up to prevent drift long-term.

3. Defaults Table completed (bug fix, stands on its own)

Added rows: Q3.5 (CUDs → none), Q14 (framework → auto-detect/["direct"]), Q19 (model → auto-detect), Q23–Q26 (agentic defaults per clarify-ai.md), Q27 (credits → unknown), and the four Category B billing prompts. Essential questions are annotated so "use defaults for the rest" is fully specified (Q2/Q3 default with a report caveat).

4. Provenance via source field on constraints

  • Every constraint with chosen_by: "extracted" carries a source field with the raw provenance signal (e.g. "terraform:availability_type=ZONAL", "ai-profile:integration.pattern=direct_sdk").
  • Every constraint with chosen_by: "default" carries source: "default:<Qid>" so report generation can flag unverified assumptions.
  • Omit source for "user" and "derived" constraints.
  • New clarify_mode: "wizard" (enum now wizard | full | fast_path | simple_hybrid); wizard is the default full flow.
  • New wizard_stage draft field for resume (sheet_pending / essentials_pending); legacy batches_completed drafts detected and handled.
  • Q9 WebSocket honesty: with no code scan, the sheet row is explicitly marked "unverified — no code scan" rather than presented as detected.
  • Generate phase Appendix F renders Source and Source signal columns, flagging default:-prefixed rows as assumptions.

5. Agent-compliance hardening (two-gate flow)

The wizard depends on the agent presenting the sheet, waiting, then asking essentials — never mixing them. Three layers enforce this:

  • Step 4 opens with a COMPLIANCE SELF-CHECK: the sheet must have been presented in a previous turn and answered before any question is emitted; sheet and questions must never share a message.
  • Handoff gate check 5: questions_asked may not overlap questions_skipped_extracted or questions_defaulted (the "ask me about X" conversion path moves the ID to questions_asked first, so legitimate conversions pass).
  • A dedicated test-plan case + proposed eval invariant mirror the gate check.

6. What is intentionally unchanged

  • Fast-path (3 questions) and simple hybrid (~6) offers — the wizard slots in as the default flow behind them; agentic hard-block preserved.
  • BigQuery specialist advisory, handoff gates (GATE_FAIL/HANDOFF_OK), phase-status protocol, Category E opt-in, answer-combination triggers, early-exit rules, preferences schema shape (value + chosen_by).
  • The full question-by-question flow — retained verbatim as the "ask me everything" variant with the same 3-batch structure and draft checkpoints.
  • Q23 remains essential except when the existing clarify-ai.md auto-skip rule fires (gateway/LangGraph/CrewAI users).
  • The multi-workload confirmation table stays in Step 4 (post-sheet); folding high-confidence workload rows into the sheet itself is a natural follow-up PR.

Question-count impact (replay of the real run above)

Flow Questions asked User turns
Current full flow 12 asked across 3 batches (9 more silently defaulted) 4–5
Wizard (this PR) Sheet + 3 essential (Q2, Q7, Q15) 2
Wizard, user corrects 2 rows Sheet + corrections + 3 essential 3

Files changed

File Change
skills/gcp-to-aws/references/phases/clarify/clarify.md Rewrite: Step 2.5 assumption sheet; Step 3 disposition catalog; Step 4 essentials + full-flow variant; completed Defaults Table; handoff gate checks 4–5; source field on constraints replacing detected_settings array
skills/gcp-to-aws/references/phases/generate/generate-artifacts-report.md Appendix F renders Source + Source signal columns from constraint source field
skills/gcp-to-aws/SKILL.md 2 copy hunks (error-handling "use defaults" row; scope-notes clarification bullet) + directory-tree comment for clarify-ai.md

Category files, fast-path preview fields, and all other phases: untouched. Downstream phases need no changes — the preferences.json shape is unchanged (source is additive).

Test plan

Draft PR — validation runs pending:

  • mise run eval:check — structural directives (FORBIDDEN blocks, BigQuery gate) still present in rewritten clarify.md
  • Fixture run: user-preferences — sheet shown before any question; preferences.json schema valid; extracted constraints have source field; clarify_mode: "wizard"
  • Fixture run: minimal-cloud-run-sql — end-to-end phases pass with wizard default
  • Fixture run: ai-workload-openai — Q15 asked; Q16–Q22 appear as sheet rows; agentic profile forces Q23–Q25 as essentials
  • Manual: "ask me everything" reproduces the legacy 3-batch flow with clarify_mode: "full"
  • Manual: "use defaults for the rest" completes with Q2/Q3 caveats recorded
  • Manual: resume from wizard_stage: "essentials_pending" draft; legacy batches_completed draft triggers the migration prompt
  • Two-gate compliance: transcript review across all three fixture runs — sheet is its own turn; no sheet-row question asked unless converted

Proposed new eval invariants (for the user-preferences fixture):

  1. metadata.clarify_mode ∈ {wizard, full, fast_path, simple_hybrid}
  2. If clarify_mode == "wizard": at least one constraint has a source field prefixed default:
  3. Every constraint with chosen_by: "extracted" or chosen_by: "default" has a source field
  4. questions_askedquestions_defaulted = ∅ and ∩ questions_skipped_extracted = ∅
  5. If clarify_mode == "wizard" and Cloud SQL in inventory: availability.chosen_by ∈ {user, extracted, default} (never absent)
  6. questions_asked ∩ (questions_skipped_extractedquestions_defaulted) = ∅ — artifact-level proxy for "sheet and questions were never mixed" (mirrors handoff gate check 5)

Logan Kleier and others added 4 commits July 7, 2026 23:31
…ssential questions

Clarify now defaults to a confirm-or-edit flow: extracted values and
documented defaults are presented as one assumption sheet (each row with
its design consequence), and only questions with no safe default are
asked directly (typically 2-7). The full question-by-question flow
remains available via "ask me everything".

Also completes the Defaults Table (Q3.5, Q14, Q19, Q23-Q27, Category B
prompts) and hardens two-gate compliance (Step 4 self-check, handoff
gate check 5). preferences.json shape is unchanged; category question
files and downstream phases are untouched.

Co-authored-by: Cursor <cursoragent@cursor.com>
…ted_settings The detected_settings array duplicated provenance info already available via chosen_by on constraint objects, without any downstream consumer. Move the useful piece — the raw source signal — to an optional 'source' field directly on each constraint object: - Extracted constraints: source like 'terraform:availability_type=ZONAL' - Default constraints: source like 'default:Q16' - User/derived: no source field This eliminates the parallel array while making provenance available end-to-end in the report's Appendix F configuration table. Changes: - clarify.md: remove detected_settings schema/array, add source to constraint examples, update override logic, validation checklist, schema rules, and handoff gate checks - generate-artifacts-report.md: Appendix F now renders Source and Source signal columns, flags default-prefixed rows as assumptions
…unt formula - generate-artifacts-report.md: Appendix F now renders warning callouts when compliance (Q2) or gcp_monthly_spend (Q3) were defaulted without explicit user confirmation — closes the Clarify→Generate gap - clarify.md Step 2.5: add Cat B example row (Cloud SQL HA from billing) to the Detected section template for better agent compliance - clarify.md Step 2.5: add formula for computing [N] essential questions so agents don't hallucinate the count in the sheet CTA
@herosjourney
herosjourney marked this pull request as ready for review July 8, 2026 22:13
@herosjourney
herosjourney requested a review from a team as a code owner July 8, 2026 22:13
Logan Kleier added 4 commits July 8, 2026 15:35
…x for ordered list continuation (items 8–16) - MD032: add blank lines before lists after inline text
…rkdownlint v0.17 detects 1/2/3 style from the list prefix and requires continuation. Restore 8–16 for Step 2 items and 8–14 for schema rules.
…s markdownlint-cli2 v0.17 detects Style: 1/1/1 when list items are separated by tables/code blocks (each item starts a new list context). Use '1.' for Step 2 items 8–16; keep sequential for the contiguous Schema Rules list (1–14).
…rint check failures (table column alignment, JSON indentation in code blocks, list item indentation). All 3 PR files now pass both markdownlint-cli2 and dprint check.
@icarthick
icarthick merged commit a7b0d2f into awslabs:main Jul 9, 2026
8 checks passed
herosjourney pushed a commit to herosjourney/startups that referenced this pull request Jul 14, 2026
…files that conflict against main (the Clarify assumption-sheet wizard, PR awslabs#127) are resolved here in the same way, ahead of the actual merge: - README.md: keep main's two-table capability breakdown and updated workflow wording; add this PR's migration-report row and validator subsection; drop the stale 'Evaluating Changes' section (its tools/eval_check.py and tests/fixtures/ no longer exist on main). - clarify.md / schema-preferences.md: unify the two independently extended preferences.json schemas. Every constraint keeps this PR's 'prompt' + 'design_consequence' fields (consumed by Generate's Appendix F) and gains main's 'source' provenance field (required when chosen_by is extracted/default). Schema Rules list and the worked JSON example updated to reflect both; JSON re-validated. - generate-artifacts-report.md: Appendix F renders a 5-column table (question/choice/source/source signal/consequence) combining this PR's dynamic per-constraint iteration with main's source-signal column, default-assumption styling, and critical-default warning callouts for Q2/Q3. Verified: JSON example still parses with all required fields present, dprint check and markdownlint-cli2 clean, existing validator pytest suite (32/32) still passes.
icarthick pushed a commit that referenced this pull request Jul 14, 2026
…on (#78)

* Enforce comprehensive migration reports with post-write validation.

Expand generate-artifacts-report.md with combined TCO, security baseline
component costs, gap analysis, and assumptions sections; add an HTML validator
script, reference fixture, and tests so stub appendices fail validation instead
of shipping as complete reports.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address Kiro review: TOC integrity, strict section checks, safer defaults.

Fix broken TOC anchors in the reference fixture; validate nav links match
section ids; require exactly one <section id> per required block; scope
GuardDuty checks to cost/security appendices; gate exec-tco on both estimate
JSON files; default to rename-on-fail; expand test coverage to 11 cases.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Close Kiro gaps: reference+JSON contract, dollar security check, REPORT_OK scope.

Validate reference HTML with estimation-infra/ai fixture JSON; replace dead
GuardDuty regex with component dollar matching; document that REPORT_OK means
structure complete not numerically audited; expand tests to 14 cases.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Document migration-report.html and validator in plugin README.

Add Generate-phase report overview, REPORT_OK scope, pytest/validator
commands, and fix stale skill/docs paths in Development section.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Remove machine-specific stub test; tighten dollar matching.

Replace the hardcoded SF Beach path with STUB_FAIL pytest coverage, add a
committed stub fixture for manual CLI validation, and use word-boundary
regex in _dollar_amount_present so $13 does not match $130.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Remove SF Beach provenance labels from fixture docs.

Drop project-specific structural-reference wording from the PR-facing
README and reference HTML footer.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Enforce migration report readability in validator and reference fixture.

Add readability gates (no Rubric:/Section N headings), rewrite the reference
report with security teaser + appendix split, and document conventions in the
spec so generated reports stay exec-friendly.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Add security teaser, verdict, and fixture-bleed validator gates.

Require exec-security-teaser when security_baseline exists, a verdict banner
when recommendation exists, and --migration-dir canary checks on real runs;
refresh the stub regression fixture and wire --migration-dir into Generate Step 4.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Clarify BigQuery requires specialist path, not migration block.

Reframe Stay if, risk severity, deferred callout, and estimate/report spec
so phased infra+AI migration proceeds while analytics target is evaluated
in parallel with AWS account team or data partner.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix CloudWatch free-tier copy and apply allowance-aware cost formula.

Correct observability messaging (5 GB logs / 10 metrics / 10 alarms always-free vs GCP's larger tier), subtract allowances in Step 4, and recascade SF Beach reference totals ($118→$112 infra, ~$503 combined savings).

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs(migrate): label cost figures as estimated monthly costs

Ensure user-facing cost output consistently uses 'estimated monthly
costs' phrasing across the estimate phase summaries, report template,
README, and report fixture. Adds an explicit cost-labeling rule to the
estimate orchestrator and each sub-estimate Present Summary.

* feat(migrate): enforce reader-facing vocabulary in migration report exec flow

Apply Anthropic frontend-design skill principles to the migration report:

- "Name things by what people control, not how the system is built":
  add a post-write gate so executive-flow sections (decision-summary,
  exec-*) cannot expose artifact filenames (*.json) or Terraform resource
  IDs (aws_<resource>.<name>). Those identifiers stay in the technical
  appendices, which remain exempt. Gated under --no-readability.
- "Structure is information": clarify the numbered-heading ban targets
  decorative labels only — genuine sequences (cluster order, phased weeks,
  migration phases, rollback steps) keep their numbering.
- "One name per concept": require a single consistent label for the
  recommended model and cost tier across verdict, tables, and appendices.

Updates the reference fixture to reader-facing copy in exec-tco and
exec-security-teaser, documents check #14 + the sequence nuance in
validate-migration-report.md and generate-artifacts-report.md, and adds
4 tests (28 pass).

* feat(migrate): add config provenance schema and ordered action lists in reports

Extend preferences.json with prompt and design_consequence fields so appendix-config
shows question-to-plan traceability, enforce ordered Next steps and Key decisions
ahead lists, and gate both in the report validator with reference fixtures.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(migrate): address PR review comments (validator exit codes, unused fixture) - Remove unused fixtures/preferences-reference.json: no test or script loads it programmatically, only referenced from fixtures README docs. Drop the dangling reference in the README too. - validate-migration-report.md / generate.md: require the agent to branch on the validator's shell exit code, not just pattern-match REPORT_OK/REPORT_FAIL text. If python3 is missing (exit 127) or the script path is wrong, neither string appears in stdout, so text-only parsing leaves that case undefined. Now: 0=pass, 1=fail-with-errors, anything else=validator did not run (tell the user, do not treat as pass).

* resolve merge conflicts vs main: unify preferences.json schema Three files that conflict against main (the Clarify assumption-sheet wizard, PR #127) are resolved here in the same way, ahead of the actual merge: - README.md: keep main's two-table capability breakdown and updated workflow wording; add this PR's migration-report row and validator subsection; drop the stale 'Evaluating Changes' section (its tools/eval_check.py and tests/fixtures/ no longer exist on main). - clarify.md / schema-preferences.md: unify the two independently extended preferences.json schemas. Every constraint keeps this PR's 'prompt' + 'design_consequence' fields (consumed by Generate's Appendix F) and gains main's 'source' provenance field (required when chosen_by is extracted/default). Schema Rules list and the worked JSON example updated to reflect both; JSON re-validated. - generate-artifacts-report.md: Appendix F renders a 5-column table (question/choice/source/source signal/consequence) combining this PR's dynamic per-constraint iteration with main's source-signal column, default-assumption styling, and critical-default warning callouts for Q2/Q3. Verified: JSON example still parses with all required fields present, dprint check and markdownlint-cli2 clean, existing validator pytest suite (32/32) still passes.

* style: fix dprint table alignment after merge Table cell padding drifted out of alignment when the merge combined this PR's content with main's edits to these three tables. Run dprint fmt to restore consistent column widths — no content changes.

* fix(ci): add migration-to-aws bandit pass, silence 2 test-only false positives The security-scanners.yml bandit job runs two passes: ai-to-aws with its own skip config, and 'everything else' with full strictness. This PR added Python under migrate/plugins/migration-to-aws/ (the report validator + its pytest suite), which fell into the strict pass and failed CI with 68 findings — all in tests/test_validate_migration_report.py: - 65x B101 (assert) — pytest's normal assertion mechanism, not production guard logic. - 1x B105 (hardcoded_password_string) — false positive on the fixture name MINIMAL_PASS (starts with 'pass'). - 1x B404 + 1x B603 (subprocess import/call without shell=True) — this is actually the *secure* invocation form (list args, no shell) used to run the validator script itself; bandit flags it for manual review, not because it's risky. Mirrors the existing ai-to-aws precedent instead of loosening the strict pass for everyone: - New migrate/plugins/migration-to-aws/bandit.yml skips B101/B105 for this plugin only (matches ai-to-aws/bandit.yml's rationale). - B404/B603 are NOT skipped at the plugin level (future subprocess use here should still be reviewed) — the one existing call site gets an inline  /  instead. - security-scanners.yml bandit job: added a third scoped pass for migration-to-aws (same pattern as ai-to-aws), excluded from the now-only-two-plugins-excluded strict 'everything else' pass, SARIF outputs merged three-way instead of two-way. - mise.toml security:bandit task updated to match (local repro of CI). Verified locally with bandit 1.8.6 (CI pins 1.9.3): all three passes exit 0, no findings. pytest suite still 32/32. dprint check clean.

* fix(ci): simplify bandit fix — exclude tests/ dir instead of new plugin config Per review: the previous commit's plugin-wide bandit.yml + third CI pass was more machinery than the problem needed. All 68 findings were confined to one file (tests/test_validate_migration_report.py); the production script (scripts/validate-migration-report.py) had zero findings and doesn't need any exemption. Revert to the two-pass structure and instead add migrate/plugins/migration-to-aws/tests to the existing 'everything else' pass's --exclude list, matching the granularity of the actual problem: - Removed migrate/plugins/migration-to-aws/bandit.yml (no plugin-wide skip needed — only one directory has the pytest-assert pattern). - security-scanners.yml: back to two bandit passes; pass 2 gains one more excluded path (the tests dir) alongside ai-to-aws. Comment explains why the exclusion is scoped to tests/ and not the whole plugin. - mise.toml security:bandit task updated to match. - Removed the inline # nosec B404/B603 markers from test_validate_migration_report.py — they're moot now that the whole file is out of scan scope, and leaving them would be misleading about why the file passes. Verified locally: bandit -r . --exclude '...,./migrate/plugins/migration-to-aws/tests' scans scripts/validate-migration-report.py (450 lines) with 0 findings, exit 0. Without the exclude, the same command flags the test file (68 findings) confirming the exclude is what's doing the work. pytest 32/32, dprint check clean.

---------

Co-authored-by: Logan Kleier <lkleier@amazon.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
azpaulp added a commit to azpaulp/startups that referenced this pull request Jul 14, 2026
Resolves conflicts in clarify.md from PR awslabs#127 (assumption-sheet wizard),
which restructured the clarify phase. Re-applied the Q7b (App Engine
compute model) additions onto main's new structure:
- Category C firing rule + reference-file range (Q7b–Q11)
- Q7b row in the Step 3 Disposition Catalog (PROPOSED/N-A)
- Q7b early-exit rule (N/A when no App Engine)
- compute_model constraint in the example preferences.json
- Q7b rows in the Answer Combination Triggers and Defaults tables

SKILL.md auto-merged (EB/App Engine additions preserved). All other
App Engine → EB work from this branch is unaffected by the merge.

Verified: fmt:check, lint:md, lint:types, test (57/57),
lint:frontmatter, gitleaks all pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants