feat(migrate): Clarify assumption-sheet wizard — confirm defaults, ask only essential questions#127
Merged
icarthick merged 8 commits intoJul 9, 2026
Conversation
…ssential questions Clarify now defaults to a confirm-or-edit flow: extracted values and documented defaults are presented as one assumption sheet (each row with its design consequence), and only questions with no safe default are asked directly (typically 2-7). The full question-by-question flow remains available via "ask me everything". Also completes the Defaults Table (Q3.5, Q14, Q19, Q23-Q27, Category B prompts) and hardens two-gate compliance (Step 4 self-check, handoff gate check 5). preferences.json shape is unchanged; category question files and downstream phases are untouched. Co-authored-by: Cursor <cursoragent@cursor.com>
…ted_settings The detected_settings array duplicated provenance info already available via chosen_by on constraint objects, without any downstream consumer. Move the useful piece — the raw source signal — to an optional 'source' field directly on each constraint object: - Extracted constraints: source like 'terraform:availability_type=ZONAL' - Default constraints: source like 'default:Q16' - User/derived: no source field This eliminates the parallel array while making provenance available end-to-end in the report's Appendix F configuration table. Changes: - clarify.md: remove detected_settings schema/array, add source to constraint examples, update override logic, validation checklist, schema rules, and handoff gate checks - generate-artifacts-report.md: Appendix F now renders Source and Source signal columns, flags default-prefixed rows as assumptions
…unt formula - generate-artifacts-report.md: Appendix F now renders warning callouts when compliance (Q2) or gcp_monthly_spend (Q3) were defaulted without explicit user confirmation — closes the Clarify→Generate gap - clarify.md Step 2.5: add Cat B example row (Cloud SQL HA from billing) to the Detected section template for better agent compliance - clarify.md Step 2.5: add formula for computing [N] essential questions so agents don't hallucinate the count in the sheet CTA
herosjourney
marked this pull request as ready for review
July 8, 2026 22:13
added 4 commits
July 8, 2026 15:35
…x for ordered list continuation (items 8–16) - MD032: add blank lines before lists after inline text
…rkdownlint v0.17 detects 1/2/3 style from the list prefix and requires continuation. Restore 8–16 for Step 2 items and 8–14 for schema rules.
…s markdownlint-cli2 v0.17 detects Style: 1/1/1 when list items are separated by tables/code blocks (each item starts a new list context). Use '1.' for Step 2 items 8–16; keep sequential for the contiguous Schema Rules list (1–14).
…rint check failures (table column alignment, JSON indentation in code blocks, list item indentation). All 3 PR files now pass both markdownlint-cli2 and dprint check.
herosjourney
pushed a commit
to herosjourney/startups
that referenced
this pull request
Jul 14, 2026
…files that conflict against main (the Clarify assumption-sheet wizard, PR awslabs#127) are resolved here in the same way, ahead of the actual merge: - README.md: keep main's two-table capability breakdown and updated workflow wording; add this PR's migration-report row and validator subsection; drop the stale 'Evaluating Changes' section (its tools/eval_check.py and tests/fixtures/ no longer exist on main). - clarify.md / schema-preferences.md: unify the two independently extended preferences.json schemas. Every constraint keeps this PR's 'prompt' + 'design_consequence' fields (consumed by Generate's Appendix F) and gains main's 'source' provenance field (required when chosen_by is extracted/default). Schema Rules list and the worked JSON example updated to reflect both; JSON re-validated. - generate-artifacts-report.md: Appendix F renders a 5-column table (question/choice/source/source signal/consequence) combining this PR's dynamic per-constraint iteration with main's source-signal column, default-assumption styling, and critical-default warning callouts for Q2/Q3. Verified: JSON example still parses with all required fields present, dprint check and markdownlint-cli2 clean, existing validator pytest suite (32/32) still passes.
icarthick
pushed a commit
that referenced
this pull request
Jul 14, 2026
…on (#78) * Enforce comprehensive migration reports with post-write validation. Expand generate-artifacts-report.md with combined TCO, security baseline component costs, gap analysis, and assumptions sections; add an HTML validator script, reference fixture, and tests so stub appendices fail validation instead of shipping as complete reports. Co-authored-by: Cursor <cursoragent@cursor.com> * Address Kiro review: TOC integrity, strict section checks, safer defaults. Fix broken TOC anchors in the reference fixture; validate nav links match section ids; require exactly one <section id> per required block; scope GuardDuty checks to cost/security appendices; gate exec-tco on both estimate JSON files; default to rename-on-fail; expand test coverage to 11 cases. Co-authored-by: Cursor <cursoragent@cursor.com> * Close Kiro gaps: reference+JSON contract, dollar security check, REPORT_OK scope. Validate reference HTML with estimation-infra/ai fixture JSON; replace dead GuardDuty regex with component dollar matching; document that REPORT_OK means structure complete not numerically audited; expand tests to 14 cases. Co-authored-by: Cursor <cursoragent@cursor.com> * Document migration-report.html and validator in plugin README. Add Generate-phase report overview, REPORT_OK scope, pytest/validator commands, and fix stale skill/docs paths in Development section. Co-authored-by: Cursor <cursoragent@cursor.com> * Remove machine-specific stub test; tighten dollar matching. Replace the hardcoded SF Beach path with STUB_FAIL pytest coverage, add a committed stub fixture for manual CLI validation, and use word-boundary regex in _dollar_amount_present so $13 does not match $130. Co-authored-by: Cursor <cursoragent@cursor.com> * Remove SF Beach provenance labels from fixture docs. Drop project-specific structural-reference wording from the PR-facing README and reference HTML footer. Co-authored-by: Cursor <cursoragent@cursor.com> * Enforce migration report readability in validator and reference fixture. Add readability gates (no Rubric:/Section N headings), rewrite the reference report with security teaser + appendix split, and document conventions in the spec so generated reports stay exec-friendly. Co-authored-by: Cursor <cursoragent@cursor.com> * Add security teaser, verdict, and fixture-bleed validator gates. Require exec-security-teaser when security_baseline exists, a verdict banner when recommendation exists, and --migration-dir canary checks on real runs; refresh the stub regression fixture and wire --migration-dir into Generate Step 4. Co-authored-by: Cursor <cursoragent@cursor.com> * Clarify BigQuery requires specialist path, not migration block. Reframe Stay if, risk severity, deferred callout, and estimate/report spec so phased infra+AI migration proceeds while analytics target is evaluated in parallel with AWS account team or data partner. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix CloudWatch free-tier copy and apply allowance-aware cost formula. Correct observability messaging (5 GB logs / 10 metrics / 10 alarms always-free vs GCP's larger tier), subtract allowances in Step 4, and recascade SF Beach reference totals ($118→$112 infra, ~$503 combined savings). Co-authored-by: Cursor <cursoragent@cursor.com> * docs(migrate): label cost figures as estimated monthly costs Ensure user-facing cost output consistently uses 'estimated monthly costs' phrasing across the estimate phase summaries, report template, README, and report fixture. Adds an explicit cost-labeling rule to the estimate orchestrator and each sub-estimate Present Summary. * feat(migrate): enforce reader-facing vocabulary in migration report exec flow Apply Anthropic frontend-design skill principles to the migration report: - "Name things by what people control, not how the system is built": add a post-write gate so executive-flow sections (decision-summary, exec-*) cannot expose artifact filenames (*.json) or Terraform resource IDs (aws_<resource>.<name>). Those identifiers stay in the technical appendices, which remain exempt. Gated under --no-readability. - "Structure is information": clarify the numbered-heading ban targets decorative labels only — genuine sequences (cluster order, phased weeks, migration phases, rollback steps) keep their numbering. - "One name per concept": require a single consistent label for the recommended model and cost tier across verdict, tables, and appendices. Updates the reference fixture to reader-facing copy in exec-tco and exec-security-teaser, documents check #14 + the sequence nuance in validate-migration-report.md and generate-artifacts-report.md, and adds 4 tests (28 pass). * feat(migrate): add config provenance schema and ordered action lists in reports Extend preferences.json with prompt and design_consequence fields so appendix-config shows question-to-plan traceability, enforce ordered Next steps and Key decisions ahead lists, and gate both in the report validator with reference fixtures. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(migrate): address PR review comments (validator exit codes, unused fixture) - Remove unused fixtures/preferences-reference.json: no test or script loads it programmatically, only referenced from fixtures README docs. Drop the dangling reference in the README too. - validate-migration-report.md / generate.md: require the agent to branch on the validator's shell exit code, not just pattern-match REPORT_OK/REPORT_FAIL text. If python3 is missing (exit 127) or the script path is wrong, neither string appears in stdout, so text-only parsing leaves that case undefined. Now: 0=pass, 1=fail-with-errors, anything else=validator did not run (tell the user, do not treat as pass). * resolve merge conflicts vs main: unify preferences.json schema Three files that conflict against main (the Clarify assumption-sheet wizard, PR #127) are resolved here in the same way, ahead of the actual merge: - README.md: keep main's two-table capability breakdown and updated workflow wording; add this PR's migration-report row and validator subsection; drop the stale 'Evaluating Changes' section (its tools/eval_check.py and tests/fixtures/ no longer exist on main). - clarify.md / schema-preferences.md: unify the two independently extended preferences.json schemas. Every constraint keeps this PR's 'prompt' + 'design_consequence' fields (consumed by Generate's Appendix F) and gains main's 'source' provenance field (required when chosen_by is extracted/default). Schema Rules list and the worked JSON example updated to reflect both; JSON re-validated. - generate-artifacts-report.md: Appendix F renders a 5-column table (question/choice/source/source signal/consequence) combining this PR's dynamic per-constraint iteration with main's source-signal column, default-assumption styling, and critical-default warning callouts for Q2/Q3. Verified: JSON example still parses with all required fields present, dprint check and markdownlint-cli2 clean, existing validator pytest suite (32/32) still passes. * style: fix dprint table alignment after merge Table cell padding drifted out of alignment when the merge combined this PR's content with main's edits to these three tables. Run dprint fmt to restore consistent column widths — no content changes. * fix(ci): add migration-to-aws bandit pass, silence 2 test-only false positives The security-scanners.yml bandit job runs two passes: ai-to-aws with its own skip config, and 'everything else' with full strictness. This PR added Python under migrate/plugins/migration-to-aws/ (the report validator + its pytest suite), which fell into the strict pass and failed CI with 68 findings — all in tests/test_validate_migration_report.py: - 65x B101 (assert) — pytest's normal assertion mechanism, not production guard logic. - 1x B105 (hardcoded_password_string) — false positive on the fixture name MINIMAL_PASS (starts with 'pass'). - 1x B404 + 1x B603 (subprocess import/call without shell=True) — this is actually the *secure* invocation form (list args, no shell) used to run the validator script itself; bandit flags it for manual review, not because it's risky. Mirrors the existing ai-to-aws precedent instead of loosening the strict pass for everyone: - New migrate/plugins/migration-to-aws/bandit.yml skips B101/B105 for this plugin only (matches ai-to-aws/bandit.yml's rationale). - B404/B603 are NOT skipped at the plugin level (future subprocess use here should still be reviewed) — the one existing call site gets an inline / instead. - security-scanners.yml bandit job: added a third scoped pass for migration-to-aws (same pattern as ai-to-aws), excluded from the now-only-two-plugins-excluded strict 'everything else' pass, SARIF outputs merged three-way instead of two-way. - mise.toml security:bandit task updated to match (local repro of CI). Verified locally with bandit 1.8.6 (CI pins 1.9.3): all three passes exit 0, no findings. pytest suite still 32/32. dprint check clean. * fix(ci): simplify bandit fix — exclude tests/ dir instead of new plugin config Per review: the previous commit's plugin-wide bandit.yml + third CI pass was more machinery than the problem needed. All 68 findings were confined to one file (tests/test_validate_migration_report.py); the production script (scripts/validate-migration-report.py) had zero findings and doesn't need any exemption. Revert to the two-pass structure and instead add migrate/plugins/migration-to-aws/tests to the existing 'everything else' pass's --exclude list, matching the granularity of the actual problem: - Removed migrate/plugins/migration-to-aws/bandit.yml (no plugin-wide skip needed — only one directory has the pytest-assert pattern). - security-scanners.yml: back to two bandit passes; pass 2 gains one more excluded path (the tests dir) alongside ai-to-aws. Comment explains why the exclusion is scoped to tests/ and not the whole plugin. - mise.toml security:bandit task updated to match. - Removed the inline # nosec B404/B603 markers from test_validate_migration_report.py — they're moot now that the whole file is out of scan scope, and leaving them would be misleading about why the file passes. Verified locally: bandit -r . --exclude '...,./migrate/plugins/migration-to-aws/tests' scans scripts/validate-migration-report.py (450 lines) with 0 findings, exit 0. Without the exclude, the same command flags the test file (68 findings) confirming the exclude is what's doing the work. pytest 32/32, dprint check clean. --------- Co-authored-by: Logan Kleier <lkleier@amazon.com> Co-authored-by: Cursor <cursoragent@cursor.com>
azpaulp
added a commit
to azpaulp/startups
that referenced
this pull request
Jul 14, 2026
Resolves conflicts in clarify.md from PR awslabs#127 (assumption-sheet wizard), which restructured the clarify phase. Re-applied the Q7b (App Engine compute model) additions onto main's new structure: - Category C firing rule + reference-file range (Q7b–Q11) - Q7b row in the Step 3 Disposition Catalog (PROPOSED/N-A) - Q7b early-exit rule (N/A when no App Engine) - compute_model constraint in the example preferences.json - Q7b rows in the Answer Combination Triggers and Defaults tables SKILL.md auto-merged (EB/App Engine additions preserved). All other App Engine → EB work from this branch is unaffected by the merge. Verified: fmt:check, lint:md, lint:types, test (57/57), lint:frontmatter, gitleaks all pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Make the Clarify phase default to a confirm-or-edit wizard instead of a question-by-question interview. Everything the plugin can extract from discovery or safely default is presented up front as one assumption sheet — each row with its one-line design consequence — and only the questions with no safe default are asked directly.
User-visible result: a typical Terraform-discovered stack goes from ~12–18 questions across 2–3 batches to 1 sheet + 2–7 essential questions, with zero loss of information the user actually needed to provide. The full interview remains one reply away ("ask me everything").
Problem
Evidence from a real GCP-to-AWS run (small dev-tier stack: Cloud Run + Cloud SQL + Gemini):
clarify.mdis incomplete: no rows for Q3.5, Q14, Q19, Q23–Q27, or the Category B billing prompts, so "use defaults for the rest" is under-specified for exactly the questions agents are most likely to improvise on.Solution
1. Step 2.5 becomes the Assumption Sheet (extracted + defaulted)
Two sections, one gate:
terraform:availability_type=ZONAL,billing:,code:) — unchanged behavior, now merged into the sheet.User replies "looks good", corrects inline (
availability: mission-critical), converts a row to a full question (ask me about ai priority), or opts out entirely (ask me everything).2. Question Disposition Catalog (new Step 3)
Every question in an active category gets exactly one disposition — DETECTED / PROPOSED / ESSENTIAL / N/A — from a single table in the orchestrator. Essential (never assumed silently):
Category question files (
clarify-global.md,clarify-compute.md,clarify-database.md,clarify-ai.md) are not modified — dispositions, defaults, and consequence lines are centralized in the orchestrator, keeping this PR to effectively one file. A maintenance note at the top of the catalog requires same-commit updates when category-file defaults change; co-locating consequence strings in the category files is a possible follow-up to prevent drift long-term.3. Defaults Table completed (bug fix, stands on its own)
Added rows: Q3.5 (CUDs → none), Q14 (framework → auto-detect/
["direct"]), Q19 (model → auto-detect), Q23–Q26 (agentic defaults perclarify-ai.md), Q27 (credits → unknown), and the four Category B billing prompts. Essential questions are annotated so "use defaults for the rest" is fully specified (Q2/Q3 default with a report caveat).4. Provenance via
sourcefield on constraintschosen_by: "extracted"carries asourcefield with the raw provenance signal (e.g."terraform:availability_type=ZONAL","ai-profile:integration.pattern=direct_sdk").chosen_by: "default"carriessource: "default:<Qid>"so report generation can flag unverified assumptions.sourcefor"user"and"derived"constraints.clarify_mode: "wizard"(enum nowwizard | full | fast_path | simple_hybrid); wizard is the default full flow.wizard_stagedraft field for resume (sheet_pending/essentials_pending); legacybatches_completeddrafts detected and handled.default:-prefixed rows as assumptions.5. Agent-compliance hardening (two-gate flow)
The wizard depends on the agent presenting the sheet, waiting, then asking essentials — never mixing them. Three layers enforce this:
questions_askedmay not overlapquestions_skipped_extractedorquestions_defaulted(the "ask me about X" conversion path moves the ID toquestions_askedfirst, so legitimate conversions pass).6. What is intentionally unchanged
GATE_FAIL/HANDOFF_OK), phase-status protocol, Category E opt-in, answer-combination triggers, early-exit rules, preferences schema shape (value+chosen_by).clarify-ai.mdauto-skip rule fires (gateway/LangGraph/CrewAI users).Question-count impact (replay of the real run above)
Files changed
skills/gcp-to-aws/references/phases/clarify/clarify.mdsourcefield on constraints replacingdetected_settingsarrayskills/gcp-to-aws/references/phases/generate/generate-artifacts-report.mdsourcefieldskills/gcp-to-aws/SKILL.mdclarify-ai.mdCategory files, fast-path preview fields, and all other phases: untouched. Downstream phases need no changes — the
preferences.jsonshape is unchanged (sourceis additive).Test plan
Draft PR — validation runs pending:
mise run eval:check— structural directives (FORBIDDEN blocks, BigQuery gate) still present in rewrittenclarify.mduser-preferences— sheet shown before any question;preferences.jsonschema valid; extracted constraints havesourcefield;clarify_mode: "wizard"minimal-cloud-run-sql— end-to-end phases pass with wizard defaultai-workload-openai— Q15 asked; Q16–Q22 appear as sheet rows; agentic profile forces Q23–Q25 as essentialsclarify_mode: "full"wizard_stage: "essentials_pending"draft; legacybatches_completeddraft triggers the migration promptProposed new eval invariants (for the
user-preferencesfixture):metadata.clarify_mode∈ {wizard, full, fast_path, simple_hybrid}clarify_mode == "wizard": at least one constraint has asourcefield prefixeddefault:chosen_by: "extracted"orchosen_by: "default"has asourcefieldquestions_asked∩questions_defaulted= ∅ and ∩questions_skipped_extracted= ∅clarify_mode == "wizard"and Cloud SQL in inventory:availability.chosen_by∈ {user, extracted, default} (never absent)questions_asked∩ (questions_skipped_extracted∪questions_defaulted) = ∅ — artifact-level proxy for "sheet and questions were never mixed" (mirrors handoff gate check 5)