[repository-quality] 🎯 Repository Quality Improvement Report - Codemod Registry Drift & Idempotency Coverage Gap

[github-actions[bot]]

Analysis Date: 2026-07-01
Focus Area: Codemod Registry Drift & Idempotency Coverage Gap
Strategy Type: Custom
Custom Area: Yes — gh-aw has 62 codemods, a growing migration surface requiring robust test coverage.

Executive Summary

The codemod sub-system (pkg/cli/codemod_*.go) is a critical migration engine with 62 registered codemods. Three test-coverage gaps compound as the registry grows: (1) TestGetAllCodemods_ContainsExpectedCodemods only asserts 42 of 62 IDs — 20 codemods (32%) could be silently removed without failing any test; (2) zero codemods have double-apply idempotency tests, leaving gh aw fix open to silent file corruption on repeated runs; (3) only 2 of 62 codemods (3%) have fuzz tests despite performing complex line-level YAML manipulation.

Full Analysis Report

Current State Assessment

Metrics Collected:

Metric	Value	Status
Total registered codemods	62	✅
IDs in ContainsExpectedCodemods test	42	❌
IDs missing from that test	20 (32%)	❌
Codemods with idempotency tests	0 of 62	❌
Codemods with fuzz tests	2 of 62 (3%)	⚠️

Findings

Strengths

• TestGetAllCodemods_InExpectedOrder enforces exact ordering — reordering fails immediately.
• TestGetAllCodemods_NoduplicateIDs prevents ID collisions.
• Two highest-risk codemods already have fuzz tests.

Areas for Improvement

• Registry coverage gap ❌ HIGH: 20 IDs absent from TestGetAllCodemods_ContainsExpectedCodemods in pkg/cli/fix_codemods_test.go.
• Zero idempotency tests ❌ HIGH: No codemod verifies applied == false on a second run against already-transformed output.
• Fuzz coverage ⚠️ MEDIUM: 60/62 codemods manipulate YAML line-by-line with no fuzz corpus.

Detailed Analysis

20 IDs missing from ContainsExpectedCodemods:
add-comment-discussion-removal, mcp-mode-to-type-migration, install-script-url-migration, bash-anonymous-removal, activation-outputs-to-sanitized-step, roles-to-on-roles, bots-to-on-bots, engine-steps-to-top-level, assign-to-agent-default-agent-to-name, playwright-allowed-domains-migration, expires-integer-to-string, app-to-github-app, github-app-app-id-to-client-id, github-repos-to-allowed-repos, features-cli-proxy-to-tools-github-mode, features-difc-proxy-to-tools-github, sandbox-mcp-container-removal, sandbox-mcp-version-removal, sandbox-agent-false-removal, run-install-scripts-to-runtimes-node.

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: Split the following tasks into individual work items.

Improvement Tasks

Task 1: Expand TestGetAllCodemods_ContainsExpectedCodemods to Cover All 62 IDs

Priority: High | Effort: Small | Focus Area: Registry Drift

Description: Add the 20 missing codemod IDs to the expectedIDs slice in TestGetAllCodemods_ContainsExpectedCodemods (pkg/cli/fix_codemods_test.go, lines 64–116).

Acceptance Criteria:

• All 62 IDs present in expectedIDs (not 42)
• go test ./pkg/cli/... -run TestGetAllCodemods_ContainsExpectedCodemods -tags "!integration" passes

Code Region: pkg/cli/fix_codemods_test.go

In `pkg/cli/fix_codemods_test.go`, add the following 20 IDs to the `expectedIDs` slice in `TestGetAllCodemods_ContainsExpectedCodemods`: `add-comment-discussion-removal`, `mcp-mode-to-type-migration`, `install-script-url-migration`, `bash-anonymous-removal`, `activation-outputs-to-sanitized-step`, `roles-to-on-roles`, `bots-to-on-bots`, `engine-steps-to-top-level`, `assign-to-agent-default-agent-to-name`, `playwright-allowed-domains-migration`, `expires-integer-to-string`, `app-to-github-app`, `github-app-app-id-to-client-id`, `github-repos-to-allowed-repos`, `features-cli-proxy-to-tools-github-mode`, `features-difc-proxy-to-tools-github`, `sandbox-mcp-container-removal`, `sandbox-mcp-version-removal`, `sandbox-agent-false-removal`, `run-install-scripts-to-runtimes-node`. Verify with `go test ./pkg/cli/... -run TestGetAllCodemods_ContainsExpectedCodemods -tags "!integration"`.

---

Task 2: Add Idempotency Tests to 5 High-Risk Codemods

Priority: High | Effort: Medium | Focus Area: Codemod Correctness

Description: No codemod verifies that a second apply is a no-op. Add a TestXxx_Idempotency to: codemod_activation_outputs_test.go, codemod_steps_run_secrets_env_test.go, codemod_checkout_persist_credentials_false_test.go, codemod_permissions_test.go, codemod_mcp_network_test.go. Each test should apply the codemod twice and assert applied == false and identical output on the second call.

Acceptance Criteria:

• Each of the 5 test files has a TestXxx_Idempotency function
• Second apply asserts applied == false and result1 == result2
• go test ./pkg/cli/... -run Idempotency -tags "!integration" passes

Code Region: pkg/cli/codemod_activation_outputs_test.go, pkg/cli/codemod_steps_run_secrets_env_test.go, pkg/cli/codemod_checkout_persist_credentials_false_test.go, pkg/cli/codemod_permissions_test.go, pkg/cli/codemod_mcp_network_test.go

For each of the 5 codemod test files listed in Code Region, add a `TestXxx_Idempotency` function. Take a transformable input from an existing test case. Apply the codemod, assert `applied == true`. Re-parse the frontmatter from the result. Apply again, assert `applied == false` and the output equals the first result. Verify with `go test ./pkg/cli/... -run Idempotency -tags "!integration"`.

---

Task 3: Add Fuzz Tests for roles, bots, engine-steps Codemods

Priority: Medium | Effort: Medium | Focus Area: Fuzz Coverage

Description: Create codemod_roles_fuzz_test.go, codemod_bots_fuzz_test.go, codemod_engine_steps_fuzz_test.go following the pattern in codemod_checkout_persist_credentials_false_fuzz_test.go. Each needs 3 seed corpus entries and must assert err == nil.

Acceptance Criteria:

• 3 new fuzz test files exist in pkg/cli/
• Each has //go:build !integration and ≥3 seed entries
• go test ./pkg/cli/ -fuzz=FuzzRolesCodemod -fuzztime=5s does not panic

Code Region: pkg/cli/codemod_roles.go, pkg/cli/codemod_bots.go, pkg/cli/codemod_engine_steps.go

Create `codemod_roles_fuzz_test.go`, `codemod_bots_fuzz_test.go`, and `codemod_engine_steps_fuzz_test.go` in `pkg/cli/`. Follow the pattern in `codemod_checkout_persist_credentials_false_fuzz_test.go`. Use `//go:build !integration`, seed with inputs from the corresponding unit test files, and assert `err == nil` in the fuzz body.

---

Task 4: Refactor ContainsExpectedCodemods to Derive from expectedCodemodOrder()

Priority: Medium | Effort: Small | Focus Area: Drift Prevention

Description: Replace the static 42-entry expectedIDs slice with a bidirectional check against expectedCodemodOrder(), so new codemods added to GetAllCodemods() without updating the order list fail the test immediately.

Acceptance Criteria:

• TestGetAllCodemods_ContainsExpectedCodemods no longer has a static partial list
• Adding a codemod to GetAllCodemods() without expectedCodemodOrder() fails both tests
• go test ./pkg/cli/... -run TestGetAllCodemods -tags "!integration" passes

Code Region: pkg/cli/fix_codemods_test.go (TestGetAllCodemods_ContainsExpectedCodemods)

Replace the static `expectedIDs` slice in `TestGetAllCodemods_ContainsExpectedCodemods` with a bidirectional check: (1) every ID in `expectedCodemodOrder()` must exist in `GetAllCodemods()`; (2) every ID in `GetAllCodemods()` must exist in `expectedCodemodOrder()`. This removes drift between the two lists.

---

📊 Historical Context

Previous Focus Areas

Date	Focus Area	Type	Custom	Key Outcomes
2026-06-30	Unauthenticated API Fallback Call Chain	Custom	Y	3 http.DefaultClient, 2 missing ctx params
2026-06-29	tolowerequalfold Linter False Positives	Custom	Y	7/17 nolint suppressions are false positives
2026-06-26	Performance	Standard	N	4x schema reparse, regexp recompile
2026-06-24	Import Pipeline Reliability	Custom	Y	31 silent unmarshal errors

---

🎯 Recommendations

Immediate Actions (This Week)

1. Add 20 missing IDs to ContainsExpectedCodemods — Priority: High
2. Add idempotency tests to 5 codemods — Priority: High

Short-term Actions (This Month)

1. Refactor ContainsExpectedCodemods (Task 4) — Priority: Medium
2. Add fuzz tests for roles/bots/engine-steps (Task 3) — Priority: Medium

---

📈 Success Metrics

• ContainsExpectedCodemods coverage: 42/62 → 62/62
• Codemods with idempotency tests: 0 → 5 immediately
• Fuzz test coverage: 2/62 → 5/62

---

Next Steps

1. Review and prioritise the tasks above
2. Assign tasks to Copilot coding agent via planner agent
3. Re-evaluate in 30 days

---

References:

• §28521783021

Generated by Repository Quality Improvement Agent

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• proxy.golang.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "proxy.golang.org"

See Network Configuration for more information.

Generated by ⚡ Repository Quality Improvement Agent · 193.1 AIC · ⌖ 12.1 AIC · ⊞ 6.7K · ◷

• expires on Jul 2, 2026, 5:58 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Repository Quality Improvement Agent.

A newer discussion is available at Discussion #42975.