Skip to content

Feature/only run necessary tests after reset#84

Merged
nawforce merged 13 commits into
mainfrom
feature/only-run-necessary-tests-after-reset
Jun 29, 2026
Merged

Feature/only run necessary tests after reset#84
nawforce merged 13 commits into
mainfrom
feature/only-run-necessary-tests-after-reset

Conversation

@jwoodwardcertinia

Copy link
Copy Markdown
Contributor

Previously, when a stalled test run was reset, we re-ran the entire suite — discarding tens of thousands of already-passed results. Now the runner keeps results from classes that finished and re-runs only the unfinished ones, so a reset costs minutes instead of re-running everything. Genuine failures are still re-run sequentially at the end.

Also: a run that stalls before processing (e.g. stuck queued) is aborted rather than left enqueued (fixing an ALREADY_IN_PROCESS cascade)

Logging changes

  • a per-reset diagnostic JSON is written alongside other output
  • Status line now appends | No progress / while a run is stalling
  • new log line: Reset / before abandoning run logged on each reset.
  • new log line: Reusing tests from completed classes; rerunning remaining tests across classes — reuse summary
  • new log line: Test run '' stuck in with no progress, abandoning this attempt — when aborting a stuck-before-processing run.

@nawforce

Copy link
Copy Markdown
Contributor

I think these are edge cases from Codex review:

  • Method-scoped reruns can be widened to whole-class reruns after a reset. In src/runner/TestRunner.ts:307, prepareRestart() returns { classId } for each pending class. If the
    current runner was created with method-specific TestItems, such as the missing-test rerun path in src/command/Testall.ts:188, a stall/reset will rerun every method in that class
    instead of only the originally requested methods. That can make “only run necessary tests after reset” false for method-filtered runs and can add unexpected test results to
    reports.

  • numberOfResets can be overwritten after recursive missing-test reruns. src/command/Testall.ts:196 recursively calls asyncRun(), whose store.saveAsyncResult() records the nested
    run’s reset count. But after it returns, src/command/Testall.ts:203 sets store.numberOfResets back to the outer run’s count. If the missing-test rerun is the one that resets, the
    final report can show 0 resets. If both reset, it reports only the outer count rather than the total.

  • Reset attempt run IDs are not preserved in final summaries. AsyncTestRunner.runInternal() recursively starts new async runs, but the returned TestRunnerResult only carries the
    final attempt’s AsyncApexJobId; src/results/TestResultStore.ts:41 then records only that one ID. This makes reports/debug follow-up less complete after a reset, and src/command/
    Testall.ts:357 will not warn about coverage incompleteness caused by reset attempts unless there was also a higher-level missing-test rerun.

These all look related to the "missing test" handling which is triggered in Testall:asyncRun when the results don't cover all of the tests we expect to see. I am not that sure how common that is in practice but we have gone to some lengths to detect and correct via a method level re-runs.

@jwoodwardcertinia

Copy link
Copy Markdown
Contributor Author

@nawforce

  • Method-scoped reruns can be widened to whole-class reruns after a reset.
    • That's fair and I have fixed it. I don't think I've ever seen this method-scoped re-running happening, but it is a defect in this change and should be addressed. There was an option to reduce the scope of the rerun if any of the tests had actually passed already, but it got pretty complicated, so I kept it simple given how often the scenario happens. (I.e. not at all ;) )
  • numberOfResets can be overwritten after recursive missing-test reruns.
    • This is also related to the little used missing-method retry mechanism. I agonized over what numberOfResets in the result summary should actually mean. In the end I went with the total of both. It highlights that numberOfResets is ambiguous. You can specify maxTestRunRetries to some value and get back more resets than that if the missing-method retry kicks in, which might be confusing.
  • Reset attempt run IDs are not preserved in final summaries.
    • This was a pre-existing issue that I'm saying is out of scope. There is another issue tangled in here that we always get a warning logged when codeCoverage = true and disableCoverageReport = false. I have created an issue for that, but it's pretty low risk.

@nawforce nawforce merged commit d7af5bd into main Jun 29, 2026
1 check passed
@nawforce nawforce deleted the feature/only-run-necessary-tests-after-reset branch June 29, 2026 22:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants