v0.19.0

.claude/commands/new-stage.md

@@ -744,6 +744,39 @@ Key invariants:
 - `given MemberGetSet = db.getSet` must be re-established at the top of each recursive call so
   navigation helpers see the updated member structure.
 
+**This pattern is not just for flattening — it is the general cure for "nested same-kind
+rewrites".** Whenever a stage rewrites a construct via `Patch.Move` or
+`ReplaceWithLast(ChangeRefAndRemove)` *plus* a satellite patch anchored on its body (an `After`
+increment, a `Before` goto, a consumed-member `Remove`), and that construct can **nest inside
+another of the same kind**, rewriting both in one pass conflicts: the inner one appears both inside
+the outer one's moved descendants (`Flattened`) AND in its own patches → ownership-order breakage
+or `Received two different patches for the same member`. Two real instances from this codebase:
+- `SimplifyRTOps` rewriting nested `for` loops (`DFForBlock` → `while` + iterator + `After`
+  increment).
+- `FlattenStepBlocks` extracting `StepBlock`s nested in conditional branches (`Move` + inserted
+  goto + consumed-goto `Remove`).
+
+The fix in both was identical: drive the rewrite with `@tailrec ...Repeatedly(db)` and **gate each
+pass to the non-nested instances** so an outer and inner are never patched together — e.g. process
+only the *innermost* (no transformable descendant) or only the *outermost* (no transformable
+ancestor), and let later passes pick up the rest:
+```scala
+// innermost-first gate (SimplifyRTOps): skip a for loop that contains another transformable one
+case fb: DFLoop.DFForBlock
+    if isTransformable(fb) && !fb.members(MemberView.Flattened).exists {
+      case inner: DFLoop.DFForBlock => isTransformable(inner); case _ => false
+    } =>
+
+// outermost-first gate (FlattenStepBlocks): skip a step that has a transformable step ancestor
+val targets = pb.members(MemberView.Flattened).collect {
+  case sb: StepBlock if isCondBranchStep(sb) && !hasCondBranchStepAncestor(sb) => sb
+}
+```
+Each pass strictly reduces the count of nested instances, so the `@tailrec` loop converges. This
+keeps the multi-phase structure intact — a stage with several phases can wrap just the affected
+phase(s) in their own `...Repeatedly` driver (as `FlattenStepBlocks` does for conditional
+extraction and structural flattening independently).
+
 ### Pattern 10 — Replace a DFOwner while preserving its children
 
 Use when you want to swap one owner block for another (e.g. `DFForBlock` → `DFWhileBlock`) but
@@ -867,6 +900,69 @@ extension [T: HasDB](t: T)
 
 ---
 
+## Debugging a Stage Failure with TRACE
+
+When a full compilation blows up inside the stage pipeline (a `SanityCheck` failure, a patch
+conflict, etc.), the fastest way to localize and reproduce it is the **TRACE log**, which prints
+the full design code string after every stage that changes the DB.
+
+### Enabling and running
+
+Put a design in `lib/src/test/scala/Playground.scala` (or `core/.../Playground.scala`) and add at
+the top of the file:
+```scala
+given options.CompilerOptions.LogLevel = _.TRACE
+```
+Then run the whole pipeline for a top-level design named `Foo` via its generated `top_Foo` main:
+```bash
+sbtn.bat ";libPlayground;lib/Test/runMain top_Foo"          # core equivalent: corePlayground + core/Test/runMain
+```
+Pass tool/backend arguments after `--`:
+```bash
+sbtn.bat ";libPlayground;lib/Test/runMain top_Foo -- simulate -t questa"     # or -t nvc/ghdl with -b vhdl
+```
+A design with **no ports + a `finish()`** is treated as a self-contained simulation top, so the
+default (no-arg) action becomes *simulate* instead of *compile*.
+
+### Reading the trace to localize the failing stage
+
+The log emits `Running stage X....` / `Finished stage X` around each stage, runs a `SanityCheck`
+after every non-`NoCheckStage`, and prints the code string after any stage that changed the DB. So:
+- The **`SanityCheck` that throws** (`Failed ownership check!`, `Received two different patches for
+  the same member`, …) fires *immediately after* the offending stage — that stage name is your
+  culprit, even if the symptom (a dangling owner, a duplicate Goto) looks structural.
+- **Don't assume the failing stage is the one you changed.** A stage can pass its own sanity check
+  and emit a valid DB, yet a *later* stage chokes on a shape your stage newly produced. Read the
+  `Running stage` sequence to find the first failure, not the first suspect.
+
+### Harvesting a self-contained reproducer from the trace
+
+The code printout emitted **immediately before** the failing stage is that stage's *input* — and
+crucially it is a **valid** design (it just passed the previous stage's sanity check). Because
+stages run before later lowering, the printed constructs are exactly the failing stage's input
+form, so you can drop that printout almost verbatim into a self-contained `<Stage>Spec` test (see
+*Test Authoring Rules* below) as the reproducer. This turns an opaque end-to-end crash into a fast,
+isolated unit test in one step — write the test, watch it fail with the same error, then fix.
+
+### Caveats
+
+- Re-running an unchanged design may short-circuit on the on-disk cache
+  (`Loading committed design from cache...`) and skip the stages (and the trace) entirely. Pass
+  **`--nocache`** to disable caching — it is a DFHDL App option that goes *before* the `--`
+  separator (it is NOT a positional arg after `--`; placing it after `--` fails with
+  `[scallop] Error: Excess arguments provided: '--nocache'`). The DFHDL App command
+  (`compile` / `simulate` / …) goes *after* `--`:
+  ```bash
+  sbtn.bat ";libPlayground;lib/Test/runMain top_Foo --nocache -- compile"
+  sbtn.bat ";libPlayground;lib/Test/runMain top_Foo --nocache -- simulate -t questa"
+  ```
+  (Alternatively clear `sandbox/<Top>` via `sbtn clearSandbox`, or edit the design — but `--nocache`
+  is the lightweight option for repeated trace runs.)
+- `libPlayground` / `corePlayground` zero out other subprojects' test sources for the session. To
+  run `StagesSpec` tests again afterwards, reset with a leading `;reload`.
+
+---
+
 ## Test Authoring Rules
 
 **Tests must be self-contained.** Each test should only exercise the stage under test. Do not write input designs that rely on a prior stage to produce the IR shape that the current stage expects — write that IR shape directly using the DFHDL DSL.
@@ -978,6 +1074,14 @@ abstract class StageSpec(stageCreatesUnrefAnons: Boolean = false)
     case classes that have no dedicated single-argument unapply.
 16. **Assuming duplicate designs have members** — designs tagged `DuplicateTag` have **no members**
     in the DB (ports, domain blocks, and values are removed during immutable DB creation). 
+17. **Rewriting nested same-kind constructs in one pass** — if your stage rewrites a construct that
+    can nest inside another of the same kind (nested `for` loops, steps-in-conditionals nested in
+    steps, etc.) via a `Move` / `ReplaceWithLast(ChangeRefAndRemove)` plus a body-anchored satellite
+    patch, doing the outer and inner in one patch list conflicts: the inner appears both in the
+    outer's `Flattened` moved descendants and in its own patches → ownership breakage or
+    `Received two different patches for the same member`. Drive it `@tailrec` and gate each pass to
+    the innermost-only or outermost-only instances (see Pattern 9). This is easy to miss because a
+    single-level test passes — **always add a nested test** for any such rewrite.
 ---
 
 ## API Notes

compiler/stages/src/main/scala/dfhdl/compiler/stages/DropLocalDcls.scala

@@ -4,6 +4,7 @@ import dfhdl.compiler.analysis.*
 import dfhdl.compiler.ir.*
 import dfhdl.compiler.patching.*
 import dfhdl.options.CompilerOptions
+import scala.annotation.tailrec
 
 //format: off
 /** This stage moves local variable and constant declarations out of their lexical scope to a
@@ -76,6 +77,36 @@ import dfhdl.options.CompilerOptions
   *     stmt1
   *     stmt2
   * }}}
+  *
+  * ===Rule 3: Declarations inside step blocks===
+  * A local variable or constant declared inside a `StepBlock` (an RT FSM state) — either directly
+  * or nested inside a conditional within the step — is lifted out of the step, because the FSM
+  * states generated from steps cannot carry declarations:
+  *   - For non-VHDL backends: moved to before the enclosing process block (design level).
+  *   - For VHDL: moved to just before the outermost (top-level) step block, keeping the declaration
+  *     at the process-body level where VHDL allows process variable declarations.
+  * {{{
+  * // Before — zz declared inside a step
+  * class ID extends RTDesign:
+  *   process:
+  *     def S_0: Step =
+  *       val zz = SInt(16) <> VAR
+  *       ...
+  *
+  * // After (Verilog) — moved to design level, before the process
+  * class ID extends RTDesign:
+  *   val zz = SInt(16) <> VAR
+  *   process:
+  *     def S_0: Step =
+  *       ...
+  *
+  * // After (VHDL) — moved to just before the top-level step, inside the process
+  * class ID extends RTDesign:
+  *   process:
+  *     val zz = SInt(16) <> VAR
+  *     def S_0: Step =
+  *       ...
+  * }}}
   */
 //format: on
 case object DropLocalDcls extends HierarchyStage:
@@ -84,7 +115,7 @@ case object DropLocalDcls extends HierarchyStage:
   def transformSubDB(rootDB: DB)(using getSet: MemberGetSet, co: CompilerOptions, rg: RefGen): DB =
     val keepProcessDcls = co.backend.isVHDL
     val patches = subDB.members.view
-      // only var or constant declarations ,
+      // only var or constant declarations,
       // and we also require their anonymous dependencies
       .flatMap {
         // skip iterator declarations
@@ -93,25 +124,43 @@ case object DropLocalDcls extends HierarchyStage:
         case m @ DclConst() if !m.isGlobal => m.collectRelMembers(includeOrigVal = true)
         case _                             => None
       }
-      .map(m => (m, m.getOwnerBlock))
-      .flatMap {
-        // declarations inside conditional blocks
-        case (dcl, cb: DFConditional.Block) =>
-          val topCondHeader = cb.getTopConditionalHeader
-          // if we don't keep process vars, we check if the owner is a process block,
-          // and if so, we need to move the declarations before it.
-          val moveBeforeMember = topCondHeader.getOwnerBlock match
-            case pb: ProcessBlock if !keepProcessDcls => pb
-            case _                                    => topCondHeader
-          Some(moveBeforeMember -> Patch.Move(dcl, Patch.Move.Config.Before))
-        // declarations inside process blocks if we should not keep them
-        case (dcl, pb: ProcessBlock) if !keepProcessDcls =>
-          Some(pb -> Patch.Move(dcl, Patch.Move.Config.Before))
-        case _ => None
-      }
+      .flatMap(dclMovePatch(_, keepProcessDcls))
       .toList
     subDB.patch(patches)
   end transformSubDB
+
+  // Computes the move patch (if any) relocating a local declaration `dcl` out of its lexical scope
+  // to a position supported by the target language. Returns `None` when the declaration is already
+  // at a valid position (directly at design level, or — under VHDL — directly inside a process).
+  private def dclMovePatch(dcl: DFMember, keepProcessDcls: Boolean)(using
+      MemberGetSet
+  ): Option[(DFMember, Patch)] =
+    val (anchor, scopeBlock) = climbToScope(dcl)
+    scopeBlock match
+      // non-VHDL: declarations are not allowed inside process blocks, so move to the design level
+      // before the process block.
+      case pb: ProcessBlock if !keepProcessDcls =>
+        Some(pb -> Patch.Move(dcl, Patch.Move.Config.Before))
+      // VHDL process scope, or design scope: move before the outermost in-scope anchor, but only
+      // when the declaration actually needs to escape an enclosing conditional or step block.
+      case _ =>
+        if anchor ne dcl then Some(anchor -> Patch.Move(dcl, Patch.Move.Config.Before))
+        else None
+
+  // Climbs from a member up through enclosing conditional and step blocks, returning the outermost
+  // in-scope anchor to move before, paired with the nearest enclosing non-conditional, non-step
+  // scope block (a ProcessBlock or DFDesignBlock, or a loop block). When the member is inside a
+  // conditional, the anchor is the top-level conditional header for that scope; otherwise it is the
+  // member itself. Step blocks are escaped one level at a time until a non-step scope is reached.
+  @tailrec private def climbToScope(m: DFMember)(using MemberGetSet): (DFMember, DFBlock) =
+    val (anchor, scopeBlock) = m.getOwnerBlock match
+      case cb: DFConditional.Block =>
+        val topCondHeader = cb.getTopConditionalHeader
+        (topCondHeader, topCondHeader.getOwnerBlock)
+      case b => (m, b)
+    scopeBlock match
+      case sb: StepBlock => climbToScope(sb)
+      case _             => (anchor, scopeBlock)
 end DropLocalDcls
 
 extension [T: HasDB](t: T)

compiler/stages/src/main/scala/dfhdl/compiler/stages/FlattenStepBlocks.scala

@@ -166,6 +166,7 @@ import scala.annotation.tailrec
   */
 //format: on
 case object FlattenStepBlocks extends HierarchyStage:
+  // TODO: Not running FoldControlSteps for now
   def dependencies: List[Stage] = List(DropRTWaits, ExplicitNamedVars, DropLocalDcls)
   def nullifies: Set[Stage] = Set()
 
@@ -182,22 +183,29 @@ case object FlattenStepBlocks extends HierarchyStage:
         case _                                   => Nil
       }.toList
     )
-    // Phase 1: conditional branch extraction (uses db0 for updated member structure)
-    val db1 = locally {
-      given MemberGetSet = db0.getSet
-      db0.patch(
-        db0.members.view.flatMap {
-          case pb: ProcessBlock if pb.isInRTDomain => collectConditionalExtractionPatches(pb)
-          case _                                   => Nil
-        }.toList
-      )
-    }
+    // Phase 1: conditional branch extraction, one level at a time (uses db0 for updated structure)
+    val db1 = extractCondBranchStepsRepeatedly(db0)
     // Phase 2: structural flattening, one level at a time (uses db1, applied repeatedly)
     val db2 = flattenRepeatedly(db1)
     // Phase 3: Goto ChangeRef
     db2.patch(gotoPatchList)
   end transformSubDB
 
+  // Repeatedly extract one nesting level of conditional-branch StepBlocks until none remain nested
+  // inside another conditional-branch step. Extracting an outer and an inner conditional-branch step
+  // in the same pass conflicts: the inner step (and its Gotos) appears both in the outer step's
+  // moved descendants (`Flattened`) and in its own extraction patches. Processing the outermost
+  // conditional-branch steps first un-nests them to ProcessBlock level, so the formerly-inner steps
+  // become outermost on the next pass.
+  @tailrec private def extractCondBranchStepsRepeatedly(db: DB)(using RefGen): DB =
+    given MemberGetSet = db.getSet
+    val patches = db.members.view.flatMap {
+      case pb: ProcessBlock if pb.isInRTDomain => collectConditionalExtractionPatches(pb)
+      case _                                   => Nil
+    }.toList
+    if patches.isEmpty then db
+    else extractCondBranchStepsRepeatedly(db.patch(patches))
+
   // Repeatedly flatten one nesting level of StepBlocks until all are direct pb children.
   @tailrec private def flattenRepeatedly(db: DB)(using RefGen): DB =
     given MemberGetSet = db.getSet
@@ -210,6 +218,19 @@ case object FlattenStepBlocks extends HierarchyStage:
 
   // --- Shared helpers ---
 
+  // A regular StepBlock that sits directly inside a conditional branch.
+  private def isCondBranchStep(s: StepBlock)(using MemberGetSet): Boolean =
+    s.isRegular && s.getOwner.isInstanceOf[DFConditional.Block]
+
+  // True if any enclosing StepBlock ancestor (up to the ProcessBlock) is itself a conditional-branch
+  // step — i.e. `s` is nested inside another conditional-branch step and must wait for a later pass.
+  @tailrec private def hasCondBranchStepAncestor(m: DFMember)(using MemberGetSet): Boolean =
+    m.getOwner match
+      case parentStep: StepBlock =>
+        isCondBranchStep(parentStep) || hasCondBranchStepAncestor(parentStep)
+      case _: ProcessBlock => false
+      case owner           => hasCondBranchStepAncestor(owner)
+
   private def collectDirectFlatSteps(owner: DFOwner)(using MemberGetSet): List[StepBlock] =
     owner.members(MemberView.Folded).flatMap {
       case sb: StepBlock if sb.isRegular => sb :: collectDirectFlatSteps(sb)
@@ -350,15 +371,19 @@ case object FlattenStepBlocks extends HierarchyStage:
     step5InterStep ++ step6
   end collectInterStepPatches
 
-  // --- Phase 1: Conditional branch extraction (uses db0's member structure) ---
+  // --- Phase 1: Conditional branch extraction (one level per pass; see
+  // `extractCondBranchStepsRepeatedly`) ---
 
   private def collectConditionalExtractionPatches(
       pb: ProcessBlock
   )(using MemberGetSet, RefGen): List[(DFMember, Patch)] =
     val flatSteps = collectDirectFlatSteps(pb)
     if flatSteps.isEmpty then return Nil
+    // Only the outermost conditional-branch steps this pass: a step nested inside another
+    // conditional-branch step is moved as part of that ancestor's descendants and is extracted on a
+    // later pass, avoiding overlapping Move/Remove patches on the shared nested members.
     val conditionalBranchSteps = pb.members(MemberView.Flattened).collect {
-      case sb: StepBlock if sb.isRegular && sb.getOwner.isInstanceOf[DFConditional.Block] => sb
+      case sb: StepBlock if isCondBranchStep(sb) && !hasCondBranchStepAncestor(sb) => sb
     }
     conditionalBranchSteps.flatMap { s =>
       val (cb, consumedGoto) = findConsumedGoto(s)