Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
d2c3c83
add DFWarning support under LogEvents, common ancestor with DFError
Apr 6, 2026
035cd15
feat: add warnings to possible `Int` usage with verilog semantics.
Apr 7, 2026
05d3c1f
fix: redundant partial naming
Apr 7, 2026
f16de52
docs: fix part-select return types, add ++ concatenation, clarify .width
Apr 7, 2026
3d0a12e
fix: resolve exponential compile time in chained transparent inline o…
Apr 7, 2026
12765d2
fix: replace cleanTypeHack with ascribeWidenedType for clean error types
Apr 7, 2026
f42aea1
test: add regression test for clean type error messages
Apr 7, 2026
7baf4d3
perf: remove unnecessary transparent from Check inline givens
Apr 7, 2026
5e066b6
perf: remove unnecessary transparent from CTName and DualSummonTrapError
Apr 7, 2026
e7bb056
perf: add FlattenInlined compiler plugin phase
Apr 7, 2026
c9347da
remove redundant inline from given
Apr 7, 2026
9a44adf
establish corePlayground and libPlayground sbt commands
Apr 7, 2026
bef73e4
belongs to previous commit
Apr 7, 2026
8c37c74
update Scala to 3.8.3
Apr 7, 2026
eb7deba
update sbt
Apr 7, 2026
f26a516
update claude with some info from what we've learned
Apr 7, 2026
f809ca4
update compile performance documentation to clarify test compilation …
Apr 7, 2026
91ee981
todo: test in ExplicitNamedVarsSpec to fix
Apr 7, 2026
4eef30d
companion to the change in 12765d2ce5350a8edbf1d977ec204796d10fb8e7 t…
Apr 8, 2026
49b7b8e
restore `transparent inline` to fix reporting issues
Apr 8, 2026
e865802
fix exponential compile time from chained transparent inline operations
Apr 8, 2026
e1b822b
add regression test for chained inline operation compile time
Apr 8, 2026
94f2c19
merge consecutive associative Func ops into multi-arg during elaboration
Apr 8, 2026
596e0a9
wip new relaxed rules for decimal operations
Apr 9, 2026
0c64d58
wip removed DFDecimal's IsScalaInt
Apr 9, 2026
0f73c0f
wip: removed DFDecimal's dependent mask types
Apr 9, 2026
92edcc6
feat: new relaxed rules for DFDecimal
Apr 9, 2026
eb7a85a
Merge branch 'training' of https://github.com/DFiantHDL/dfhdl_by_agen…
Apr 9, 2026
b8fe939
fix #375
Apr 9, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
207 changes: 207 additions & 0 deletions .claude/commands/compile-perf.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,207 @@
# DFHDL Compile Time Performance Guide

## Quick Setup for Debugging Compile Times

```bash
sbtn.bat corePlayground # Only compiles core/src/test/scala/Playground.scala
sbtn.bat libPlayground # Only compiles lib/src/test/scala/Playground.scala
sbtn.bat compile # Compile dependencies (run after clean or code changes)
sbtn.bat "core/Test/compile" # Compile just the Playground test file
```

Enable profiling in `build.sbt` (in `pluginTestUseSettings`):
```scala
"-Yprofile-enabled",
// "-Yprofile-trace:compiler.trace"
```

Always use `sbtn.bat` (sbt client), never `sbt`.

## Test Methodology

### Measuring Compile Time

1. Run `corePlayground` or `libPlayground` once per session (persists until `shutdown`)
2. Run `clean` then `compile` to build dependencies
3. Edit `Playground.scala` with the test code
4. Run `core/Test/compile` — this compiles ONLY the Playground file
5. To force recompile without changing code, add/change a comment (e.g., `// v2`)
6. Compare `Total time` or `posttyper` phase timing with `-Yprofile-enabled`

When running the full test suite, always separate compilation from execution:
```bash
sbtn.bat Test/compile # compile all tests first — measure compile workload
sbtn.bat test # then run tests — measure runtime workload
```
This avoids conflating compile time with test runtime in measurements.

### When to Run `clean`

- **After modifying `internals/` or `plugin/`** — the incremental compiler doesn't always detect changes in macro code or plugin classes. A stale compiled macro will produce the OLD tree even though the source changed. Always `clean` after touching:
- `internals/src/main/scala/dfhdl/internals/Exact.scala`
- `internals/src/main/scala/dfhdl/internals/Checked.scala`
- `plugin/src/main/scala/plugin/*.scala`
- Any file that defines macros or compiler plugin phases
- **After modifying `core/`** — changes to `DFVal.scala`, `DFDecimal.scala`, etc. affect inline definitions that downstream code depends on
- **NOT needed** when only editing `Playground.scala` — incremental compilation handles this correctly

### OOM and JVM Issues

The sbt JVM can run out of memory when compiling large test suites or after many incremental compilations. Symptoms:
- `java.lang.OutOfMemoryError: Java heap space`
- `GC overhead limit exceeded`
- Compile times suddenly spike to 5+ minutes
- sbt becomes unresponsive

Recovery:
```bash
taskkill //F //IM java.exe # Kill all Java processes (Windows)
sleep 2 # Wait for cleanup
sbtn.bat clean # Start fresh
sbtn.bat compile # Rebuild
```

The `sbtn` client will auto-start a new JVM server. Always kill java BEFORE starting sbtn again, otherwise the old server may still hold ports/locks.

When running the full test suite (`sbtn.bat test`), watch for OOM — the test compilation of all spec files plus Playground can exhaust 1GB heap. If the `lib/src/test/scala/Playground.scala` or `core/src/test/scala/Playground.scala` contain heavy test code (many operands, design hierarchies), comment them out before running the full suite.

### Printing Trees for Debugging

In the `FlattenInlinedPhase` plugin, use `prepareForUnit` / `transformUnit` overrides to print trees before/after transformation:
```scala
override def prepareForUnit(tree: Tree)(using Context): Context =
if tree.source.path.toString.contains("Playground.scala") then
println(tree.show)
ctx

override def transformUnit(tree: Tree)(using Context): Tree =
val result = treeMap.transform(tree)
if tree.source.path.toString.contains("Playground.scala") then
println(result.show)
result
```

Use `System.err.println` for debug output that needs to bypass sbt's output buffering.

### Upstream Scala 3 Bug

The root cause is a performance bug in `PostTyper.scala` (line ~568 in current nightly):
```scala
case tree @ Inlined(call, bindings, expansion) if !tree.inlinedFromOuterScope =>
...
withMode(Mode.NoInline)(transform(call)) // expensive, result DISCARDED
val callTrace = Inlines.inlineCallTrace(call.symbol, pos)(...)
cpy.Inlined(tree)(callTrace, ...) // callTrace replaces call entirely
```
`transform(call)` re-transforms the full call tree (running `checkBounds`, `normalizeTypeArgs`, etc. on any TypeApply inside it), but the result is thrown away — the next line computes `callTrace` (a minimal `Ident`) and uses that instead. With N chained transparent inlines producing O(N) Inlined nodes with increasingly complex call trees, this causes O(N²) or worse behavior.

**Fix**: Either remove `withMode(Mode.NoInline)(transform(call))` entirely, or replace it with just the side-effecting checks needed (which `CrossVersionChecks.checkRef` already covers on the line above). See https://github.com/scala/scala3/blob/main/CONTRIBUTING.md for contribution guide.

A Scala 3 clone is available at `C:\Users\OronPort\IdeaProjects\forclaude\scala3` for building a minimal reproducer and preparing the upstream PR.

## Root Cause: Transparent Inline Expansion

The primary compile time bottleneck comes from Scala 3's `transparent inline` mechanism used extensively in DFHDL for type-level computation. The key file is `internals/src/main/scala/dfhdl/internals/Exact.scala`.

### How It Works

Operations like `+`, `-`, `<>`, `apply` are defined as:
```scala
transparent inline def +(inline rhs: SupportedValue)(using DFCG): DFValAny =
exactOp2[FuncOp.+.type, DFC, DFValAny](lhs, rhs)
```

`exactOp2` is a `transparent inline` macro that:
1. Extracts the exact types of `lhs` and `rhs` via `exactInfo`
2. Summons an `ExactOp2` typeclass instance for those exact types
3. Returns the result with the precise output type

### The Exponential Problem

When chaining operations like `a + b + c + d`, each `+` is `transparent inline`, and:
- The `inline lhs` parameter causes the entire previous expansion to be substituted
- The compiler's `posttyper` phase re-processes the growing tree
- Tree sizes grow linearly but `posttyper` time grows super-linearly

### What We Fixed

1. **`cleanTypeHack` removal** — Was an extra `transparent inline` + `inline match` layer around every `exactOp*` call. Replaced with `ascribeWidenedType` using `.dealias` type ascription. See `Exact.scala`.

2. **`flattenInlined` in macros** — The macros now flatten nested `Inlined` nodes from subexpressions by extracting val bindings into a flat `Block`.

3. **`ascribeWidenedType`** — Adds a `Typed` node with `.dealias` type to prevent unreduced type projections (`ExactOp2Aux[...].Out`) from leaking into error messages.

4. **Non-transparent Check/CTName/DualSummonTrapError givens** — Removed unnecessary `transparent` from `inline given` instances that don't need type narrowing.

5. **`FlattenInlinedPhase` compiler plugin** — Runs between `typer` and `posttyper`. Flattens nested `Inlined` trees and pre-minimizes the `call` field of `Inlined` nodes to a simple `Ident` pointing to the top-level class. This prevents PostTyper from re-transforming complex call trees (which contain TypeApply with heavy type arguments) for every `Inlined` node. This is the single biggest compile time fix — it reduced 8-addition chains from 33s to 2s and eliminated the exponential growth entirely (20 additions: previously impossible, now 7s).

## Key Files

| File | Role |
|---|---|
| `internals/src/main/scala/dfhdl/internals/Exact.scala` | ExactOp1/2/3 macros, cleanTypeHack, flattenInlined |
| `internals/src/main/scala/dfhdl/internals/Checked.scala` | Check1/Check2 constraint macros |
| `plugin/src/main/scala/plugin/FlattenInlinedPhase.scala` | Compiler plugin phase for tree optimization |
| `plugin/src/main/scala/plugin/Plugin.scala` | Plugin registration |

## Profiling Results

The `posttyper` phase is the main bottleneck. Use `-Yprofile-enabled` to measure.

### Compiler Phase Breakdown (10-operand `a + b + c + ... + j`)
- `typer`: ~1-2s (macros run here, very fast)
- `posttyper`: 20-70s depending on optimizations (the bottleneck)
- `inlining`: <1s
- All other phases: <1s each

### What PostTyper Does

PostTyper performs post-typing checks on inline-expanded code:
- Overriding checks
- Type bound verification
- Import resolution
- Re-traverses the expanded tree multiple times

### What Drives PostTyper Time

**The `call` field of `Inlined` nodes** — PostTyper's `Inlined` handler calls `withMode(Mode.NoInline)(transform(call))` on every Inlined node. The result is then immediately discarded and replaced by `inlineCallTrace(call.symbol, pos)`, which is just an `Ident` to the top-level class. With chained inlines producing 100+ Inlined nodes, each with a complex call tree containing TypeApply, PostTyper re-transforms all of these call trees for nothing. Pre-minimizing the call to the same Ident that PostTyper would produce eliminates this entirely.

**NOT type lambda complexity** — replacing complex `[LS, RS] =>> LS || !RS` with trivial `[LS, RS] =>> Boolean` had zero impact on posttyper time.

**NOT node count alone** — removing 200 nodes from a 2300-node tree gave marginal improvement.

**NOT InferredTypeTree spans** — PostTyper's `checkInferredWellFormed` gates on `span.isZeroExtent` for TypeTree nodes. Marking all 457 zero-extent TypeTrees with non-zero spans had zero impact — this check is not the bottleneck.

**NOT TypeApply type arg dealiasing** — dealiasing all TypeApply type arguments had negligible impact. The types are already small individually; the issue was the call-tree re-transformation, not type complexity.

### What We Tried That Didn't Work

1. **Replacing TypeLambda args with `Any`** — Violates HK upper bounds (`[T <: Int] =>> Boolean` can't accept `Any`).

2. **Replacing TypeLambda args with `Nothing`** — Crashes Check macro when it tries to interpret `Nothing` as a lambda body for runtime checks.

3. **Replacing TypeLambda args with upper-bound lambdas** — Works for compile-time checks but breaks `CheckNUBLP` runtime fallback in `lib`.

4. **Simplifying `asInstanceOf` type args** — The `CheckOK.asInstanceOf[Check[...]]` pattern doesn't appear in trees because Check givens are now non-transparent (the `asInstanceOf` is inside the macro, not in expanded code).

5. **Replacing `CheckNUB.ok[...](...)(...)` with `CheckNUBOK`** — Pattern matching failed because these calls are inside nested Inlined nodes that the TreeMap doesn't visit in the right order.

6. **Removing Check type signatures from trees** — User confirmed no performance impact. The Check types are not the bottleneck.

### Resolved

The compile time issue is resolved. The `minimizeCall` optimization in `FlattenInlinedPhase` eliminates the exponential growth in PostTyper. This is also a general Scala 3 compiler bug — PostTyper's `Inlined` handler re-transforms call trees whose results are immediately discarded. A minimal reproducer and upstream patch should be filed against the Scala 3 compiler.

## Checked.scala Architecture

`Check1` / `Check2` are type-level constraint systems:
- **Check**: Trait with `Cond` and `Msg` type lambdas, macro-generated
- **CheckNUB**: "Not Upper Bounded" variant, handles unknown types at compile time
- **CheckNUB.ok**: Fast path when condition is statically true (returns `CheckNUBOK`)
- **CheckNUBLP**: Fallback when condition needs runtime evaluation (summons `Check` which triggers the macro)
- **CheckOK / CheckNUBOK**: Singleton no-op objects

The macro (`checkMacro`) checks `condOpt`:
- `Some(true)` → returns `CheckOK.asInstanceOf[Check[...]]` (no runtime check needed)
- `Some(false)` → compile error
- `None` → generates runtime check code (`if (!cond) throw ...`)
19 changes: 12 additions & 7 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,17 +16,21 @@ Outputs: Verilog, SystemVerilog, VHDL.

## Build System

**Tool**: SBT 1.12.2 — **Scala**: 3.8.1 (nightly resolver enabled)
**Tool**: SBT 1.12.9 — **Scala**: 3.8.3 (nightly resolver enabled)

```bash
sbt compile # compile all subprojects
sbt test # run all unit tests
sbt testApps # run simulation/app tests (requires OSS CAD tools)
sbt quickTestSetup # limit test scope to lib/Playground.scala only (fast iteration)
sbt clearSandbox # delete sandbox/ directory
sbt docExamplesRefUpdate # copy generated HDL from sandbox/ to lib/src/test/resources/ref/
sbtn compile # compile all subprojects
sbtn Test/compile # compile all tests (separate from running them)
sbtn test # run all unit tests
sbtn testApps # run simulation/app tests (requires OSS CAD tools)
sbtn corePlayground # limit test scope to core/Playground.scala only (fast iteration)
sbtn libPlayground # limit test scope to lib/Playground.scala only (fast iteration)
sbtn clearSandbox # delete sandbox/ directory
sbtn docExamplesRefUpdate # copy generated HDL from sandbox/ to lib/src/test/resources/ref/
```

Always use `sbtn` (sbt client) instead of `sbt` for faster startup. On Windows use `sbtn.bat`.

## Subproject Structure

Dependencies flow left to right:
Expand Down Expand Up @@ -109,6 +113,7 @@ CI installs these via OSS CAD Suite:
## Claude Instructions

- When asked to **create a new compiler stage** or **modify an existing compiler stage**, always invoke the `/new-stage` skill before doing any work.
- When working on **compile time performance**, invoke the `/compile-perf` skill to review the methodology, known bottlenecks, and what has already been tried.

## Licenses

Expand Down
5 changes: 3 additions & 2 deletions build.sbt
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
commands += DFHDLCommands.quickTestSetup
commands += DFHDLCommands.libPlayground
commands += DFHDLCommands.corePlayground
commands += DFHDLCommands.clearSandbox
commands += DFHDLCommands.testApps
commands += DFHDLCommands.docExamplesRefUpdate

// format: off
val projectName = "dfhdl"
val compilerVersion = "3.8.1"
val compilerVersion = "3.8.3"

inThisBuild(
List(
Expand Down
6 changes: 6 additions & 0 deletions compiler/ir/src/main/scala/dfhdl/compiler/ir/DFMember.scala
Original file line number Diff line number Diff line change
Expand Up @@ -582,6 +582,12 @@ object DFVal:
def hasNonBubbleInit(using MemberGetSet): Boolean = dcl.initRefList match
case DFRef(dfVal) :: _ => !dfVal.isBubble
case _ => false
// Funcs with associative ops (&, |, ^, ++) may have more than 2 args when consecutive
// same-op anonymous Funcs are merged during elaboration (e.g., `a ^ b ^ c` produces
// a single `Func(^, [a, b, c])` instead of nested binary Funcs).
// +, -, * are NOT merged because carry promotion assumes binary (2-arg) Funcs.
// ++ is only merged for flat DFBits concatenation, not struct/vector/string.
// The position spans from the first operand to the last in the merged chain.
final case class Func(
dfType: DFType,
op: Func.Op,
Expand Down
2 changes: 2 additions & 0 deletions compiler/ir/src/main/scala/dfhdl/compiler/ir/DFTags.scala
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ case object ExtendTag extends DFTag
type ExtendTag = ExtendTag.type
case object TruncateTag extends DFTag
type TruncateTag = TruncateTag.type
case object ImplicitlyFromIntTag extends DFTag
type ImplicitlyFromIntTag = ImplicitlyFromIntTag.type
case class DFHDLVersionTag(version: String) extends DFTag

opaque type DFTags = Map[String, DFTag]
Expand Down
9 changes: 5 additions & 4 deletions compiler/ir/src/main/scala/dfhdl/compiler/ir/DFType.scala
Original file line number Diff line number Diff line change
Expand Up @@ -249,11 +249,12 @@ final case class DFDecimal(
end DFDecimal

object DFDecimal extends DFType.Companion[DFDecimal, Option[BigInt]]:
enum NativeType derives CanEqual, ReadWriter:
case BitAccurate, Int32
type NativeType = Boolean
object NativeType:
type BitAccurate = BitAccurate.type
type Int32 = Int32.type
type BitAccurate = false
type Int32 = true
val BitAccurate: false = false
val Int32: true = true
end DFDecimal

import DFDecimal.NativeType
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,14 @@ private abstract class NamedAliases extends Stage:
.filterNot(_.isAllowedMultipleReferences)
// tuple with the suggested name
.map(m => (m, m.suggestName.getOrElse("anon")))
// group dfhdl-equivalent values, as long as they are in the same scope
// group dfhdl-equivalent values, as long as they are in the same scope.
// conditional headers are excluded from grouping because their =~ comparison
// does not account for block contents (conditions/branches), so structurally
// different conditionals could be incorrectly merged.
.groupByCompare(
(l, r) => l._1 =~ r._1 && l._1.isInsideOwner(r._1.getOwner),
(l, r) =>
!l._1.isInstanceOf[DFConditional.Header] &&
l._1 =~ r._1 && l._1.isInsideOwner(r._1.getOwner),
_._2.hashCode()
)
// split to list of aliases and list of suggested names for each group
Expand Down Expand Up @@ -112,7 +117,8 @@ case object NamedVerilogSelection extends NamedAliases:
case alias: DFVal.Alias.ApplyIdx =>
List(alias.relValRef.get)
case func @ DFVal.Func(op = op, args = DFRef(lhs) :: _ :: Nil)
if !lhs.hasVerilogName && carryOps.contains(op) && func.width > lhs.width =>
if isBasicVerilog && !lhs.hasVerilogName && carryOps.contains(op) &&
func.width > lhs.width =>
List(lhs)
case func: DFVal.Func =>
func.getReadDeps.headOption match
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -239,7 +239,7 @@ class DropStructsVecsSpec extends StageSpec(stageCreatesUnrefAnons = true):
| val v2 = Bits(32) X 4 <> VAR init arg2.as(Bits(32) X 4)
| val v3 = Bits(8) X 4 <> VAR.REG
| val sel = UInt(2) <> IN
| o := ((arg(31, 24) ^ arg(23, 16)) ^ v(sel.toInt)) ^ v3(sel.toInt)
| o := arg(31, 24) ^ arg(23, 16) ^ v(sel.toInt) ^ v3(sel.toInt)
| o2 := arg2(127, 96)
| v3(sel.toInt).din := v(sel.toInt)
| o3 := v2(sel.toInt)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -263,4 +263,27 @@ class ExplicitNamedVarsSpec extends StageSpec:
|end ID""".stripMargin
)
}

// TODO: this causes an exception
// test("AES regression test") {
// case class AESByte() extends Opaque(Bits(8))

// class xtime extends DFDesign:
// val lhs = AESByte <> IN
// val o = AESByte <> OUT
// val shifted = lhs.bits << 1
// val anon: Bits[8] <> VAL =
// if (lhs.bits(7))
// val o: Bits[8] <> CONST = h"1b"
// shifted ^ o
// else shifted
// o <> anon.as(AESByte)
// end xtime

// val top = (new xtime).explicitNamedVars
// assertCodeString(
// top,
// """|""".stripMargin
// )
// }
end ExplicitNamedVarsSpec
Loading
Loading