diff --git a/docs/array-design.md b/docs/array-design.md index e7e6bfbca..ee426c518 100644 --- a/docs/array-design.md +++ b/docs/array-design.md @@ -105,7 +105,7 @@ Indexed dimensions (e.g., numeric dimensions like `Periods(5)`) can use position - All expression types: `StaticSubscript`, `TempArray`, `TempArrayElement`, `AssignTemp` #### Bytecode VM -- View stack operations: `PushVarView`, `PushTempView`, `PushStaticView` +- View stack operations: `PushStaticView`, `PushTempView`, `PushVarViewDirect` - View manipulation: `ViewSubscriptConst/Dynamic`, `ViewRange/Dynamic`, `ViewStarRange`, `ViewWildcard`, `ViewTranspose` - Iteration: `BeginIter`, `LoadIterElement`, `StoreIterElement`, `NextIterOrJump`, `EndIter` - Reductions: `ArraySum`, `ArrayMax`, `ArrayMin`, `ArrayMean`, `ArrayStddev`, `ArraySize` diff --git a/docs/design-plans/2026-03-06-finish-salsa-migration.md b/docs/design-plans/2026-03-06-finish-salsa-migration.md index 2061eb106..eedcfb57b 100644 --- a/docs/design-plans/2026-03-06-finish-salsa-migration.md +++ b/docs/design-plans/2026-03-06-finish-salsa-migration.md @@ -62,6 +62,20 @@ only re-parse variables that actually reference the affected dimensions. - **finish-salsa-migration.AC3.2 Success:** `PREVIOUS(x)` where `x = SMTH1(input, 1)` compiles to module expansion (not `LoadPrev`) through the salsa incremental path. - **finish-salsa-migration.AC3.3 Success:** Editing an unrelated variable does not trigger re-parse of variables in the same model (salsa cache stability preserved). +> **Annotation (2026-07-26, GH #568).** This document is a record of what was +> decided in March, not a live contract; it is annotated rather than rewritten. +> AC4.3, AC4.5 and AC5.2 have been *partly* overtaken and must not be actioned +> as written. `all_deps` (and `direct_deps`/`module_deps`/`module_output_deps`) +> is gone, but `set_dependencies` deliberately REMAINS as a thin adapter that +> reads `db::dep_graph::model_dependency_graph` -- it is the monolithic +> `compiler::Module`'s only source of runlists, and that monolith is the live +> differential oracle for the run-invariance classifier +> (`db::invariance::salsa_and_monolithic_paths_agree`). Deleting it per AC4.3 +> costs that oracle. AC4.5 is answered in `docs/tech-debt.md` item 17: the +> `Variable::errors`/`unit_errors` fields are a LIVE error channel the salsa +> path reads, not legacy residue, and removing them without first replacing the +> channel silently drops diagnostics. + ### finish-salsa-migration.AC4: Monolithic compilation path removed - **finish-salsa-migration.AC4.1 Success:** `compile_project` (free function in interpreter.rs) does not exist. - **finish-salsa-migration.AC4.2 Success:** `Simulation::compile()` does not exist. diff --git a/docs/design/engine-performance.md b/docs/design/engine-performance.md index 280c320fc..15d8c5e25 100644 --- a/docs/design/engine-performance.md +++ b/docs/design/engine-performance.md @@ -202,12 +202,14 @@ Op2`, 2→1). A leaf *assignment* `dst = a op b` keeps the existing (`BinOpAssignCurr` ≪ `Op2`). **Where it runs.** A late `ByteCode::fuse_three_address` pass applied to the Vm's -flow/stock execution bytecode at `Vm::new`, reusing `peephole_optimize`'s +flow/stock execution bytecode at `Vm::new`, reusing the symbolic peephole's jump-target guard + old→new PC remap and preserving `max_stack_depth`. It runs at -`Vm::new` rather than compile time deliberately: the `CompiledSimulation` stays a -pure, *symbolizable*, salsa-cached artifact (the symbolic roundtrip tests -symbolize it; the fused opcodes have no symbolic form), and the `Vm`'s execution -copy is where the optimization lives. Per-`Vm` fusion is a linear scan, negligible +`Vm::new` rather than compile time deliberately: the fused opcodes have no +symbolic form (`SymbolicOpcode` deliberately has no 3-address variants), so +running the pass earlier would produce bytecode the salsa-cached artifact cannot +represent -- and the `CompiledSimulation` must stay the pure resolution of the +cached symbolic fragments. The `Vm`'s private execution copy is where the +optimization lives. Per-`Vm` fusion is a linear scan, negligible vs a run. Initials are left unfused (run once; `extract_assign_curr_offsets` reads their `AssignCurr` targets). @@ -330,9 +332,12 @@ only conclusive for effects that exceed ~4%. ### R4. `RuntimeView` allocation + `flat_offset` (~20% of post-win run) -`PushVarView`/`PushTempView` rebuild `SmallVec`s (dims, strides, dim_ids) on every -execution; `flat_offset` (10.3%) recomputes row-major offsets per element. For -arrayed models this is now the #2 run cost. +`PushTempView`/`PushVarViewDirect` rebuild `SmallVec`s (dims, strides, dim_ids) +on every execution; `flat_offset` (10.3%) recomputes row-major offsets per +element. For arrayed models this is now the #2 run cost. (This item was written +when a third opcode, `PushVarView`, shared the cost; it was deleted in GH #964's +stage 2 as unemittable by codegen, which is part (a) of the proposal below +already realized for the whole-array case.) Proposal: (a) push more views through the compile-time `PushStaticView` path (precomputed `StaticArrayView`) and store dynamic view descriptors in the @@ -400,8 +405,9 @@ re-derivation. (b) is broader but touches many call sites. 2. ~~**R1 (bounds-check elimination)**~~ — INVESTIGATED, dropped: measured sub-noise (~0) ceiling; bounds checks are effectively free at opt-level=3. 3. ~~**R2 (3-address binop fusion)**~~ — DONE. Flow opcodes −23.5%, run −6.8% on - C-LEARN; a late `fuse_three_address` pass at Vm::new (the `CompiledSimulation` - stays symbolizable). A full register VM would cut more but is a large rewrite. + C-LEARN; a late `fuse_three_address` pass at Vm::new (the fused opcodes have no + symbolic form, so they must not exist before assembly). A full register VM would + cut more but is a large rewrite. 4. ~~**R4 (RuntimeView)**~~ — largely DONE via round 2's `dense_linear_start` fast paths (`flat_offset` 8.2% -> ~4% of a smaller run); the residual is the strict-slice `vector_elm_map` base and `offset_for_iter_index`'s diff --git a/docs/tech-debt.md b/docs/tech-debt.md index ec3ada721..28a35f12c 100644 --- a/docs/tech-debt.md +++ b/docs/tech-debt.md @@ -143,13 +143,19 @@ Known debt items consolidated from CLAUDE.md files and codebase analysis. Each e - **Owner**: unassigned - **Last reviewed**: 2026-02-15 -### 17. Remove Legacy Error Fields from Variable/ModelStage Types +### 17. Embedded Error Fields on Variable/ModelStage Types - **Component**: simlin-engine -- **Severity**: low -- **Description**: The `errors` and `unit_errors` fields on `Variable`, and the `errors` field on `ModelStage0`/`ModelStage1`, are now redundant with the salsa incremental compilation pipeline. Diagnostics are collected via `collect_all_diagnostics` / `collect_model_diagnostics` from tracked functions, making the embedded error fields dead weight carried through the monolithic compilation path. Removing them would simplify the data model and reduce confusion about the source of truth for errors. This cleanup was identified as Step 13 in the incremental compilation design (`docs/design/incremental-compilation.md`) but is not required by any acceptance criterion. +- **Severity**: RESOLVED (2026-07-26) -- **the original premise was half wrong; do not act on it** +- **Description**: This entry used to claim that the `errors`/`unit_errors` fields on `Variable` and the `errors` field on `ModelStage0`/`ModelStage1` were all "redundant with the salsa incremental compilation pipeline ... dead weight carried through the monolithic compilation path", and that removing them would simplify the data model. Every read site was classified before acting, and the four fields fall into two very different groups. + + **`Variable::errors` and `Variable::unit_errors` are LIVE error channels, not residue.** They are how parsing and lowering report a failure on a variable, and the salsa diagnostics are *downstream* of them: `db::var_fragment::lower_var_fragment` reads `parsed.variable.unit_errors()` (the malformed-``-string rows), `parsed.variable.equation_errors()` (parse errors, where the conveyor/queue driven-flow `EmptyEquation` suppression applies) and `lowered.equation_errors()` (`MismatchedDimensions` and everything else `lower_ast` raises), and turns each entry into a `Diagnostic`. Emptying the lowered read makes every dimension-mismatch diagnostic vanish; emptying the parsed read turns a spec-sanctioned empty equation on a conveyor-driven flow into a phantom error. Deleting these fields without first replacing the channel would silently drop diagnostics. The claim is now pinned as a test rather than left as prose: `db::diagnostic_tests::variable_error_fields_are_the_lowering_channel` asserts both that the stage's value carries the error in the field and that the matching diagnostic reaches `collect_all_diagnostics`. + + **`ModelStage0::errors` and `ModelStage1::errors` are read only by test-only code**, which is the half of the claim that held. `ModelStage0::errors` carries the duplicate-canonical-ident collision (GH #891) and is read only by `ModelStage1::new`; `ModelStage1::errors` is the monolithic path's simulatability gate (`compiler::Module::new` refuses a model with a non-empty list) and is otherwise read only by tests. They are kept and documented as such on the types: the gate is the reason they exist, `compiler::Module::new` has no salsa database to consult instead, and it is the only thing stopping that path from mis-compiling a resolved recurrence SCC (see `project::tests::build_module_refuses_a_resolved_recurrence_scc`). What WAS dead has been deleted -- `Variable::push_error` and `ModelStage1::get_unit_errors`, which existed for the mutation-based error collection the second dependency walk did before GH #568 unified the cycle gate, plus `Variable::push_unit_error` and the write-only `ModelStage1::unit_warnings`, which had no caller and no non-`None` writer respectively even before that. + + Test helpers that used to report the monolithic path's embedded errors (`test_common::TestProject::compile`/`build_module`) now report `collect_all_diagnostics` instead: the production source of truth, and a different and better set rather than a strict superset. It gains the unknown-dependency, bare-lookup-table and assembly diagnostics and a source location on every row; it loses the model-level `VariablesHaveErrors` roll-up, the `("project", UnitDefinitionErrors)` row, and the old path's cycle verdict on element-acyclic recurrences -- the last being the whole point of deleting it. - **Owner**: unassigned -- **Last reviewed**: 2026-02-22 +- **Last reviewed**: 2026-07-26 ### 18. Dimension-Granularity Incremental Invalidation diff --git a/scripts/check-docs.py b/scripts/check-docs.py index 4a5f09dee..ea75433e9 100755 --- a/scripts/check-docs.py +++ b/scripts/check-docs.py @@ -137,11 +137,15 @@ def main() -> int: # All CLAUDE.md files for claude_md in repo_root.rglob("CLAUDE.md"): - # Skip generated/noise directories + # Skip generated/noise directories. `.claude-scratch` is git-excluded + # agent scratch space INSIDE the repo, so a working copy of a CLAUDE.md + # routinely lands there mid-task; walking it reports every relative path + # in that copy as broken (~150 errors) and fails the pre-commit hook on + # a tree that is actually fine. rel = claude_md.relative_to(repo_root) parts = rel.parts if any(p in ("node_modules", "target", "build", "lib", "lib.browser", "lib.module", - "third_party") + "third_party", ".claude-scratch") for p in parts): continue files_to_check.append(claude_md) diff --git a/src/simlin-engine/CLAUDE.md b/src/simlin-engine/CLAUDE.md index b95345be8..73eb2dcb1 100644 --- a/src/simlin-engine/CLAUDE.md +++ b/src/simlin-engine/CLAUDE.md @@ -17,13 +17,13 @@ Equation text flows through these stages in order: - `context.rs` - Symbol tables and variable metadata; `lower_preserving_dimensions()` skips Pass 1 dimension resolution to keep full array views for array-producing builtins. Handles `@N` position syntax resolution: in scalar context (no active A2A dimension, not inside an array-reducing builtin), `DimPosition(@N)` resolves to a concrete element offset; inside array-reducing builtins (`preserve_wildcards_for_iteration`), dimension views are preserved for iteration. Two wildcard-preservation contexts: `with_preserved_wildcards()` for reducers (SUM, MEAN, etc.) where `ActiveDimRef` resolves to a concrete offset, and `with_vector_builtin_wildcards()` for array-producing builtins (VectorSortOrder, VectorElmMap, etc.) where `ActiveDimRef` is promoted to `Wildcard` to preserve the full array view - `expr.rs` - Expression compilation - `invariance.rs` - Run-invariance classification (time-invariant hoisting, GH #712, stage B1). The pure **functional core** `exprs_are_invariant(exprs, classify_offset)` walks a variable's lowered `Vec` and returns whether it is run-invariant (same value every timestep): exhaustive + default-variant over every `Expr`/`BuiltinFn` variant, parameterized by an offset-classification callback that resolves a referenced slot to its owner's verdict (`OffsetClass::{Invariant, Variant}`). Invariant = literals/`Dt`/`TimeStep`/`StartTime`/`FinalTime`/`Pi`/`Inf`, `Init(_)` of any var (init buffer frozen), GF lookups with invariant indices, pure builtins/reducers/array-ops of invariant args, and other invariant vars by offset. Variant (default) = `Time`/`Pulse`/`Ramp`/`Step`/`Previous`/`ModuleInput`/`EvalModule`, stocks, module-instance slots. The two compile paths (`compiler::Module` monolithic, test-only; salsa `db::invariance`) feed it different callbacks but share this walk, so they classify identically -- pinned by `db::invariance`'s `salsa_and_monolithic_paths_agree`. See [the design note](/docs/design-plans/2026-06-04-time-invariant-hoisting.md). - - `codegen.rs` - Bytecode emission; routes array-producing builtins through dedicated opcodes instead of element-wise iteration. `emit_array_reduce()` is the shared helper for single-argument array builtins (SUM, SIZE, STDDEV, MIN, MAX, MEAN): pushes view, emits reduction opcode, pops view + - `codegen.rs` - Bytecode emission; routes array-producing builtins through dedicated opcodes instead of element-wise iteration. Codegen emits **symbolic** opcodes: a variable operand is a `compiler::VarRef` (canonical name + element offset) copied straight out of the lowered `Expr`, never a slot, so a compiled fragment says nothing about where its variables live and `symbolic::resolve_module` assigns every address exactly once, at assembly. `ModuleCtx` is the compiler's WHOLE input contract -- a borrowed struct holding exactly what `Compiler` reads (`ident`, `inputs`, `temp_sizes`, the three runlists, `var_sizes`, `tables`, `dimensions`, `dimensions_ctx`) and nothing more; there is deliberately no offset map and no slot count, because the only thing codegen still needs from the symbol table is a variable's EXTENT (`full_source_len`, the VECTOR ELM MAP bound). `VarSizes` is keyed by the whole `VarRef`, not by name, and `context::whole_variable_extents` is its only constructor -- shared with lowering, which reads the same table through `ContextCore::var_sizes` for the GH #578 constant-offset ELM MAP fold, so the fold and the opcode cannot disagree about where a source's storage ends. Keying by reference is what makes a CROSS-MODULE source right: `m·x` lowers to `VarRef { name: m, element_offset: x's slot inside the instance }`, so a name-keyed lookup answered with the module instance's whole block size, and reads past `x`'s end silently landed on the next sub-model variable instead of yielding `:NA:`. A module instance therefore contributes no entry of its own and one per sub-model variable at that variable's slot, recursively through nested instances (`array_tests::cross_module_array_reference_tests`). Both callers build one: the `#[cfg(test)]` monolithic `Module::compile` borrows every field off its owned `Module` and then resolves the emitted symbolic module against its own layout, and the production per-variable fragment compiler borrows the salsa-cached project-global dimension context/converted dims and keeps the fragment symbolic until assembly. The `#[cfg(test)]` gate on `Module` is a load-bearing assertion, not tidiness: it makes "no production `compiler::Module` literal remains" (GH #964) a compile error rather than a claim. There is no un-fused `Opcode::AssignNext`: a stock update is emitted straight as the fused `BinOpAssignNext`, and `Var::new`'s `check_stock_updates_are_emittable` rejects any update whose lowered form does not end in an `Op2` with a per-variable `NotSimulatable` -- so an eventual `non_negative` (GH #545) clamp that wrapped the update in `MAX` surfaces as a diagnostic instead of a stock that silently never integrates. `emit_array_reduce()` is the shared helper for single-argument array builtins (SUM, SIZE, STDDEV, MIN, MAX, MEAN): pushes view, emits reduction opcode, pops view - `dimensions.rs` - Dimension checking/inference - `subscript.rs` - Array subscript expansion and iteration - `pretty.rs` - Debug pretty-printing - - `symbolic.rs` - Layout-independent symbolic bytecode layer (the backbone of incremental compilation): opcodes reference variables by name (`SymVarRef { name, element_offset }`) rather than model-global offset, so salsa caches a per-variable `PerVarBytecodes` fragment that survives variable add/remove. Pipeline: concrete bytecode → `symbolize_*` → `SymbolicByteCode` → `resolve` → concrete bytecode. `FragmentMerger` (with `TempStrategy { Recycle, Sum }`) is the shared core of `concatenate_fragments_with_gf` (plain sequential phase concat -- `Recycle` max-merges temps to match the monolithic `Module::compile`) and `combine_scc_fragment` (the interleaved multi-member SCC fragment -- `Sum` gives each member a disjoint temp range since segments overlap). `GfDedup` content-de-duplicates graphical-function blocks across fragments (#582), so a dependency arrayed GF re-extracted by N consumers is laid out once and every consumer's `base_gf` is remapped to it. -6. **`src/bytecode.rs`** - Instruction set definition, opcodes, type aliases (`LiteralId`, `ModuleId`, `DimId`, `TempId`, etc.). Includes `LoadPrev`/`LoadInitial` opcodes for `PREVIOUS()`/`INIT()` intrinsics, the scalar `Lookup` opcode plus `LookupArray` (per-element arrayed graphical function `g[D!](index)`: pops the shared scalar index, reads the arrayed GF's full storage view, evaluates `graphical_functions[base_gf + i]` per element into a temp array view -- so a wrapping reducer / vector op such as VECTOR SELECT applies over the arrayed-GF result; out-of-range element ⇒ NaN like scalar `Lookup`; GH #580), and vector operation opcodes (`VectorSelect`, `VectorElmMap`, `VectorSortOrder`, `Rank`, `AllocateAvailable`, `AllocateByPriority`) that operate on view-stack arrays and write results to temp storage. -7. **`src/vm.rs`** - Stack-based bytecode VM. Hot loop uses proven-safe unchecked array access validated at compile time by `ByteCodeBuilder`. Maintains `prev_values` and `initial_values` snapshot buffers for `LoadPrev`/`LoadInitial` opcodes. Implements vector operation dispatch (VectorSelect, VectorElmMap, VectorSortOrder, Rank, AllocateAvailable, AllocateByPriority) and the per-element arrayed-GF `LookupArray`. `Opcode::VectorElmMap` and `Opcode::VectorSortOrder` dispatch into the sibling helper modules below (extracted purely for the per-file line cap). Array reducers (ArrayMax, ArrayMin, ArrayMean, ArrayStddev) return NaN for empty views; ArraySum returns 0.0 (additive identity). When conveyor/queue plans are attached, the VM runs the special-stock side-table passes each step (`init_belts` / `publish_container_values` / `run_coupled_passes`); `get_value` mid-run therefore previews the resting `curr` on CLONED side tables rather than reading a pass-driven flow's placeholder zero. The reset/`run_to`/constant-override tests live in the sibling **`vm_reset_run_to_and_constants_tests.rs`** (split out for the per-file line cap). + - `symbolic.rs` - Layout-independent symbolic bytecode layer (the backbone of incremental compilation): opcodes reference variables by name (`SymVarRef`, an alias for `compiler::VarRef { name: Ident, element_offset }`) rather than model-global offset, so salsa caches a per-variable `PerVarBytecodes` fragment that survives variable add/remove. Pipeline: lowered `Expr` (names) → codegen → `SymbolicByteCode` → `resolve` → concrete bytecode. Addresses travel in ONE direction and are assigned ONCE: there is no symbolization pass, no reverse offset map, and no per-fragment layout (GH #964). The bytecode builder and its peephole optimizer live here too (`SymbolicByteCodeBuilder`, tested in the sibling `symbolic_builder_tests.rs`): both are address-independent, so running them before resolution keeps `resolve_bytecode` a strict 1:1 mapping -- which the run-invariant flow-prefix boundary and the SCC per-element segmentation both depend on. `resolve_bytecode` is the SOLE producer of concrete bytecode and therefore where the VM's fixed-stack safety proof is discharged (a program deeper than `STACK_CAPACITY` is a compile error, not an abort). Sixteen `SymbolicOpcode` variants carry `#[allow(dead_code)]`: codegen constructs none of them, which the dead-code lint now proves (codegen is the only producer of a `SymbolicOpcode`); they are the superseded halves of incremental view-stack construction and broadcast iteration, and retiring them plus their `Opcode` twins, VM arms and wasm arms is sequenced as its own change. `FragmentMerger` is the shared core of `concatenate_fragments_with_gf` (the sequential phase concat) and `combine_scc_fragment` (the interleaved multi-member SCC fragment). Its contract is written out on the type as eight obligations M1-M8, stated as properties of the MERGED FRAGMENT rather than as parity with any other compiler -- referential integrity (a renumbered opcode names the same resource value it named in its own fragment), disjointness and tiling of the flat resources, id-type capacity, GF sharing by content only, temp non-aliasing, 1:1 opcode preservation, `SymVarRef`s untouched, and agreement between the per-phase renumber and the all-phases merge. M3 (id-type capacity) is about ASSIGNED ids, so the counts that become a later phase's base are carried in `usize` and the `u16` bound is discharged in exactly one function, `resource_base` -- the only point that sees both a base and the length of the fragment about to consume it. Stating it per place instead left the three places disagreeing: a table of exactly 65,536 entries is addressable and the merger accepted it, while the cross-phase and initials-phase narrowings rejected it, so a model whose every id was valid failed to assemble. "Assigned" also means assigned to something the module KEEPS, which is why the all-phases aggregation of the shared context tables is `merge_context_side_channels` rather than a full merge: each phase retains its own literal pool (every compiled initial keeps one; flows and stocks keep one each), so an aggregate pool over all three is discarded -- and bounding it failed assembly for models around 33k scalar stocks whose every retained pool was well inside the limit. Each names the test that pins it; the property tests are `src/compiler/symbolic_merge_proptest.rs` (sequential) and `src/db/combined_fragment_proptest.rs` (interleaved). `TempStrategy { Recycle, Sum }` is how a caller DECLARES its emission shape: `Recycle` collapses temps by identity (safe because the sequential concat emits each fragment as one contiguous run, and necessary because summing 0-based per-fragment counts overflows the `u8` `TempId`, #583), `Sum` gives each fragment a disjoint range (required because the SCC interleave makes members' live ranges overlap). `GfDedup` content-de-duplicates graphical-function blocks across fragments (#582), so a dependency arrayed GF re-extracted by N consumers is laid out once and every consumer's `base_gf` is remapped to it. +6. **`src/bytecode.rs`** - Instruction set definition, opcodes, type aliases (`LiteralId`, `ModuleId`, `DimId`, `TempId`, etc.). Includes `LoadPrev`/`LoadInitial` opcodes for `PREVIOUS()`/`INIT()` intrinsics, the scalar `Lookup` opcode plus `LookupArray` (per-element arrayed graphical function `g[D!](index)`: pops the shared scalar index, reads the arrayed GF's full storage view, evaluates `graphical_functions[base_gf + i]` per element into a temp array view -- so a wrapping reducer / vector op such as VECTOR SELECT applies over the arrayed-GF result; out-of-range element ⇒ NaN like scalar `Lookup`; GH #580), and vector operation opcodes (`VectorSelect`, `VectorElmMap`, `VectorSortOrder`, `Rank`, `AllocateAvailable`, `AllocateByPriority`) that operate on view-stack arrays and write results to temp storage. There is deliberately no un-fused `AssignNext` and no `PushVarView`: codegen cannot emit either (a stock update always ends in an `Op2`, so it is fused into `BinOpAssignNext` by `symbolic::SymbolicByteCodeBuilder::fuse_trailing_op2_into_assign_next` at emit time; a full-array variable view is always a `PushStaticView` or `PushVarViewDirect`), and carrying them cost a VM arm, a wasm arm, and two `SymbolicOpcode` variants apiece for nothing. +7. **`src/vm.rs`** - Stack-based bytecode VM. Hot loop uses proven-safe unchecked array access validated at compile time by `compiler::symbolic::resolve_bytecode` -- the single place concrete bytecode is produced, so nothing reaches the VM unchecked. Both failure modes (over-depth, and a `stack_effect` metadata underflow) are reported as a compile `Err`, so an unprovable program is rejected rather than executed. Maintains `prev_values` and `initial_values` snapshot buffers for `LoadPrev`/`LoadInitial` opcodes. Implements vector operation dispatch (VectorSelect, VectorElmMap, VectorSortOrder, Rank, AllocateAvailable, AllocateByPriority) and the per-element arrayed-GF `LookupArray`. `Opcode::VectorElmMap` and `Opcode::VectorSortOrder` dispatch into the sibling helper modules below (extracted purely for the per-file line cap). Array reducers (ArrayMax, ArrayMin, ArrayMean, ArrayStddev) return NaN for empty views; ArraySum returns 0.0 (additive identity). When conveyor/queue plans are attached, the VM runs the special-stock side-table passes each step (`init_belts` / `publish_container_values` / `run_coupled_passes`); `get_value` mid-run therefore previews the resting `curr` on CLONED side tables rather than reading a pass-driven flow's placeholder zero. The reset/`run_to`/constant-override tests live in the sibling **`vm_reset_run_to_and_constants_tests.rs`** (split out for the per-file line cap). - **`src/vm_vector_sort_order.rs`** - Genuine-Vensim VECTOR SORT ORDER. Ranks WITHIN each currently-iterated source slice (the innermost/last-declared dim is the sorted axis; outer dims select independent rows), 0-based: result position `j` of a row holds the 0-based source index *within that row* of its `j`-th element in sorted order (`direction == 1` ascending, else descending; stable ties). A 1-D view is the degenerate single-row case (in-row ranks == whole-view ranks). The prior whole-flattened-view absolute-index behavior (GH #585) made a multi-row source feed out-of-range flat indices into a downstream single-column ELM MAP; ground truth is real Vensim DSS `/test/test-models/tests/vector_order/output.tab` (ranks include `0`, impossible for a 1-based permutation). RANK is a distinct, correctly 1-based opcode. - **`src/vm_vector_elm_map.rs`** - Genuine-Vensim VECTOR ELM MAP: result element `i` = `source[base_i + round(offset[i])]` over the source variable's FULL row-major contiguous storage, where `base_i` is the flat position arg-1's element reference establishes and the offset steps the source's innermost dim (stride 1). An offset+base outside `[0, full_source_len)`, or a NaN offset, yields genuine IEEE NaN (the out-of-range result Vensim documents as `:NA:`; this is the absorbing NaN, NOT the finite `crate::float::NA` sentinel). NO modulo / NO wraparound (the bug the prior sliced-view-no-base implementation had). 8. **`src/alloc.rs`** - Allocation helpers for VM priority allocation: `allocate_available()` (bisection-based priority allocation), `alloc_curve()` (per-requester allocation curves for 6 profile types), `normal_cdf()`/`erfc_approx()`. @@ -92,8 +92,8 @@ Diagnostics (`collect_all_diagnostics`) run on the user's `SourceProject`, so sy - **`src/datamodel.rs`** - Core structures: `Project`, `Model`, `Variable`, `Equation` (including `Arrayed` variant with `default_equation` for EXCEPT semantics and `has_except_default` bool flag), `Dimension` (with `mappings: Vec` replacing the old `maps_to` field, and `parent: Option` for indexed subdimension relationships), `DimensionMapping`, `DataSource`/`DataSourceKind`, `UnitMap`, `MacroSpec` (a macro-marked `Model`'s calling convention: `parameters`/`primary_output`/`additional_outputs`; `Model.macro_spec` is `None` for every ordinary model; `salsa::Update` so it rides directly on the `SourceModel` input). `Model::new_macro(name, params, additional_outputs, body_variables)` is the shared port-synthesis + `MacroSpec`-construction step used by *both* the MDL converter and the XMILE reader: it sets `can_be_module_input` on each formal-parameter body variable (synthesizing a `Flow`/`Aux` placeholder port when absent) so `collect_module_idents` treats the macro as an ordinary sub-model. View element types (`Aux`, `Stock`, `Flow`, `Alias`, `Cloud`) carry an optional `ViewElementCompat` with original Vensim sketch dimensions/bits for MDL roundtrip fidelity. `StockFlow` has an optional `font` string for the Vensim default font spec. - **`src/variable.rs`** - Variable variants (`Stock`, `Flow`, `Aux`, `Module`), `ModuleInput`, `Table` (graphical functions). `classify_dependencies()` is the primary API for extracting dependency categories from an AST in a single walk, returning a `DepClassification` with five sets: `all` (every referenced ident), `init_referenced`, `previous_referenced`, `previous_only` (idents only inside PREVIOUS), and `init_only` (idents only inside INIT/PREVIOUS). `parse_var_with_module_context` accepts a `module_idents` set so `PREVIOUS(module_var)` rewrites through a scalar helper aux instead of `LoadPrev`. Per-element graphical functions: `build_tables` materializes one `Table` per element of an arrayed GF and `reorder_arrayed_element_tables` places each at the element's *flat row-major declared-dimension index*, NOT its `Equation::Arrayed` `elems` Vec position -- because the runtime selects a per-element table by the row-major dimension offset (`vm.rs` `Lookup`/`LookupArray`: `graphical_functions[base_gf + element_offset]`). Elements lacking a GF get an empty placeholder so the index stays aligned. The salsa dependency-table path mirrors this in `db.rs::extract_tables_from_source_var`. - **`src/dimensions.rs`** - `DimensionsContext` for dimension matching, subdimension detection, and element-level mappings. Supports indexed subdimensions via `parent` field (child maps to first N elements of parent). `has_mapping_to()` checks for element-level dimension mappings between two dimensions. `mapped_element_correspondence(iterated_dim, source_dim)` (GH #527) is the reusable element-level correspondence for a mapped dimension pair -- per iterated element, the source element the executed simulation reads (both declaration directions, single-hop only, POSITIONAL mappings only; an explicit element map returns `None` because the executed A2A lowering resolves positionally and ignores it -- GH #753 and the tracked positional-vs-element-map execution inconsistency (GH #756) are the gate for re-enabling; `None` ⇒ callers keep their conservative broadcast, a superset of the true edges). It is what keeps the LTM element graph (`expand_same_element`'s mapped diagonal) and link-score dimensions (`link_score_dimensions` -- whose mapped arm is additionally gated on the edge having a `Bare`-classified site via `model_edge_shapes`, so the arrayed retarget fires exactly when the element graph expands the diagonal rather than the element-mapped `DynamicIndex` cross-product) in lockstep with the classifier (`classify_iterated_dim_shape`, whose mapped arm gates on this same correspondence in BOTH declaration directions since GH #757 -- via `ltm_agg::classify_axis_access` / `iterated_axis_slot_elements` -- so a reverse-declared positional pair classifies `Bare` and gets the diagonal) and -- for positional mappings -- the simulation; the agg machinery's mapped sliced reducers (GH #534) consume it through `ltm_agg::iterated_axis_slot_elements` (the per-source-element preimage inversion), so the same gate governs hoisting and the emitters' slot remap. `SubdimensionRelation` caches parent-child offset mappings for both named (element containment) and indexed (declared parent) dimensions -- **`src/model.rs`** - Model compilation stages (`ModelStage0` -> `ModelStage1` -> `ModuleStage2`), dependency resolution, topological sort. `collect_module_idents` pre-scans datamodel variables to identify which names will expand to modules (preventing incorrect `LoadPrev` compilation). `init_referenced_vars` extends the Initials runlist to include variables referenced by `INIT()` calls, ensuring their values are captured in the `initial_values` snapshot. Unit checking uses salsa tracked functions in `db.rs`. The two `#[cfg(test)]` `ModelStage0` constructors are the salsa-FREE twin of `db::stages::model_stage0`: `new_in_project(project_models, x_model, ..)` builds a stage from a `datamodel::Model` with no database, resolving module-function calls against the whole project's `MacroRegistry` (which is what makes it a faithful oracle for a model that CALLS a macro defined in a sibling model), and `new(x_model, ..)` is the single-model wrapper the many one-model fixtures use. They derive the module-ident set, the macro registry and the duplicate-ident errors along completely different routes than the query, which is exactly why `db::stages_tests` can use them as an oracle. The former salsa-cached `ModelStage0::new_cached` -- a test-only third copy of the query's construction -- was deleted; its coverage now points at the live query. `enumerate_modules_inner` records a model's instantiation BEFORE descending into it, so that a module cycle terminates: the "have I seen this model" test is the recursion guard, so a model still being walked has to count as seen. (A cycle through `main` already terminated, because `enumerate_modules` records `main` up front; one below `main` did not.) This matches its salsa twin `db::assemble::enumerate_module_instances_inner`. Recording early cannot LOSE an instantiation because the insert is unconditional at every module site -- only the recursion is guarded; the set-of-input-sets value is a separate fact, and it is why the ORDER instantiations arrive in is unobservable. -- **`src/project.rs`** - `Project` struct aggregating models. `from_salsa(datamodel, db, source_project, cb)` builds a Project from a pre-synced salsa database: it READS `db::stages::model_stage1` for every project model and clones each memo (the clone is mandatory, not incidental -- `model_deps.take()`, `set_dependencies` and the `model_cb` all mutate, and `set_dependencies` pushes equation errors onto the variables themselves, while a `returns(ref)` memo is shared with every other reader). It used to build its own whole-project `ModelStage0` map and lower it inline, a second salsa-native copy that had silently drifted from `db::stages` on three fields. Its model order and the Initials runlists `set_dependencies` derives are deterministic: both `topo_sort` seeds are sorted first, since `topo_sort` breaks ties by visit order and the seeds came from a `HashMap`'s keys and a `HashSet` respectively (the `Project`-path twin of the GH #595 fix in `db::dep_graph`, safe to change because nothing shipped reads these runlists -- `compiler::Module::new`, the sole reader of `ModelStage1::instantiations`, is called only from `#[cfg(test)] TestProject::build_module`, while production `src/db/assemble.rs` builds `compiler::Module` by struct literal from salsa fragments). `from_datamodel(datamodel)` is a convenience wrapper that creates a local DB and syncs. **Neither is reachable from production**: inside the crate every caller is a test, and the only in-repo user of the public `From` impl is the engine's own `tests/integration/simulate_ltm.rs`, feeding the `ltm_finding::discover_loops` convenience wrapper (the shipped analysis path, libsimlin's `simlin_analyze_discover_loops`, goes through `discover_loops_with_graph` and builds no `Project`). Treat it as a monolith kept honest as a test oracle, not as live code. Production compiles via `db::compile_project_incremental` with `ltm_enabled`/`ltm_discovery_mode` on `SourceProject`. +- **`src/model.rs`** - Model compilation stages (`ModelStage0` -> `ModelStage1` -> `ModuleStage2`). It performs NO dependency analysis of its own: `ModelStage1::set_dependencies` reads the production dependency graph (`db::dep_graph::model_dependency_graph`) once per module instantiation and copies its runlists into `ModuleStage2`. Until GH #568 it ran a second, independent walk here -- its own transitive closure (`all_deps`), its own cross-model output resolution (`module_output_deps`), its own `topo_sort` runlists and its own `CircularDependency` -- and that second gate genuinely disagreed with production's, rejecting the element-acyclic recurrence SCCs `resolve_recurrence_sccs` resolves. There is one gate now, pinned by `project.rs`'s `the_circular_dependency_gate_is_the_production_one` in both the resolves and the rejects direction. `collect_module_idents` pre-scans datamodel variables to identify which names will expand to modules (preventing incorrect `LoadPrev` compilation). Unit checking uses salsa tracked functions in `db.rs`. The two `#[cfg(test)]` `ModelStage0` constructors are the salsa-FREE twin of `db::stages::model_stage0`: `new_in_project(project_models, x_model, ..)` builds a stage from a `datamodel::Model` with no database, resolving module-function calls against the whole project's `MacroRegistry` (which is what makes it a faithful oracle for a model that CALLS a macro defined in a sibling model), and `new(x_model, ..)` is the single-model wrapper the many one-model fixtures use. They derive the module-ident set, the macro registry and the duplicate-ident errors along completely different routes than the query, which is exactly why `db::stages_tests` can use them as an oracle. The former salsa-cached `ModelStage0::new_cached` -- a test-only third copy of the query's construction -- was deleted; its coverage now points at the live query. `enumerate_modules_inner` records a model's instantiation BEFORE descending into it, so that a module cycle terminates: the "have I seen this model" test is the recursion guard, so a model still being walked has to count as seen. (A cycle through `main` already terminated, because `enumerate_modules` records `main` up front; one below `main` did not.) This matches its salsa twin `db::assemble::enumerate_module_instances_inner`. Recording early cannot LOSE an instantiation because the insert is unconditional at every module site -- only the recursion is guarded; the set-of-input-sets value is a separate fact, and it is why the ORDER instantiations arrive in is unobservable. +- **`src/project.rs`** - `Project` struct aggregating models. `from_salsa(datamodel, db, source_project, cb)` builds a Project from a pre-synced salsa database: it READS `db::stages::model_stage1` for every project model and clones each memo (the clone is mandatory, not incidental -- `model_deps.take()`, `set_dependencies` and the `model_cb` all mutate it, while a `returns(ref)` memo is shared with every other reader). Each stage is kept paired with the `SourceModel` handle it came from, because `set_dependencies` reads the production dependency graph and that query is keyed on the handle, not on the model's name. It used to build its own whole-project `ModelStage0` map and lower it inline, a second salsa-native copy that had silently drifted from `db::stages` on three fields. Its model order is deterministic because the `topo_sort` seed is sorted first -- `topo_sort` breaks ties by visit order and the seed came from a `HashMap`'s keys (the `Project`-path twin of the GH #595 fix in `db::dep_graph`); the Initials runlists inherit the dependency graph's own determinism since GH #568 unified the two gates. `compiler::Module::new` is the sole reader of `ModelStage1::instantiations`; its callers are `#[cfg(test)] TestProject::build_module` and three direct calls in the `compiler`'s dimension tests, all `#[cfg(test)]`, while production `src/db/assemble.rs` never builds a `compiler::Module` at all. It REFUSES a model whose dependency graph carries a resolved recurrence SCC (`ModelStage1::set_dependencies` records a `NotSimulatable` for one): GH #568 unified the cycle GATE but not the EMITTER, and the per-element interleave `db::assemble::combine_scc_fragment` performs has no monolithic equivalent, so emitting the members whole would read a co-member's element before assigning it -- a silent wrong answer where the second gate used to give an accidental refusal. A model that can REACH a module cycle is gated out of dependency resolution first, exactly as the production entry points gate it (`db::diagnostic::module_cycle_diagnostic`, GH #806): the dependency graph's recurrence-SCC refinement descends into the recursive `model_module_map`, and salsa turns that into an unrecoverable dependency-graph panic -- a process abort under `panic = abort`, reachable from the public `From`. Such a model records the cycle as its error and keeps empty runlists. `from_datamodel(datamodel)` is a convenience wrapper that creates a local DB and syncs. **Neither is reachable from production**: inside the crate every caller is a test, and the only in-repo user of the public `From` impl is the engine's own `tests/integration/simulate_ltm.rs`, feeding the `ltm_finding::discover_loops` convenience wrapper (the shipped analysis path, libsimlin's `simlin_analyze_discover_loops`, goes through `discover_loops_with_graph` and builds no `Project`). Treat it as a monolith kept honest as a test oracle, not as live code. Production compiles via `db::compile_project_incremental` with `ltm_enabled`/`ltm_discovery_mode` on `SourceProject`. - **`src/results.rs`** - `Results` (variable offsets + timeseries data), `Specs` (time/integration config) - **`src/patch.rs`** - `ModelPatch`/`ProjectPatch` (both `Clone`) for representing and applying model changes. `ModelOperation` variants: `UpsertVariable`, `DeleteVariable`, `RenameVariable`, `UpsertView`, `DeleteView`, `UpdateStockFlows`, `SetLoopName` (names a feedback loop by variable list, writing `LoopMetadata` entries). `ModelPatch` is consumed by `incremental_layout()` to determine what variables were added/removed/renamed @@ -103,22 +103,24 @@ The primary compilation path uses salsa tracked functions for fine-grained incre - **`src/db.rs`** - The module ROOT of the salsa pipeline. After the db-submodule split it holds only the database itself (`SimlinDb` + its `Db` trait, the sync/staged/restore/`current_source_project` methods, the `StdlibModels` cache, the `impl salsa::Database`/`Db`), the `compile_project_incremental()` production entry point, the LTM-glue surface (`LtmSyntheticVar`/`LtmVariablesResult` result types + the `module_link_score_equation` module-link helper (composite reference / ceteris-paribus / signed `black_box_unit_transfer_equation` fallback) called by the per-shape `link_score_equation_text_shaped` (which lives in `src/db/ltm/compile.rs`) -- since Track A the SOLE derivation of a link score's equation text: the former `(from, to)`-keyed `link_score_equation_text` twin was deleted and assembly's standard scalar Bare score now sources the shaped query directly (`compile_ltm_var_fragment`), so the compiled and reported equations are one value and cannot drift; `module_composite_ports` reads the sub-model's `model_ltm_variables` to decide whether a port has a composite, so the same composite reference is used in exhaustive AND discovery mode)/`find_model_output_ports_for_module`/`module_input_pathways_from_edges`/`generate_max_abs_selection` helpers; the last folds a module input port's "strongest pathway" composite through O(1)-sized accumulator helper variables -- inlining the fold into one nested expression doubles the equation text per pathway, which OOMed on macro modules with hundreds of pathways. Since PR #684 a stockless **passthrough** with input→output pathways ALSO emits pathway/composite vars, and since GH #748 the stock-free early return in `model_ltm_variables` fires only when the model is genuinely STATELESS -- no parent-level stocks, no input ports, and no (transitively) stock-carrying module instance (`modules_carry_state`), so a module-only root whose state lives inside a SMOOTH/DELAY or user sub-model is scored instead of silently emitting nothing, and the exhaustive-mode loop-score builder overrides each `x→m` module link with a **per-exit-port pathway selection** (`compute_module_link_overrides` in `src/db/ltm/mod.rs` emits a `$⁚ltm⁚link_score⁚{x}→{m}⁚via⁚{exit}` alias selecting the pathway the loop actually traverses, threaded into `generate_loop_score_variables` as a `(loop_id, link_index)` side-table) because the composite max-abs-selects across ALL pathways and so picks the wrong output port for a multi-output module; `find_model_output_ports` now sorts its result (GH #680) so the parent's pathway recomputation matches the sub-model's emitted `$⁚ltm⁚path⁚{port}⁚{idx}` indices, the residual being that pinned loops pass an empty override map), the `set_project_ltm_enabled`/`set_project_ltm_discovery_mode` setters plus the `LtmEnabledGuard` RAII scope guard (transiently flips `ltm_enabled` and unconditionally restores it on drop; shared by libsimlin's `simlin_project_get_errors` / from-wasm FFIs (GH #466) and `simlin-mcp-core`'s `read_model`/`edit_model` diagnostic passes (GH #662) so the LTM-only advisory harvest behaves identically across every diagnostic-collection surface), and the `#[cfg(test)]` test-module mounts. Everything else moved into per-concern db submodules (`input`/`query`/`stages`/`sync`/`diagnostic`/`layout`/`var_fragment`/`fragment_compile`/`assemble`/`dep_graph`/`analysis`/`ltm`/...) and is re-exported at the root so the historical `simlin_engine::db::...` surface (and the `use super::*` globs in the root-mounted test modules and `implicit_deps.rs`) keep resolving. `SimlinDb` OWNS its sync state (a private `sync_state: Option` field; the `#[salsa::db]` macro finds `storage` by type, and the field is only mutated via `&mut self` during sync, never during parallel query execution, so no interior mutability is needed), so incrementality is automatic: `db.sync(project) -> SourceProject` threads the prior handles internally (a no-op re-sync stays a salsa cache hit), `db.sync_staged(project) -> (SourceProject, Option)` additionally returns the PRE-staging handles for an exact rollback, `db.restore(project, prev)` re-syncs the prior datamodel with those handles, and `db.current_source_project()` reads the latest handle. A SECOND, `pub(crate)` slot -- `expanded_state: Option`, driven by `db.sync_expanded(expanded_datamodel)` -- holds the conveyor/queue-expanded twin of the project so the special-stock build path is incremental too (see "Conveyors and queues"); it stays `None` until a special-stock model is first built, is never cleared thereafter (salsa cannot reclaim inputs, so clearing would only force a second set to be minted), and its handle is reachable only as `sync_expanded`'s return value, which is what keeps a rolled-back staged patch from poisoning it. `sync_from_datamodel` (fresh `&db` path returning `SyncResult`) and `sync_from_datamodel_incremental` stay public as the internal mechanism the methods wrap and for the read-only test path; `collect_all_diagnostics(db, project: SourceProject)` iterates `project.models(db)`. `SourceProject` carries `ltm_enabled` and `ltm_discovery_mode` flags for LTM compilation, plus `macro_declarations: Vec<(String, Option)>` -- the ordered, pre-dedup list of project models' canonical names + macro specs that the demand-driven `project_macro_registry` query reads to re-derive the macro build error (the canonical-name-keyed `models` map collapses the duplicate/colliding model names the AC5.3 validation needs, so the ordered declaration list supplies them). `SourceModel` carries `macro_spec: Option` so `project_macro_registry` is keyed only on the macro-marked models. `collect_module_idents`/`equation_is_module_call` now consult the `MacroRegistry` so a `y = MYMACRO(...)` caller is pre-classified as module-backed (correct `PREVIOUS`/`INIT` rewrite); a macro-body variable's `enclosing_model` is the macro name so a renamed `init`/`previous` builtin inside a like-named macro resolves to the intrinsic (#554). `Diagnostic` includes a `severity` field (`Error`/`Warning`) and `DiagnosticError` variants: `Equation`, `Model`, `Unit`, `Assembly`. The salsa inputs store the `datamodel::*` types directly on their multi-field structs (`SourceProject.sim_specs`/`dimensions`/`units`, `SourceModel.model_sim_specs`, `SourceVariable.equation`/`gf`/`module_refs`) -- per-field salsa read tracking is what gives fine-grained invalidation, so no mirror-projection types are needed; `datamodel_variable_from_source` re-assembles the kind-tagged `datamodel::Variable` for the parser. `datamodel::Equation::Arrayed` carries `has_except_default` to drive EXCEPT default application. `VariableDeps` includes `init_referenced_vars` to track variables referenced by `INIT()` calls. Dependency extraction uses two calls to `classify_dependencies()` (one for the dt AST, one for the init AST) instead of separate walker functions. `parse_source_variable_with_module_context` is the sole parse entry point (the non-module-context variant was removed). `variable_relevant_dimensions` provides dimension-granularity invalidation: scalar variables produce an empty dimension set so dimension changes never invalidate their parse results. `compile_var_fragment` is the salsa-tracked per-variable fragment compiler; its lowering half lives in the sibling `src/db/var_fragment.rs` (`lower_var_fragment`). For a recurrence SCC that `db::dep_graph::resolve_recurrence_sccs` resolved (element-acyclic), the per-element symbolic segments of every member are interleaved into ONE combined `PerVarBytecodes` along the SCC's `element_order` by `combine_scc_fragment` (the per-element-granular generalization of `compiler::symbolic::concatenate_fragments`), built from each member's exact production symbolic fragment via `var_phase_symbolic_fragment_prod(.., scc.phase)` (loud-safe: an unsourceable member or a segmentation error returns `None`/`Err` and the model keeps its `CircularDependency`). `assemble_module` skips each member's per-variable fragment and injects this combined fragment at the first member's runlist slot (the dt fragment into the flow fragments, the init fragment as one synthetic-ident `SymbolicCompiledInitial`); per-write `SymVarRef` identity is preserved, so variable layout offsets are unchanged and each member stays individually addressable. - **`src/db/input.rs`** (a `db` submodule) - The salsa INPUT layer: the interned key types (`LtmLinkId`/`ModuleIdentContext`/`ModuleInputSet`, the last with its `empty`/`from_canonical_set`/`from_names`/`canonical_input_set` round-trip), the `SourceVariableKind` tag, the three `#[salsa::input]` structs (`SourceProject`/`SourceModel`/`SourceVariable`) that store the synced datamodel field-by-field for fine-grained invalidation, the `PinnedLoopSpec` projection of a non-deleted `LoopMetadata` (a pinned loop's name + canonical variable set, carried on `SourceModel.pinned_loops`; the `LoopMetadata` UIDs are resolved to canonical names at sync time since UIDs never enter the db), the `source_var_is_table_only` lookup-only predicate, `SourceModel.declared_variable_idents` (the ordered, pre-dedup AS-WRITTEN variable-ident list -- the raw data the GH #885 duplicate-canonical-ident check needs, which the canonical-keyed `variables` map collapses; the per-variable analogue of `SourceProject.macro_declarations`), and `datamodel_variable_from_source` (re-assembles the kind-tagged `datamodel::Variable` for the parser). Re-exported at the `db.rs` root. -- **`src/db/query.rs`** (a `db` submodule) - The demand-driven read queries over the inputs: per-variable parsing (`parse_source_variable_with_module_context` + `_impl`), the project-global cached contexts (`project_units_context_result` -- carrying the units `Context` AND the `definition_errors` building it produced, with `project_units_context` a salsa PROJECTION over its `ctx` -- not a plain accessor, since a reader of one would take a dependency on the whole result and rebuild whenever only the errors moved, which is every intermediate state of typing a malformed unit equation; the projection backdates on the equal `Context` and keeps readers' dependencies exactly as narrow as they were. The errors ride the RESULT rather than a salsa accumulator because a bad unit declaration is a project-level fact that an accumulator both reported once per model and lost entirely after an unrelated revision bump -- plus `project_datamodel_dims`, `project_dimensions_context`, `project_converted_dimensions`), the module-ident context (`model_module_ident_context`), the per-variable direct-dependency extraction (`variable_direct_dependencies` + `_impl`, the `VariableDeps` value, the `canonical_module_input_set` helper), the implicit-variable metadata (`ImplicitVarMeta`/`model_implicit_var_info`), the recursive `model_module_map`, the explicit-edges-only module-reference graph (`ModuleReferenceGraph`/`project_module_graph`, the cycle gate every compile / diagnostic / analysis entry point consults first so a module cycle is a `CircularDependency` instead of salsa's dependency-graph cycle panic -- GH #806; its rustdoc carries the argument for why omitting the IMPLICIT edges is sound now that `MacroRegistry::build`'s Pass 4 rejects the one shape that could close a cycle through one, plus the enumerated-not-proved residual that residual rests on), and the dimension-read queries (`variable_relevant_dimensions`/`variable_dimensions`/`variable_size`). `variable_relevant_dimensions` gives dimension-granularity invalidation (scalars produce an empty set, so dimension changes never invalidate them). +- **`src/db/query.rs`** (a `db` submodule) - The demand-driven read queries over the inputs: per-variable parsing (`parse_source_variable_with_module_context` + `_impl`), the project-global cached contexts (`project_units_context_result` -- carrying the units `Context` AND the `definition_errors` building it produced, with `project_units_context` a salsa PROJECTION over its `ctx` -- not a plain accessor, since a reader of one would take a dependency on the whole result and rebuild whenever only the errors moved, which is every intermediate state of typing a malformed unit equation; the projection backdates on the equal `Context` and keeps readers' dependencies exactly as narrow as they were. The errors ride the RESULT rather than a salsa accumulator because a bad unit declaration is a project-level fact that an accumulator both reported once per model and lost entirely after an unrelated revision bump -- plus `project_datamodel_dims`, `project_dimensions_context`, `project_converted_dimensions`), the module-ident context (`model_module_ident_context`), the per-variable direct-dependency extraction (`variable_direct_dependencies` + `_impl`, the `VariableDeps` value, the `canonical_module_input_set` helper), the per-name variable lookup `model_variable_by_name` (a salsa FIREWALL over the `SourceModel::variables` map field: reading the map directly gives the reader a dependency on the whole variable SET, which is what made every per-variable fragment recompile when any variable was added, deleted or renamed; the query still reads the map but returns one handle, so it backdates and a fragment depends only on the dependencies it looks up), the implicit-variable metadata (`ImplicitVarMeta`/`model_implicit_var_info`), the recursive `model_module_map`, the explicit-edges-only module-reference graph (`ModuleReferenceGraph`/`project_module_graph`, the cycle gate every compile / diagnostic / analysis entry point consults first so a module cycle is a `CircularDependency` instead of salsa's dependency-graph cycle panic -- GH #806; its rustdoc carries the argument for why omitting the IMPLICIT edges is sound now that `MacroRegistry::build`'s Pass 4 rejects the one shape that could close a cycle through one, plus the enumerated-not-proved residual that residual rests on), and the dimension-read queries (`variable_relevant_dimensions`/`variable_dimensions`/`variable_size`). `variable_relevant_dimensions` gives dimension-granularity invalidation (scalars produce an empty set, so dimension changes never invalidate them). - **`src/db/stages.rs`** (a `db` submodule) - The two name-keyed, PRE-LAYOUT model-compilation stages as salsa-cached `returns(ref)` queries. This is the ONLY place in the crate they are built from salsa inputs. `Project::from_salsa` used to carry a second inline copy, silently disagreeing on three fields; this body took `from_salsa`'s behaviour in all three, so retiring that copy was a deletion rather than a merge, and `from_salsa` now clones these memos. Three other `ModelStage0`/`ModelStage1` constructions remain, none of them a whole-model salsa build and each deliberate: the per-variable MINI Stage0 in `db::var_fragment::lower_var_fragment` and `db::ltm::compile` (scoped to one variable's dependencies -- pointing them here would add a project-wide dependency edge to every fragment compile), and `db::units::check_conveyor_param_units`' throwaway augmented `ModelStage1::new` over a CLONE of the cached Stage0 (see that module's header). `model_stage0(db, model, project) -> ModelStage0` parses a model's variables (through the per-variable `parse_source_variable_with_module_context` memos) plus the implicit SMOOTH/DELAY/TREND helpers those parses synthesize; `model_stage1(db, model, project) -> ModelStage1` lowers that Stage0's equations to `Expr2` via `ModelStage1::new` against a `ScopeStage0` holding the project's dimension context and the Stage0 of every model in `model_scope_models(db, model, project)` -- this model plus the transitive closure of the models its module variables instantiate. Both were previously rebuilt from scratch by each consumer -- `db::units::check_model_units` is tracked PER MODEL yet rebuilt both stages for EVERY project model on each call, making whole-project unit diagnostics quadratic in the model count (GH #966); the unit pass now READS these. Two decisions live here rather than at a call site, both currently inert in the stage VALUE and therefore pinned directly by `stages_tests`: `model_is_stdlib` marks a model implicit only when the `stdlib⁚` prefix is followed by a name in `stdlib::MODEL_NAMES` (a bare-prefix model an import produced is a user model, not a template), and `source_model_is_stdlib` is the canonicalizing handle-level wrapper `db::units`' unit-check skip gate now shares (GH #988) -- that gate used to carry its own looser bare-prefix spelling, so an imported `stdlib⁚` model silently lost unit checking while the stage query staged it as an ordinary user model, and `extra_module_idents` adds EVERY variable name of a stdlib model to its `ModuleIdentContext` so `PREVIOUS(module_input)` inside a stdlib body would rewrite through a scalar helper aux (no shipped stdlib body calls PREVIOUS/INIT, so this changes no parse today -- it exists so every path reaching a stdlib model's parse hits the same interned context and shares one set of parse memos instead of minting a second). Stage0 records duplicate-canonical-ident model errors from the memoized `db::model_duplicate_variables` groups (GH #885/#891); nothing on the unit path reads that field. `model_scope_models` is that closure, as an iterative worklist rather than a recursive tracked query -- a module cycle is a project a user can draw, and recursing a tracked query on one is salsa's unrecoverable dependency-graph panic (GH #806). Its edges come from each model's Stage0 `variables`, so they include the IMPLICIT modules builtin and macro expansion synthesized: a macro call targets the macro's own model, an ordinary user model that can be arrayed, which is why `db::project_module_graph` (explicit edges only, deliberately, to stay parse-free) is the wrong source here. A module's own IDENT is an edge too when it names a project model, because `resolve_relative` predates a module ident differing from its target's name and keys each dotted `src` component as a MODEL name; dropping that edge turns a project that lowered cleanly into one reporting `BadModuleInputSrc`. `check_model_units` reads the SAME closure for its inference map (`units_infer` consults its models map only for module targets, recursing along the same edges), so the two halves cannot disagree about what is reachable. The narrowing is what makes an unrelated model's edit invalidate neither a model's lowered stage nor its unit check, pinned by the execution counters in both directions; the whole-project map cost that, and cost the reads too. `model_scope_stage0` resolves the closure to Stage0s and REPAIRS a missing self entry (a handle that outlived its place in the project's model map), because without it `get_dimensions` returns `None` for every reference in the model and every arrayed equation lowers as though it were scalar; the Stage1 map in `check_model_units` deliberately does NOT repair it, since that absence is its signal that the handle is not the project's model. Three fixtures in `stages_tests` are what can see a narrowing go too far, each catching a different mistake: `arrayed_module_project` (`main` reduces over an ARRAYED output of its DIRECT target `sub_a` -- catches a self-only scope; before it existed, emptying the scope map entirely left every test in the crate green), `chain_project` (`main -> sub_a -> sub_c` with `main` reducing over `sub_a.sub_c.out_by_region`, where `sub_c` is reachable only transitively -- the only thing that catches a "self + direct targets" scope, so the closure must be TRANSITIVE), and `omitting_stdlib_models_from_the_lowering_scope_is_inert_today` (dropping the `stdlib⁚*` models wholesale is harmless for LOWERING, and that test asserts the precise reason -- no stdlib template is arrayed or instantiates a module -- so it fails the moment that stops being true; its module half doubles as the tripwire for the stdlib-is-a-SINK premise of `MacroRegistry::build`'s Pass 4 closure argument, so if it ever reds, the abort-class consequence is that `project_module_graph` no longer sees every cycle; the closure does not take that shortcut, and since the inference map shares it, a `stdlib⁚*`-skipping closure is no longer inert end to end -- it reds `unit_checking_test::test_smth1_unit_mismatch_initial`). Test-only thread-local execution counters (`reset_query_executions`/`query_executions`, covering both stage queries AND `check_model_units`, so one reset covers every counter a test in this area reads) let `stages_tests` count query-body entries: pointer equality of a `returns(ref)` memo cannot prove a body did not run, since salsa backdates a re-executed query whose value compares equal without moving the memo. - **`src/db/stages_tests.rs`** - Tests for the two stage queries: value oracles against the salsa-free `ModelStage0::new_in_project` (Stage0) and `ModelStage1::new` over a whole-project scope of those (Stage1), including `every_shape_project`, the combined fixture carrying every shape the deleted `Project::from_salsa` copy handled -- an implicit stdlib `SMTH1` expansion, an explicit user sub-model, a macro-marked model AND a caller of it, and duplicate canonical idents; the three Stage0 semantics decisions above; the GH #966 cost claim (a whole-project `collect_all_diagnostics` BUILDS each model's Stage0/Stage1 at most once, and only for the models something reaches -- in a fixture instantiating no stdlib template, the spliced stdlib set is never staged at all, which is the scope narrowing measured; re-reading every model's stages once per model, the pre-#966 access pattern, then rebuilds none) plus its companion `project_from_salsa_reads_the_cached_stages` (`from_salsa` demands each model's stages once through the queries and the unit pass afterwards rebuilds none -- the only evidence that distinguishes reading the memo from building an equal second copy, since the deleted build produced EQUAL values by design); and diagnostic reachability across the move (a unit warning still reaches both `check_model_units::accumulated` and `collect_all_diagnostics`, as does `project_macro_registry`'s `DuplicateMacroName`, which `check_model_units` now reaches ONLY two tracked queries deeper); and the scope narrowing itself -- the closure's shape (transitive, implicit/macro edges included, module-ident edge, finite on a cycle) and its INVALIDATION consequences in both directions, since a scope that is too wide only costs incrementality while one that is too narrow answers from a stale memo. Two of those tests demonstrate a CONSEQUENCE rather than a shape, because a name-list assertion cannot show that losing an edge would mis-lower or mis-diagnose anything. `dropping_a_macro_models_scope_edge_changes_the_lowered_value` is the macro-side twin of `arrayed_module_project`: `main` reduces over an ARRAYED macro output, so removing the macro model from the scope changes the lowered VALUE of exactly that caller, with a SCALAR-output control that changes nothing -- which pins the mechanism as dimension resolution through the macro edge rather than sensitivity to the map's contents. Its fixture guard reads the model's STAGE0 (not its scope) on purpose, so that a macro-edge-dropping narrowing reds it on the value rather than on a structural check. `a_two_hop_cross_module_unit_mismatch_is_still_reported` is the unit guard for the TRANSITIVE row of the matrix (the stdlib row already had `unit_checking_test::test_smth1_unit_mismatch_initial`): a widget/gadget contradiction that closes only after `units_infer` walks BOTH module hops, so a direct-targets-only closure drops the grandchild's constraints and the diagnostic silently disappears. Note what a scope mis-lowering can and cannot corrupt: `model_stage1`'s consumers are unit checking and the test-only `Project::from_salsa`, while simulation compiles from the per-variable fragment path's own mini Stage0s and never reads this stage. The two invalidation tests drive `sync_from_datamodel_incremental`, which REUSES the `SourceProject` handle: a value read through an older `SyncResult` after an edit is the POST-edit value, so every read is made against the sync current at that moment, and each has a control step (re-syncing the identical project must re-execute nothing) so a count is attributable to the edit. Every equality oracle here ranges over USER models only: `ModelStage0` derives `PartialEq`, it compares parsed `f64` constants, and every SMOOTH/DELAY/TREND template declares `initial_value = NAN`, so a stdlib stage carrying a NaN does not compare equal even to ITSELF (GH #987/#981). The one stdlib oracle uses `npv`, the single template with no NaN literal. - **`src/db/sync.rs`** (a `db` submodule) - Datamodel -> salsa-input sync: the `SyncResult`/`SyncedModel`/`SyncedVariable` handle maps, the `Clone`-able `PersistentSyncState`/`PersistentModelState`/`PersistentVariableState` snapshots threaded between sync calls (with `to_sync_result`/`to_synced_model`/`from_sync_result`), the one-shot stdlib-input builder (`build_stdlib_models`, feeding the root's `StdlibModels` cache), the fresh (`sync_from_datamodel`) and incremental (`sync_from_datamodel_incremental`) entry points + their per-variable helpers (`source_variable_from_datamodel`, `update_source_variable`), the `macro_declarations_from_datamodel` extractor, `pinned_loops_from_datamodel` (resolves a model's `loop_metadata` UIDs to canonical variable names for `SourceModel.pinned_loops`), and `expand_maps_to_chains` (the `maps_to`/`mappings` reachability closure the parser uses to size a variable's dimension dependency). - **`src/db/diagnostic.rs`** (a `db` submodule) - Compilation diagnostics: the `CompilationDiagnostic` salsa accumulator, the typed `Diagnostic` value (`severity` field + `DiagnosticError` variants `Equation`/`Model`/`Unit`/`Assembly`), the per-model `model_all_diagnostics` query that triggers every diagnostic source (`compile_var_fragment` per variable, the unit-check pass, and -- when `ltm_enabled` -- `model_ltm_fragment_diagnostics`), the `model_duplicate_variables` query + `emit_duplicate_variable_diagnostics` (GH #885: an Error-severity `DuplicateVariable` diagnostic per group of variables whose names canonicalize to the same ident, derived from the raw `declared_variable_idents` input; `compile_project_incremental` fails hard on the same query, and `queue_compile::build_compiled` runs the identical datamodel-level check before expansion), and the drain helpers `collect_model_diagnostics` (one model) / `collect_all_diagnostics` (whole synced project). Three diagnostics deliberately bypass the accumulator: `collect_all_diagnostics` emits each directly from a memoized derivation, and their row counts DIFFER. The **macro-registry build error** (`project_macro_registry`) is at most one row per project, naming no model or variable. The **unit-definition errors** (`project_units_context_result`) are one row per declaration that failed to parse -- several are normal -- naming no model but carrying the unit's name as `variable`. The **module-reference cycle** (`project_module_graph`) is one row per MODEL that can reach a cycle, carrying that model's name and skipping its per-model passes; that one is deliberately per-model and must stay so, since a model reaching no cycle is still processed normally and an unrelated draft cycle cannot hide a valid model's diagnostics (GH #806) -- collapsing it to a single project-level row would revert that. It is emitted by `module_cycle_diagnostic`, which `collect_model_diagnostics` consults and `collect_all_diagnostics` inherits by delegating to it. That gate used to be inline in the whole-project loop only, so the per-model entry point -- equally `pub` -- drove a cyclic model's passes into `model_module_map` and salsa's dependency-graph cycle panic; one function for the gate is what keeps the two entry points from disagreeing about whether the same project panics. What the first two share is their ORIGIN, not their count: each is derived once per project and each used to be accumulated from inside its deriving query's body, which both over-reported (every model's subtree reaches the query, so the DFS found the one value once per model) and lost the diagnostic entirely once an unrelated revision bump let the DFS prune the subtree. - **`src/db/layout.rs`** (a `db` submodule) - The salsa-tracked per-model *body* layout query `compute_layout` (offsets from 0; the root's `IMPLICIT_VAR_COUNT` shift is applied later at assembly via `VariableLayout::root_shifted`). Lays out explicit variables, implicit (SMOOTH/DELAY/TREND) helpers, and -- when `ltm_enabled` -- the LTM synthetic variables and their implicit helpers. -- **`src/db/fragment_compile.rs`** (a `db` submodule) - The *emission half* of per-variable compilation (the *lowering half* is the sibling `src/db/var_fragment.rs`). `compile_var_fragment` is the salsa-tracked per-variable fragment compiler; `compile_implicit_var_fragment`/`compile_implicit_var_phase_bytecodes` do the same for the implicit (SMOOTH/DELAY/TREND) helpers, sharing the `lower_implicit_var` parent->implicit->parse->lower prefix. The compile+symbolize tail (`compile_phase_to_per_var_bytecodes`) lives in `src/db/assemble.rs`. -- **`src/db/assemble.rs`** (a `db` submodule) - Module/simulation assembly: the table/metadata extraction helpers (`extract_tables_from_source_var`, `build_module_inputs`, `build_stub_variable`, `build_submodel_metadata`), the per-variable compile+symbolize tail (`compile_phase_to_per_var_bytecodes`, the `VarFragmentResult`/`PerVarOffsetMap` values), the production element-graph source `var_phase_symbolic_fragment_prod` (the layout-independent `SymVarRef` substrate the cycle gate's element-order probe consumes), the resolved recurrence-SCC interleaver (`segment_member_by_element` / `combine_scc_fragment`, the per-element-granular generalization of `concatenate_fragments`), the salsa-tracked `assemble_module` / `assemble_simulation`, module-instance enumeration (`enumerate_module_instances`), and the flattened-offset map builder (`calc_flattened_offsets_incremental`). `assemble_module` injects each resolved SCC's combined per-element fragment at the first member's runlist slot (the dt fragment into flows, the init fragment as one synthetic-ident `SymbolicCompiledInitial`), preserving per-write `SymVarRef` identity so layout offsets are unchanged. `build_submodel_metadata` registers not only a sub-model's explicit/implicit source variables but -- when `ltm_enabled` -- its LTM synthetic vars (and their implicit helpers) at their `compute_layout` offsets, so a parent fragment's cross-module reference to a sub-model LTM var resolves the same way the full flattened-offset assembly does. This is what lets the exhaustive-mode input→macro link score (the composite-reference form `"{module}·$⁚ltm⁚composite⁚{port}"`) compile in the standalone fragment compiler (`compile_ltm_equation_fragment`) instead of stubbing to a constant 0 (GH #548): the composite link score through a SMOOTH/DELAY/TREND/NPV macro is the max-magnitude path score through the macro (LTM ref 6.3/6.4), computed by the sub-model's `$⁚ltm⁚composite⁚{port}` aux. +- **`src/db/fragment_compile.rs`** (a `db` submodule) - The *emission half* of per-variable compilation (the *lowering half* is the sibling `src/db/var_fragment.rs`). `compile_var_fragment` is the salsa-tracked per-variable fragment compiler; `compile_implicit_var_fragment`/`compile_implicit_var_phase_bytecodes` do the same for the implicit (SMOOTH/DELAY/TREND) helpers, sharing the `lower_implicit_var` parent->implicit->parse->lower prefix. The emission tail (`compile_phase_to_per_var_bytecodes`) lives in `src/db/assemble.rs` and is the **single fragment emission entry point** every emitter -- explicit, implicit, and both LTM ones -- goes through (GH #964). Neither half assigns offsets any more: lowering hands codegen a `Vec` whose variable references are names, so the private per-fragment layout and the reverse offset map that used to undo it are gone. Runlist membership is read through the `var_runlist_membership` projection, not the whole `ModelDepGraphResult`, so an unrelated variable add/delete/rename no longer invalidates every fragment in the model. +- **`src/db/assemble.rs`** (a `db` submodule) - Module/simulation assembly: the table/metadata extraction helpers (`extract_tables_from_source_var`, `build_module_inputs`, `build_stub_variable`, `build_submodel_metadata`), the per-variable emission tail (`compile_phase_to_per_var_bytecodes`, the SINGLE fragment emission entry point since GH #964's stage 2; `fragment_emit_ctx`, which assembles the phase-invariant `compiler::ModuleCtx` its callers borrow into it; the `VarFragmentResult`/`PerVarSizes` values; and `temp_sizes_by_id`, the temp-id-ORDERED flattening of the temp-size map), the production element-graph source `var_phase_symbolic_fragment_prod` (the layout-independent `SymVarRef` substrate the cycle gate's element-order probe consumes), the resolved recurrence-SCC interleaver (`segment_member_by_element` / `combine_scc_fragment`, the per-element-granular generalization of `concatenate_fragments`), the salsa-tracked `assemble_module` / `assemble_simulation`, module-instance enumeration (`enumerate_module_instances`), and the flattened-offset map builder (`calc_flattened_offsets_incremental`). Until stage 2 there were THREE hand-copied emitters (this one plus two inline copies in `src/db/ltm/compile.rs`), each building a stand-in one-variable `compiler::Module` by struct literal and deep-cloning `offsets`/`tables`/`dimensions`/`dimensions_ctx`/`module_refs` per phase; the temp-size ordering fix below had to be applied to all three at once, which is what a mirror family costs. `module_refs` and `runlist_order` turned out to be read by no line of codegen at all, so collapsing the copies also deleted `build_caller_module_refs` -- a whole second per-variable dependency walk whose only consumer was those dead fields. `temp_sizes_by_id`'s ordering is load-bearing because `PerVarBytecodes` is a salsa-cached value with a derived `PartialEq`, so a `HashMap`-iteration-order flip made an identical fragment compare unequal, defeating backdating and making the compiled artifact irreproducible run to run, GH #595's class; `FragmentMerger::absorb` folds the entries order-independently, which is why the defect had no bytecode consequence and survived undetected. `assemble_module` injects each resolved SCC's combined per-element fragment at the first member's runlist slot (the dt fragment into flows, the init fragment as one synthetic-ident `SymbolicCompiledInitial`), preserving per-write `SymVarRef` identity so layout offsets are unchanged. `build_submodel_metadata` registers not only a sub-model's explicit/implicit source variables but -- when `ltm_enabled` -- its LTM synthetic vars (and their implicit helpers) at their `compute_layout` offsets, so a parent fragment's cross-module reference to a sub-model LTM var resolves the same way the full flattened-offset assembly does. This is what lets the exhaustive-mode input→macro link score (the composite-reference form `"{module}·$⁚ltm⁚composite⁚{port}"`) compile in the standalone fragment compiler (`compile_ltm_equation_fragment`) instead of stubbing to a constant 0 (GH #548): the composite link score through a SMOOTH/DELAY/TREND/NPV macro is the max-magnitude path score through the macro (LTM ref 6.3/6.4), computed by the sub-model's `$⁚ltm⁚composite⁚{port}` aux. - **`src/db/analysis.rs`** - Salsa-tracked causal graph analysis: `model_causal_edges`, `model_loop_circuits`, `model_cycle_partitions`, `model_detected_loops`. Element-level tracked functions: `model_element_causal_edges` (emits per-reference element edges via `emit_edges_for_reference`, driven by the `db::ltm_ir` classification IR: each reference's access shape -- `Bare`, `FixedIndex(elements)`, `PerElement { axes }` (the GH #525 iterated+literal mix, expanded as the diagonal-with-pinned-axes rows from the shared `read_slice_rows` derivation, broadcast over unshared target dims), `Wildcard`, or `DynamicIndex` -- and its `routing` (`Direct` or `ThroughAgg`) come from `model_ltm_reference_sites`; a `ThroughAgg` reference is routed through a synthetic `$⁚ltm⁚agg⁚{n}` aggregate node by `emit_agg_routed_edges` -- only the rows the reducer's `read_slice` reads: `source[,,] → agg[]` then `agg[] → to[e]` (`agg.result_dims` drives the fan-out, diagonal/broadcast like `Bare`; an `Iterated` axis's slot coordinate is remapped to the corresponding TARGET-dim element via `ltm_agg::iterated_axis_slot_elements` when the axis is a positionally-mapped pair -- GH #534, identity in the literal case), never the all-pairs N×M cross-product; a whole-extent reducer (all-`Reduced`) degenerates to "every source element → scalar agg → every target element". `emit_agg_routed_edges` derives the agg's result-axis dimensions from `AggNode::result_dims` (resolved against `from` ⌢ `to` dims), so it handles a *scalar* feeder of a (possibly arrayed) hoisted reducer (`scale` in `growth[D1] = SUM(matrix[D1,*] * scale)`): `from_dims.is_empty()` ⇒ emit `from → agg[]` (or the bare `from → agg` when the agg is scalar) rather than the malformed `from[]` node the row-layout machinery would mint. `Direct` `DynamicIndex` references (`arr[i+1]`, ranges, and the not-hoistable dynamic-index reducer carve-out `SUM(pop[idx,*])` -- reclassified from `Wildcard` by the IR) still expand to the conservative cross-product. A `Bare` edge between differently-named but MAPPED dimensions (`x[Region] → target[State]` with a `State→Region` mapping) projects the mapping's element-correspondence DIAGONAL via `expand_same_element` + `DimensionsContext::mapped_element_correspondence` -- one source element per target element, not the `Region × State` cross-product (GH #527) -- WHEN a usable correspondence exists (positional mappings only; explicit element maps are declined since execution resolves positionally, see the helper's gate); an unmapped or element-mapped disjoint-named pair keeps the broadcast), `model_element_loop_circuits` (Johnson's algorithm on element graph; legacy path retained for non-LTM consumers), `model_element_cycle_partitions` (stock-to-stock SCCs at element granularity). Tiered enumeration: `model_edge_shapes` (per-`(from, to)` `BTreeSet` projected from the IR's classified sites) and `model_loop_circuits_tiered` (variable-level Johnson + cycle classification + slow-path-subgraph Johnson, keeping synthetic agg nodes in the slow-path subgraph projection so cross-element loops through a hoisted reducer are recovered) replace the full-element Johnson run for LTM compilation. `classify_cycle` labels each variable-level cycle as `PureScalar`, `PureSameElementA2A`, or `CrossElementOrMixed`; pure cycles emit one Loop directly while cross-element / mixed cycles drive the slow-path subgraph (the induced element subgraph over their nodes). `TieredCircuitsResult.slow_path_largest_scc` exposes the cross-element subgraph's largest SCC for the LTM auto-flip gate. Produces `DetectedLoop` structs with polarity (each carrying an optional `name`: `None` for an enumerated loop, `Some(label)` for a modeler-pinned one, plus a `partition: Option` -- a RESULT-SCOPED dense index into `DetectedLoopsResult.partitions` (reusing the discovery surface's `ltm_finding::DiscoveredPartition`), `None` for a loop with no parent-level stock; same stability caveat as `FoundLoop.partition`, GH #685). `model_detected_loops` builds its exhaustive loop set with the SAME machinery the scored surface uses -- `model_loop_circuits_tiered` feeding `db::ltm::build_loops_from_tiered` (under the shared `cross_agg_loop_budget`) plus the shared `recover_agg_hop_polarities` pass -- so the detected and scored loop sets, polarities, and ids are identical BY CONSTRUCTION for every model shape (GH #746; the runtime id join `reclassify_loops_from_results` / pysimlin `get_relative_loop_score` reads `$⁚ltm⁚loop_score⁚{id}` keyed purely on the id, and the previous variable-level-Johnson + per-agg-splice path only bijected for all-scalar cycles -- arrayed cycles silently joined another loop's series). Cross-element loops therefore surface per element instance with element-subscripted variables, mirroring cross-element pins. It resolves cycle partitions over the ELEMENT-level graph (`model_element_cycle_partitions` -- the same granularity the scored `loop_partitions` and the discovery surface use), and `resolve_loop_partitions` mirrors `ltm_finding::attach_partition_metadata` exactly (first-appearance dense remap, `DiscoveredPartition { stocks, loop_count }`) so the two surfaces report partitions in the same shape AND at the same granularity: the partition stock SETS are a usable cross-surface key for scalar and arrayed models alike (an A2A loop's single `partition` index is its first resolving slot's partition; the scored surface's `loop_partitions[id]` carries the full per-slot vector). It appends the model's *pinned* loops (`db::ltm::model_pinned_loops`, deduped against the enumerated set by canonical variable-cycle rotation; a pin that duplicates an enumerated loop transfers its `name` onto the surviving enumerated `DetectedLoop` rather than being discarded), and pinned loops get a partition too. It consults the SHARED `db::ltm::model_ltm_mode` query for the exhaustive-vs-discovery decision -- the SAME gate `model_ltm_variables` uses -- so the two query surfaces never disagree: in discovery mode (whether the user forced `ltm_discovery_mode`, the variable-level SCC exceeded `MAX_LTM_SCC_NODES`, or the slow-path late-flip fired) it returns *only* the pinned loops, so a pinned loop surfaces through `simlin_analyze_get_loops` even in discovery mode (and is backed by the `loop_score` that mode actually emits). Before this unification `model_detected_loops` gated only on the `causal_graph_with_modules` SCC size and ignored both the user flag and the slow-path flip, so a small user-forced-discovery model dropped the pin entirely. `RefShape` and `emit_edges_for_reference` (plus the element-name expansion helpers) live here; the AST walker / agg-routing decision lives in `src/db/ltm_ir.rs`. - **`src/db/ltm_ir.rs`** (a `db` submodule, a child of `db.rs` kept in its own file purely for the per-file line cap; like `ltm_agg`, it builds on `crate::db`) - The single salsa-tracked place a causal edge's access shape *and* aggregate-node routing are decided. `model_ltm_reference_sites(db, model, project) -> LtmReferenceSitesResult` walks each variable's `Expr2` AST once, consults `enumerate_agg_nodes` (the sole "hoistable maximal reducer" decider), and buckets every `Var`/`Subscript` reference by its `(from, to)` causal edge into a `Vec` (`shape` + `target_element` + `routing ∈ {Direct, ThroughAgg{agg}}`); the byte-identical `route_through_agg = !routed_aggs.is_empty() && in_reducer` decision and the `aggs_in_var(to).filter(is_synthetic && reads from)` filter exist here and nowhere else. `model_element_causal_edges`, `model_edge_shapes`, and `model_ltm_variables` are pure readers. Also hosts the AST-walker helpers moved out of `src/db/analysis.rs` (`collect_reference_sites` / `classify_subscript_shape` / `resolve_literal_index` / `collect_reference_shapes`); `classify_subscript_shape` carries the AC1.4 fix -- a subscript whose indices are *all* `Wildcard`/`StarRange` (the reducer-style whole-extent access, e.g. `SUM(x[*:Dim])`) classifies as `Wildcard` to agree with `enumerate_agg_nodes`'s read-slice hoisting test. `classify_iterated_dim_shape` consumes `ltm_agg::classify_axis_access` per axis (T6 of the shape-expressiveness design; a `Reduced` result is post-filtered to `None` -- a non-reducer reference never collapses an axis): an ALL-`Iterated` subscript -- indices that are exactly the target equation's iterated dimensions, position-matched to the source's declared dims by name or by a positional mapping in EITHER declaration direction (GH #511/#527/#757) -- classifies `Bare` (a same-element-on-shared-dims reference projected via `expand_same_element`); a MIXED `Iterated`+`Pinned` subscript (`pop[Region, young]` inside an A2A-over-`Region` equation, GH #525) classifies `RefShape::PerElement { axes }`, whose element edges are the diagonal-with-pinned-axes rows from the shared `read_slice_rows` derivation (never the cross-product whose phantom circuits carried silent confident loop scores) and whose link scores are per-(row, FULL-target-element) scalars `$⁚ltm⁚link_score⁚{from}[{row}]→{to}[{e}]` (the row a function of `e`: project `e` onto the Iterated axes, fill Pinned with literals -- including the BROADCAST case where the Iterated dims are a strict subset of the target's), emitted by `emit_per_element_link_scores`/`generate_per_element_link_equation` and routed per-circuit by the loop builder's `is_per_element_edge` predicate (keyed on the same `model_edge_shapes` IR projection); an all-`Pinned` subscript falls through to `classify_subscript_shape`'s `FixedIndex` (so every existing Bare/FixedIndex link-score name is untouched). A *partially*-iterated subscript that is a *reducer argument* (`SUM(matrix[D1,*])`, `SUM(pop[NYC,*])`, `SUM(matrix3d[D1,NYC,*])`) is hoisted into a synthetic agg by `enumerate_agg_nodes` (its read slice is statically describable), so the reference is `ThroughAgg`-routed and its (`Wildcard`) shape is ignored; the only `Direct` `Wildcard` reducer references remaining are a *whole-RHS* variable-backed reducer's argument (`total = SUM(population[*])`), which keeps `Wildcard`, a DE-HOISTED array-valued reducer's wildcard arg (`RANK(pop[*], 1)` -- GH #771: RANK never sets `in_reducer`, so the site keeps `Wildcard` and takes the conservative cross-product), and a not-hoistable reducer's argument -- a *dynamic index* (`SUM(pop[idx,*])`, `idx` non-literal) or an ELEMENT-mapped sliced reducer the correspondence declines (GH #756; the positionally-mapped case is hoisted since GH #534 -- both declaration directions since GH #757 -- with the `Iterated` axis carrying the (target, source) dim pair). `model_ltm_reference_sites` reclassifies such a not-hoisted `Direct` `Wildcard` `in_reducer` site as `DynamicIndex` (#514) so the `Wildcard`-shape conservative cross-product never fires from a `Direct` site that *could* have been hoisted (the de-hoisted RANK arg is not hoist-eligible, so it deliberately stays `Wildcard`). (`classify_iterated_dim_shape`'s mapped branch -- a *whole-equation*-iterated subscript like `x[State]` inside `target[State] = x[State] * c`, not a sliced reducer argument -- is a separate path and still classifies as `Bare`.) The mapped case needs a `DimensionsContext` (built from `project_datamodel_dims`), so the IR is recomputed when a dimension's mappings change. - **`src/db/ltm_ir_tests.rs`** - Tests for the reference-site IR: the per-AST-site `(shape, in_reducer)` contract (the `ref_site_*` regression guards, ported from `src/db/analysis.rs`) plus the public `ClassifiedSite` contract -- `(shape, target_element, routing)` per site, the AC1.4 `StarRange` consistency, and the AC1.5 SIZE / scalar-source-reducer `Direct` routing, each cross-checked against `enumerate_agg_nodes`. - **`src/db/macro_registry.rs`** (a `db` submodule, like `db::ltm_ir`, only to keep `db.rs` under the per-file line cap) - The per-project macro-registry salsa query. `project_macro_registry(db, project) -> MacroRegistryResult` wraps the pure `module_functions::MacroRegistry` (resolver) and exposes the build error. Registry-build *validation* (recursion cycle, duplicate macro name, macro/model name collision, a module inside a macro body) can't be re-derived from the canonical-name-keyed `SourceProject.models` (it collapses duplicate / colliding model names), so the query is demand-driven from two salsa inputs: the ordered pre-dedup `SourceProject.macro_declarations` list (canonical name + `macro_spec`, one entry per project model in declaration order) drives Passes 1-2 (duplicate macro name, macro/model collision), and the macro-marked models' bodies -- read from `models` -- drive Pass 3 (recursion cycle, from the equations) and Pass 4 (a `Variable::Module` in a macro body, from the variable kinds). `build_error_from_inputs` reconstructs a `Vec` in declaration order (canonical name + spec + macro bodies; non-macro bodies empty, which Passes 1-2 don't read and Passes 3-4 skip) and calls the UNCHANGED `MacroRegistry::build` on it, so the produced typed `(ErrorCode, message)` is byte-identical to building over the original datamodel `Vec`. Because the query reads only macro_declarations + macro-marked models' bodies (it skips `macro_spec.is_none()` models' bodies), an ordinary equation edit does NOT invalidate it. **On a build failure the returned registry is EMPTY, and that is load-bearing for CYCLE SAFETY, not just for error quality**: `collect_all_diagnostics` runs every model's passes after emitting `build_error`, and `analysis::analyze_model` never reads `build_error` at all, so only the empty registry stops a rejected macro's call sites from re-synthesizing the implicit module edge `db::project_module_graph` cannot see (a process abort under `panic=abort`). A "keep a partial registry alongside the error" refactor reopens that hole; the argument and its pinning test are on the query's rustdoc. It does NOT accumulate: `build_error` is a plain memoized value that its two consumers read directly -- `compile_project_incremental` (to fail with a clear message) and `collect_all_diagnostics` (to emit the one project-level `Diagnostic`). Accumulating it from inside the query body was wrong twice over: every model's `model_all_diagnostics` subtree reaches this query through `model_module_ident_context`, so the DFS reported N identical copies for an N-model project, and after an unrelated revision bump the deep-verify path recomputed the pruning flags as `Empty` and the diagnostic vanished from the collection entirely. `enclosing_macro_for_var` resolves a macro-body variable's enclosing-macro name (#554, the renamed-intrinsic precedence). -- **`src/db/dep_graph.rs`** (a `db` submodule, like `db::ltm_ir` / `db::macro_registry`, only to keep `db.rs` under the per-file line cap) - Owns the **model dependency-graph cycle gate** and its shared relations. The production gate `model_dependency_graph_impl` lives here (the transitive-closure DFS `compute_transitive`/`compute_inner`; the SCC-aware back-edge break `same_resolved_scc`; the SCC-as-collapsed-node transitive accumulation -- every member of a resolved recurrence SCC ends with the identical member-free union of the SCC's *external* successors so the topo sort never re-sees the intra-SCC cycle; and the SCC-contiguous topological runlist sort `topo_sort_str` so a resolved SCC's members are emitted as one byte-stable contiguous block at the SCC's slot for Phase-2-Task-6 combined-fragment injection). **Deterministic initials runlist**: the init set is a `HashSet`, so `model_dependency_graph_impl` sorts it (`init_list.sort_unstable()`) before `topo_sort_str` -- `topo_sort_str` breaks ties (variables with no ordering edge between them) by visit order, so without the sort the same model compiled twice produced different init orderings and therefore different initial values for any unordered pair (GH #595 tracks the deeper soundness gap; the Flows/Stocks runlists are already deterministic by filtering the pre-sorted `var_names`). **dt stock-submodel-output chain-break**: `build_var_info` filters a `submodel·subvar` dt dependency whose `subvar` is a Stock out of the dt phase (a stock breaks the chain, read from the prior timestep) for **every** reader -- not just module-kind variables -- mirroring the legacy `model.rs::module_output_deps` gate; a non-module reader of a stock submodel output (e.g. `v = SMOOTH(...)·output` where the output is an INTEG stock) would otherwise gain a spurious same-step `reader -> module` edge that `topo_sort_str` breaks arbitrarily, sometimes emitting the module before its input (a stale read each flows step). The init phase keeps the edge (stocks do not break the chain there), so only `dt_deps` are filtered. The dep-graph result types (`SccPhase`, `ResolvedScc`, `ModelDepGraphResult`) and the thin `#[salsa::tracked]` wrapper `crate::db::model_dependency_graph` (keyed on an interned `ModuleInputSet`, empty = the no-inputs case) live here too -- the wrapper delegates straight to `model_dependency_graph_impl` -- and are re-exported at the `db.rs` root. Also holds the single shared **cycle relation** `walk_successors(var_info, name, phase: SccPhase) -> Vec<&Ident>` parameterized by phase: Module/absent ⇒ `[]` in both phases; a Stock ⇒ `[]` in `Dt` only (a stock is a dt sink but NOT an init sink -- the one per-phase difference); otherwise the phase's dep set (`dt_deps` for `Dt`, `initial_deps` for `Initial`) filtered to known targets, with stock-targeted deps dropped in `Dt` only, module-targets kept -- exactly the successor set `compute_inner` iterates per phase. (`SccPhase` is defined here; it already keys `ResolvedScc.phase`/`combine_scc_for_phase`, so the dt/init distinction is the same one this relation makes.) Plus the shared `VarInfo` map builder `build_var_info` (consumed verbatim by `model_dependency_graph_impl` and the `#[cfg(test)]` SCC accessor, so the accessor observes the *exact* `var_info` the engine builds -- never a reconstruction), and the recurrence-SCC element-acyclicity refinement `resolve_recurrence_sccs` / `refine_scc_to_element_verdict` / `symbolic_phase_element_order` (GH #575: the cross-member-comparable symbolic `SymVarRef` element graph; N=1 self-recurrence is just the 1-member case). Defining the relation once and using it in both the gate and the accessor makes the accessor's relation the engine's relation by construction. Also hosts the `#[cfg(test)]` SCC accessor: `dt_cycle_sccs` (uncapped `crate::ltm::scc_components` Tarjan over the `walk_successors(.., SccPhase::Dt)` adjacency → `DtCycleSccs { multi, self_loops }`, sorted/byte-stable), the pure `dt_cycle_sccs_consistency_violation` predicate (functional core), and `dt_cycle_sccs_engine_consistent` (imperative shell -- cross-checks the instrumented SCC set against the engine's real `CircularDependency` flagging on the same compiled model, panics on divergence), plus `array_producing_vars` / `var_noninitial_lowered_exprs` (the set of variables whose engine-lowered non-initial exprs contain an array-producing builtin, over the same `build_var_info` universe). +- **`src/db/dep_graph.rs`** (a `db` submodule, like `db::ltm_ir` / `db::macro_registry`, only to keep `db.rs` under the per-file line cap) - Owns the **model dependency-graph cycle gate** and its shared relations. The production gate `model_dependency_graph_impl` lives here (the transitive-closure DFS `compute_transitive`/`compute_inner`; the SCC-aware back-edge break `same_resolved_scc`; the SCC-as-collapsed-node transitive accumulation -- every member of a resolved recurrence SCC ends with the identical member-free union of the SCC's *external* successors so the topo sort never re-sees the intra-SCC cycle; and the SCC-contiguous topological runlist sort `topo_sort_str` so a resolved SCC's members are emitted as one byte-stable contiguous block at the SCC's slot for Phase-2-Task-6 combined-fragment injection). **Deterministic initials runlist**: the init set is a `HashSet`, so `model_dependency_graph_impl` sorts it (`init_list.sort_unstable()`) before `topo_sort_str` -- `topo_sort_str` breaks ties (variables with no ordering edge between them) by visit order, so without the sort the same model compiled twice produced different init orderings and therefore different initial values for any unordered pair (GH #595 tracks the deeper soundness gap; the Flows/Stocks runlists are already deterministic by filtering the pre-sorted `var_names`). **dt stock-submodel-output chain-break**: `build_var_info` filters a `submodel·subvar` dt dependency whose `subvar` is a Stock out of the dt phase (a stock breaks the chain, read from the prior timestep) for **every** reader -- not just module-kind variables (`model.rs::module_output_deps` used to state the same rule; GH #568 deleted that walk, so this is now the only statement of it); a non-module reader of a stock submodel output (e.g. `v = SMOOTH(...)·output` where the output is an INTEG stock) would otherwise gain a spurious same-step `reader -> module` edge that `topo_sort_str` breaks arbitrarily, sometimes emitting the module before its input (a stale read each flows step). The init phase keeps the edge (stocks do not break the chain there), so only `dt_deps` are filtered. The per-variable projection `var_runlist_membership` -> `RunlistMembership` lives here too: it returns only the three "is this variable in the initials / flows / stocks runlist" bits `compile_var_fragment` reads, so it backdates and an unrelated dep-graph change no longer invalidates every fragment (GH #964). The dep-graph result types (`SccPhase`, `ResolvedScc`, `ModelDepGraphResult`) and the thin `#[salsa::tracked]` wrapper `crate::db::model_dependency_graph` (keyed on an interned `ModuleInputSet`, empty = the no-inputs case) live here too -- the wrapper delegates straight to `model_dependency_graph_impl` -- and are re-exported at the `db.rs` root. Also holds the single shared **cycle relation** `walk_successors(var_info, name, phase: SccPhase) -> Vec<&Ident>` parameterized by phase: Module/absent ⇒ `[]` in both phases; a Stock ⇒ `[]` in `Dt` only (a stock is a dt sink but NOT an init sink -- the one per-phase difference); otherwise the phase's dep set (`dt_deps` for `Dt`, `initial_deps` for `Initial`) filtered to known targets, with stock-targeted deps dropped in `Dt` only, module-targets kept -- exactly the successor set `compute_inner` iterates per phase. (`SccPhase` is defined here; it already keys `ResolvedScc.phase`/`combine_scc_for_phase`, so the dt/init distinction is the same one this relation makes.) Plus the shared `VarInfo` map builder `build_var_info` (consumed verbatim by `model_dependency_graph_impl` and the `#[cfg(test)]` SCC accessor, so the accessor observes the *exact* `var_info` the engine builds -- never a reconstruction), and the recurrence-SCC element-acyclicity refinement `resolve_recurrence_sccs` / `refine_scc_to_element_verdict` / `symbolic_phase_element_order` (GH #575: the cross-member-comparable symbolic `SymVarRef` element graph; N=1 self-recurrence is just the 1-member case). Defining the relation once and using it in both the gate and the accessor makes the accessor's relation the engine's relation by construction. Also hosts the `#[cfg(test)]` SCC accessor: `dt_cycle_sccs` (uncapped `crate::ltm::scc_components` Tarjan over the `walk_successors(.., SccPhase::Dt)` adjacency → `DtCycleSccs { multi, self_loops }`, sorted/byte-stable), the pure `dt_cycle_sccs_consistency_violation` predicate (functional core), and `dt_cycle_sccs_engine_consistent` (imperative shell -- cross-checks the instrumented SCC set against the engine's real `CircularDependency` flagging on the same compiled model, panics on divergence), plus `array_producing_vars` / `var_noninitial_lowered_exprs` (the set of variables whose engine-lowered non-initial exprs contain an array-producing builtin, over the same `build_var_info` universe). - **`src/db/dep_graph_tests.rs`** - Tests for the dt- *and init*-phase cycle-relation primitives, the `#[cfg(test)]` SCC accessor, the recurrence-SCC element-acyclicity verdict, and the multi-member symbolic builder, in their own file alongside the production code to keep `db.rs`/`src/db/tests.rs` under the per-file line cap: the dt-phase `walk_successors` invariant per node kind (Stock/Module/Aux/absent, BTreeSet-sorted order), `dt_cycle_sccs` integration (clean DAG / two-node cycle / self-loop, each via the consistency-checked accessor), byte-stability, the pure consistency predicate in both divergence directions plus all consistent pairings, and `array_producing_vars` membership over the four positive/negative cases. Phase 2 adds: the init-phase `walk_successors` invariant (a stock is NOT an init sink, modules omitted, unknown-dep filtering, BTreeSet-sorted order) and the init-phase relation/verdict tests (`resolve_init_*` / `init_recurrence_behind_stock_*` / `two_stock_init_*` -- a stock-broken dt chain with a forward init element recurrence resolves init-only, a genuine init element cycle stays `CircularDependency`, and a both-relations aux self-recurrence is not double-resolved as a separate init SCC); the multi-member symbolic-builder / GH #575 regression guards (`resolve_dt_two_member_ref_shaped_scc_resolves_interleaved`, the genuine multi-variable element 2-cycle / scalar 2-cycle staying `Unresolved` via the symbolic builder, the N=1 self-recurrence remaining byte-identical to Phase 1, an unsourceable member ⇒ `Unresolved` with no panic, and the `var_phase_symbolic_fragment_prod` no-panic contract); and the combined-fragment preconditions (`model_dep_graph_*` -- a resolved multi-member SCC survives the dependency graph with `has_cycle == false`, members carry the SCC's external deps, and the SCC's members are emitted as one byte-stable contiguous runlist block even with an interposing external variable, so Task 6's combined-fragment injection lands in correct relative order). - **`src/db/var_fragment.rs`** (a `db` submodule, like `db::dep_graph` / `db::ltm_ir` / `db::macro_registry`, only to keep `db.rs` under the per-file line cap) - The *lowering* half of per-variable compilation. `lower_var_fragment` parses the source variable, lowers its equation, builds the minimal symbol-table/metadata `Context`, and runs `compiler::Var::new` per phase to yield the lowered `Vec`; the bytecode-emission half (`compile_phase`) stays with the salsa-tracked caller `crate::db::compile_var_fragment`. The split exists because the lowered `Vec` (which does not implement `salsa::Update`) is the reuse surface the cycle-gate's element-order probe needs the engine's *own* production lowering of (no reconstruction), and a plain function cannot accumulate salsa diagnostics -- so diagnostics are returned as data (`LoweredVarFragment`, with `Fatal` aborting the variable and a per-phase `Err` dropping only that phase) and replayed by the caller. -- **`src/db/invariance.rs`** (a `db` submodule) - The per-model run-invariance classification query (time-invariant hoisting, GH #712, stage B1). `model_flows_invariant(db, model, project, is_root, module_inputs) -> Arc>` is the imperative shell around the pure `compiler::invariance` classifier: it iterates `model_dependency_graph`'s `runlist_flows` (a topological order, so a single ordered pass reaches a fixpoint -- every dep is classified before its reader) and, for each flow var, reads the `FlowInvarianceSupport` precomputed on the salsa-cached `compile_var_fragment` result (`assemble::compute_flow_invariance_support`: `locally_pure` = the shared classifier with an all-Invariant offset callback, catching variant builtins; `dep_names` = `collect_expr_offsets` over the FLOW exprs reverse-mapped through the mini-layout, skipping `INIT()` args). The fixpoint is `invariant(v) iff locally_pure(v) && dep_names(v) subset-of invariant-set` -- no re-lowering, so classification adds ~0 compile cost. Conservative: only the ROOT module is classified (submodules return empty); resolved-recurrence-SCC members, stocks, modules, table-only holders, and implicit/LTM synthetic vars are excluded (variant). `assemble_module` reads this set to **partition the flow runlist** [invariant..., dynamic...] (stable, relative order preserved) and records the invariant-prefix opcode length as `CompiledModule.flows_invariant_opcode_len` (0 for submodules / no-invariant models). B1 is behavior-neutral (the reordered single flow program is unchanged for the VM and wasmgen); B2 will run the invariant prefix once per `run_to`. The soundness oracle (`tests/integration/simulate.rs` `oracle_*`, `CompiledSimulation::invariant_flow_offsets`) asserts every hoisted slot is bit-constant across all saved steps (C-LEARN: 868 slots, 0 violations; WORLD3: 78, 0). +- **`src/db/invariance.rs`** (a `db` submodule) - The per-model run-invariance classification query (time-invariant hoisting, GH #712, stage B1). `model_flows_invariant(db, model, project, is_root, module_inputs) -> Arc>` is the imperative shell around the pure `compiler::invariance` classifier: it iterates `model_dependency_graph`'s `runlist_flows` (a topological order, so a single ordered pass reaches a fixpoint -- every dep is classified before its reader) and, for each flow var, reads the `FlowInvarianceSupport` precomputed on the salsa-cached `compile_var_fragment` result (`assemble::compute_flow_invariance_support`: `locally_pure` = the shared classifier with an all-Invariant reference callback, catching variant builtins; `dep_names` = `collect_expr_refs` over the FLOW exprs, reading the owning variable's name straight off each `VarRef` and skipping `INIT()` args -- it used to reverse-map slot offsets through a private per-fragment layout, guarded by a `debug_assert!` against silently dropping a dependency, and GH #964 removed the lookup rather than the guard). The fixpoint is `invariant(v) iff locally_pure(v) && dep_names(v) subset-of invariant-set` -- no re-lowering, so classification adds ~0 compile cost. Conservative: only the ROOT module is classified (submodules return empty); resolved-recurrence-SCC members, stocks, modules, table-only holders, and implicit/LTM synthetic vars are excluded (variant). `assemble_module` reads this set to **partition the flow runlist** [invariant..., dynamic...] (stable, relative order preserved) and records the invariant-prefix opcode length as `CompiledModule.flows_invariant_opcode_len` (0 for submodules / no-invariant models). B1 is behavior-neutral (the reordered single flow program is unchanged for the VM and wasmgen); B2 will run the invariant prefix once per `run_to`. The soundness oracle (`tests/integration/simulate.rs` `oracle_*`, `CompiledSimulation::invariant_flow_offsets`) asserts every hoisted slot is bit-constant across all saved steps (C-LEARN: 868 slots, 0 violations; WORLD3: 78, 0). +- **`src/db/fragment_char_tests.rs`** + **`src/db/fragment_char_golden/`** - Characterization pins for the per-variable fragment compiler (the gate GH #964's round-trip deletion is measured against), the fragment-level analogue of `src/db/ltm_char_tests.rs`. Twelve fixtures (scalar chain + stock; the three arrayed equation shapes incl. an EXCEPT default; static + dynamic subscripts; a RUNTIME array view + an arrayed stock; `SUM` plus the array-producing `VECTOR SORT ORDER`/`RANK`/`VECTOR ELM MAP`; scalar and per-element arrayed graphical functions; a standalone lookup-only table with two callers; an explicit sub-model instance rendered at BOTH its input-agnostic and its real instance input set plus a `SMTH1` call; `PREVIOUS`/`INIT` in their direct-opcode and capture-helper forms; a resolved recurrence SCC whose combined fragment is rendered too; and an LTM model in BOTH loop-enumeration modes) each assert three independent things, since no one of them subsumes another: a golden of the rendered `PerVarBytecodes` (opcode stream with `SymVarRef` operands, literal pool, GF blocks, module decls, static views, temp sizes, dim lists) plus the model's `compute_layout` body layout and the VM result table (`UPDATE_FRAGMENT_GOLDEN=1` regenerates, mirroring `UPDATE_LTM_GOLDEN`); a hand-written, never-regenerated EXHAUSTIVE `variable -> compiled phases` map, because a golden that pins an artifact is structurally blind to that artifact being stably ABSENT and a careless re-capture would bless a vanished fragment; and hand-computed VM spot checks, which stay meaningful when a later commit legitimately changes bytecode shape. The opcode renderer's match is exhaustive with no `_` arm, so a new `SymbolicOpcode` variant cannot enter a golden unrendered; the fixtures reach ALL NINE `SymVarRef`-carrying opcodes -- there were eleven until the two codegen could not emit (`PushVarView`, plain `AssignNext`) were deleted outright, so exhaustive coverage is now a property of the suite rather than a coincidence. Building the runtime-view fixture surfaced and fixed a genuine bug: `codegen::full_source_len` had no arm for a DYNAMIC `Expr::Subscript`, so `VECTOR ELM MAP(arr[i], offsets)` bounded its reads by 1 element and returned NaN for every element whenever `i` selected anything but `arr`'s first (pinned by `array_tests::...::elm_map_dynamic_source_subscript_uses_full_variable_extent_vm`). The file additionally holds the **incrementality** measurements, which use `#[cfg(test)]` per-thread execution records (`db::fragment_compile`'s `reset_fragment_executions`/`fragment_executions`) rather than memo pointers -- a fragment is *designed* to be layout-independent, so it compares equal across a layout edit whether or not the compile re-ran, salsa backdates on that equality, and every pointer-based check passes either way. What they assert: an equation-only edit recompiles just the edited variable plus its direct consumers (one hop, via the dependency-granular mini `ModelStage0`, which parses each dep), and adding, deleting or renaming an UNRELATED variable recompiles only the variable that actually changed (`["added"]`, nothing, and `["renamed"]` respectively) -- GH #964's "layout-only edits reuse unchanged cached fragments" criterion, now held as a CACHE property and not merely as fragment-value equality. Getting there needed the first two coarse salsa edges narrowed at once (`var_runlist_membership` for the whole-`ModelDepGraphResult` read; `model_variable_by_name` for the whole-`SourceModel::variables` read), since fixing either alone was unmeasurable while the other stood, plus `model_implicit_var_by_name` for the whole-`model_implicit_var_info` read. **The tightness is REAL but BOUNDED, and both bounds are pinned by `implicit_helper_add_is_tight_but_module_helper_add_is_not` rather than described here.** It holds for adding/deleting/renaming a plain aux and for a variable synthesizing a PREVIOUS/INIT helper. It does NOT hold for a variable that instantiates a MODULE (`SMTH1`, `DELAY`, a user sub-model): every pre-existing fragment still recompiles, for two reasons that are not projections away -- `project.models` gains the spliced `stdlib⁚smth1` template, and `model_module_ident_context` is an INTERNED handle whose id changes when the module-ident set grows, which gives every variable's parse a brand-new cache key, and a new key cannot backdate at all. **These counts are C1c's gate** within those bounds: a stage-3 rewrite that reintroduces a model-wide dependency reds here on the count -- before, they were saturated at "everything" and could not have. `compile_implicit_var_fragment` is not a tracked query at all, so every implicit helper recompiles with `assemble_module`; `compile_ltm_var_fragment` is per-`(from, to)`-link and ignores value-only edits, but its *invalidation* on a structural edit is nondeterministic (33 of 40 repetitions of one fixed edit sequence re-execute both of the circuit's link fragments, 7 re-execute one; the returned fragment value is equal before and after in every repetition and salsa backdates it, so it is wasted recompilation rather than a wrong answer -- it reproduces under direct demand, so it is not assembly's map-walk order, and root-causing the residual is a salsa dependency-verification question rather than a fragment-compiler one). +- **`src/db/fragment_determinism_tests.rs`** - Compilation must be a function of its inputs: each test compiles the SAME model on independent fresh databases (12 repeats, since a two-entry `HashMap` comes out in the "right" order about half the time) and asserts byte-identical output. Deliberately independent of the characterization harness -- these pin the determinism FIXES, that pins behavior, and the two must be revertable apart. Covers the three ordering defects a `HashMap` introduced into salsa-cached values, none of which changed a single simulation result (which is how all three survived): `temp_sizes` built from `HashMap::iter` (fixed by `db::assemble::temp_sizes_by_id`; needs a fragment with TWO array-producing builtins, since one temp cannot express an ordering), the `graphical_functions` layout and every `base_gf` operand assigned by iterating `module.tables` in `Compiler::new` (needs two table-bearing variables in one fragment; it reaches shipped models -- `test/metasd/theil-statistics/Theil_2011.mdl` compiles a fragment holding `["dummy_data","dummy_simulation"]` -- and the assembled root `CompiledModule` differed on 18 of 23 repeats before the fix), and the same `temp_sizes` ordering at the SEPARATE LTM call site (an LTM link score is woven from its target's equation, so an arrayed target built from two array-producing builtins gives the synthetic score fragment six temps; reverting either call site alone leaves the other's test green). The LTM *implicit-helper* copy has no reachable temps to order -- its fragments are the `PREVIOUS` capture auxes, which the generators pin to a single element and are therefore always scalar. - **`src/db/combined_fragment_tests.rs`** - Structural tests for `combine_scc_fragment` (segment ordering along `element_order`, write-`SymVarRef` identity preservation, single trailing `Ret`, per-member resource renumbering with no cross-member collision, merged side-channels). Numeric correctness is the end-to-end job of the `ref.mdl`/`interleaved.mdl` simulation tests. In its own file to keep `db.rs`/`src/db/tests.rs` under the per-file line cap. - **`src/db/ltm/`** - LTM (Loops That Matter) variable generation as salsa tracked functions, split into a directory (one large cohesive concern with heavy internal coupling; a directory keeps the internal helpers `pub(super)`/`pub(crate)` *within* the `ltm` subtree without widening to crate scope, and the names that escape -- the `db.rs`-root `pub use ltm::{...}` / `use ltm::*` surface and the `ltm::` paths `src/db/assemble.rs` uses -- are re-exported at `src/db/ltm/mod.rs`): - **`mod.rs`** - the orchestration root: `model_ltm_variables` (the unified entry point), the entry/glue (`ltm_module_idents`, the submodule `use`/`pub use` wiring, the `#[cfg(test)] #[path]` test mount of the sibling `src/db/ltm_tests.rs` -- which stays one level up from the directory, so the relative mount path is one segment up), and `LtmImplicitVarMeta`/`model_ltm_implicit_var_info`. diff --git a/src/simlin-engine/proptest-regressions/compiler/symbolic_merge_proptest.txt b/src/simlin-engine/proptest-regressions/compiler/symbolic_merge_proptest.txt new file mode 100644 index 000000000..ea946dd34 --- /dev/null +++ b/src/simlin-engine/proptest-regressions/compiler/symbolic_merge_proptest.txt @@ -0,0 +1,31 @@ +# Seeds for failure cases proptest has generated in the past. It is +# automatically read and these particular cases re-run before any +# novel cases are generated. +# +# It is recommended to check this file in to source control so that +# everyone who runs the test benefits from these saved cases. +# +# Provenance: every seed below was captured during the C2 mutation experiments +# on `FragmentMerger` (GH #964 row C), not from a production bug. Between them +# they fail under: a resource base that ignores the running merged length +# (M1/M2), a phase `ctx_base` dropped from that base (M8), the sequential concat +# switched to `TempStrategy::Sum` (M5), and an initials-phase accumulator frozen +# in `renumber_initials_phase` (M8). +# +# The two GF seeds cover DIFFERENT halves of M4's gap totality, and the +# distinction is the whole reason they are both here: +# +# * seed 1 (unreferenced block FIRST, referenced second) is the LEADING / +# interior gap -- `gf_blocks_of_fragment`'s `if start > cursor` arm. Nothing +# else in the repository pins it: dropping that arm alone leaves all 12 +# fragment characterization goldens, the entire `file_io` corpus, and both +# `combined_fragment_tests` gap tests green, and the property needs a +# fresh draw to find it. With this seed it fails at case 0. +# * seed 2 (a single unreferenced block) is the TRAILING gap -- the +# `if cursor < gf_len` arm. Two pre-existing `combined_fragment_tests` +# already catch that one; it is kept because it is the cheaper witness and +# it costs nothing to replay. +cc d8964ed6418b3ef65bfe305ad90c5cd67f3b42afab94ae8e24b28c48a298dce0 # shrinks to specs = [FragSpec { tag: 1, n_literals: 0, gf_blocks: [GfBlockSpec { content: 2, len: 1, referenced: false }, GfBlockSpec { content: 0, len: 1, referenced: true }], n_modules: 0, n_views: 0, views_on_temps: false, temp_sizes: [], dim_lists: [], with_loop: false, n_elements: 2 }] +cc 7f1b73ed71c899310dd23ab3b2781ea05decc05060eb94f87c59ad45e0a5a066 # shrinks to specs = [FragSpec { tag: 1, n_literals: 0, gf_blocks: [GfBlockSpec { content: 0, len: 1, referenced: false }], n_modules: 0, n_views: 0, views_on_temps: false, temp_sizes: [], dim_lists: [], with_loop: false, n_elements: 1 }] +cc d241e3281e0a81a70a1029c59eb2d463285ad3efd1baae6bba9c85a56eca79a9 # shrinks to specs = [FragSpec { tag: 1, n_literals: 0, gf_blocks: [], n_modules: 0, n_views: 0, views_on_temps: false, temp_sizes: [], dim_lists: [], with_loop: false, n_elements: 1 }, FragSpec { tag: 2, n_literals: 0, gf_blocks: [], n_modules: 0, n_views: 0, views_on_temps: false, temp_sizes: [], dim_lists: [], with_loop: false, n_elements: 1 }, FragSpec { tag: 3, n_literals: 0, gf_blocks: [], n_modules: 1, n_views: 0, views_on_temps: false, temp_sizes: [], dim_lists: [], with_loop: false, n_elements: 1 }, FragSpec { tag: 4, n_literals: 0, gf_blocks: [], n_modules: 1, n_views: 0, views_on_temps: false, temp_sizes: [], dim_lists: [], with_loop: false, n_elements: 1 }] +cc de8a8fbdc2ba067cfe1a52718ccba734ade5be82ca95222b0665456edb945f0a # shrinks to specs = [FragSpec { tag: 1, n_literals: 0, gf_blocks: [GfBlockSpec { content: 2, len: 1, referenced: false }, GfBlockSpec { content: 0, len: 1, referenced: true }], n_modules: 0, n_views: 0, views_on_temps: false, temp_sizes: [], dim_lists: [], with_loop: false, n_elements: 2 }] diff --git a/src/simlin-engine/proptest-regressions/db/combined_fragment_proptest.txt b/src/simlin-engine/proptest-regressions/db/combined_fragment_proptest.txt new file mode 100644 index 000000000..b3bebec32 --- /dev/null +++ b/src/simlin-engine/proptest-regressions/db/combined_fragment_proptest.txt @@ -0,0 +1,16 @@ +# Seeds for failure cases proptest has generated in the past. It is +# automatically read and these particular cases re-run before any +# novel cases are generated. +# +# It is recommended to check this file in to source control so that +# everyone who runs the test benefits from these saved cases. +# +# Provenance: captured during the C2 mutation experiments on `FragmentMerger` +# (GH #964 row C), not from a production bug. Both seeds fail under both +# interleaving mutations: `combine_scc_fragment` switched to +# `TempStrategy::Recycle`, and a static view's `Temp` base shifted by +# `ctx_base.temps` instead of the per-fragment `temp_offset` -- the second of +# which passed lib 5365/0 and `file_io` 637/0 until `temp_uses` learned to +# resolve views. They guard the two-channel temp scan, not a historical defect. +cc d743fe134fbb25c2d91d28ea13339a6da094f1aefb2f09452f18af6ee8341503 # shrinks to specs = [MemberSpec { index: 0, n_elements: 1, n_literals: 0, n_temps: 1, n_modules: 0, n_views: 0, n_gf_tables: 0, with_loop: false }, MemberSpec { index: 1, n_elements: 1, n_literals: 0, n_temps: 1, n_modules: 0, n_views: 1, n_gf_tables: 0, with_loop: false }] +cc 1f353451224703f806aa7db528b9846f6232b4e4c18019cb8f562d0242ff63ed # shrinks to specs = [MemberSpec { index: 0, n_elements: 1, n_literals: 1, n_temps: 1, n_modules: 0, n_views: 1, n_gf_tables: 0, with_loop: false }, MemberSpec { index: 1, n_elements: 1, n_literals: 1, n_temps: 1, n_modules: 0, n_views: 1, n_gf_tables: 0, with_loop: false }] diff --git a/src/simlin-engine/src/array_tests.rs b/src/simlin-engine/src/array_tests.rs index 7e648fd26..16e1c873a 100644 --- a/src/simlin-engine/src/array_tests.rs +++ b/src/simlin-engine/src/array_tests.rs @@ -5487,4 +5487,351 @@ mod arrayed_vector_sort_order_per_slice_tests { } project.assert_vm_result_incremental("sorted", &[10.0, 20.0, 30.0, 1.0, 5.0, 15.0]); } + + /// A VECTOR ELM MAP whose source is subscripted by a *variable* index + /// (`src[idx]`, resolved at runtime) must bound its reads by the source + /// VARIABLE's full extent, exactly as a literal subscript does. + /// + /// `codegen::full_source_len` recovered that extent for + /// `Expr::StaticSubscript` and `Expr::Var` but had no arm for the dynamic + /// `Expr::Subscript`, so it fell through to the 1-element default. Every + /// read outside `[0, 1)` then returned the documented out-of-range `:NA:` + /// (NaN) -- which is EVERY read whenever the base was not the source's + /// first element, and every non-zero offset regardless. Both directions are + /// covered below, and neither is expressible with a literal subscript, so + /// no existing test could see it. + /// + /// vals = [10, 20, 30, 40]; idx = 2 (1-based) selects flat base 1. + /// picked[j] = vals_full[1 + offs[j]], offs = [0, 1, 2] + /// = [vals[1], vals[2], vals[3]] = [20, 30, 40] + #[test] + fn elm_map_dynamic_source_subscript_uses_full_variable_extent_vm() { + let project = TestProject::new("elm_map_dynamic_source_extent") + .indexed_dimension("D", 4) + .indexed_dimension("E", 3) + .array_with_ranges( + "vals[D]", + vec![("1", "10"), ("2", "20"), ("3", "30"), ("4", "40")], + ) + .array_with_ranges("offs[E]", vec![("1", "0"), ("2", "1"), ("3", "2")]) + .scalar_aux("idx", "2") + .array_aux("picked[E]", "VECTOR ELM MAP(vals[idx], offs[E])"); + project.assert_compiles_incremental(); + project.assert_vm_result_incremental("picked", &[20.0, 30.0, 40.0]); + } +} + +#[cfg(test)] +mod cross_module_array_reference_tests { + //! A reference into a sub-model (`m.x`) does not name `x`. Lowering + //! collapses it to `VarRef { name: m, element_offset: x's slot inside the + //! instance }`, because the parent model's layout has one entry spanning the + //! whole sub-model and none for the dotted name. Every question the compiler + //! asks about "the variable this reference names" therefore has to resolve + //! through the sub-model's own layout, and three of them did not: + //! + //! * `codegen::full_source_len` looked the extent up by NAME, so a VECTOR + //! ELM MAP over a cross-module source was bounded by the module + //! instance's whole slot count -- reads past the source's end landed on + //! the NEXT sub-model variable instead of yielding `:NA:`; + //! * `Context::full_var_len_for_base`, the lowering-side twin that folds a + //! constant-offset ELM MAP, asked the module variable for its dimensions + //! and got none, so it bounded the same source at ONE element and folded + //! every read to NaN; + //! * `Expr3LowerContext::get_dimensions` did a direct, non-dotted metadata + //! lookup, so a cross-module array looked like a scalar to the Expr2 -> + //! Expr3 lowering and a wildcard subscript on one was rejected as + //! `CantSubscriptScalar`. + //! + //! Every assertion below is the value the identical IN-MODEL shape produces, + //! which is what makes them a specification rather than a transcript: the + //! local twins are `elm_map_dynamic_source_subscript_uses_full_variable_extent_vm` + //! and the rest of this file. All three defects predate GH #964's symbolic + //! emission -- the offset-scan it replaced resolved a cross-module base to + //! the module variable for exactly the same reason. + + use crate::datamodel; + use crate::test_common::TestProject; + use crate::testutils::{x_aux, x_model, x_module, x_module_named, x_project}; + + fn arrayed(ident: &str, dim: &str, elems: &[(&str, &str)]) -> datamodel::Variable { + datamodel::Variable::Aux(datamodel::Aux { + ident: ident.to_string(), + equation: datamodel::Equation::Arrayed( + vec![dim.to_string()], + elems + .iter() + .map(|(e, eq)| (e.to_string(), eq.to_string(), None, None)) + .collect(), + None, + false, + ), + documentation: String::new(), + units: None, + gf: None, + ai_state: None, + uid: None, + compat: datamodel::Compat::default(), + }) + } + + fn a2a(ident: &str, dim: &str, eqn: &str) -> datamodel::Variable { + datamodel::Variable::Aux(datamodel::Aux { + ident: ident.to_string(), + equation: datamodel::Equation::ApplyToAll(vec![dim.to_string()], eqn.to_string()), + documentation: String::new(), + units: None, + gf: None, + ai_state: None, + uid: None, + compat: datamodel::Compat::default(), + }) + } + + /// The two arrayed sources every fixture below reads through: + /// + /// avals = [10, 20, 30, 40] at slots 0..4 + /// bvals = [100, 200, 300, 400] at slots 4..8 + /// + /// The layout is alphabetical, so `avals` really is at its model's base -- + /// the case where a wrong extent lookup and a right one are hardest to tell + /// apart, since a reference to it satisfies "element offset is 0" either way. + fn source_vars() -> Vec { + vec![ + arrayed( + "avals", + "D", + &[("1", "10"), ("2", "20"), ("3", "30"), ("4", "40")], + ), + arrayed( + "bvals", + "D", + &[("1", "100"), ("2", "200"), ("3", "300"), ("4", "400")], + ), + ] + } + + /// `main`, holding whatever sub-models a fixture supplies plus the variable + /// under test. + /// + /// `picked[E]` is the arrayed subject; `probe` is its scalar twin, so one + /// builder serves both an arrayed and a scalar equation. + /// + /// offs = [0, 1, 2] idx = 4 + fn project_with( + module: datamodel::Variable, + sub_models: Vec, + picked_eqn: &str, + probe_eqn: &str, + ) -> TestProject { + let main = x_model( + "main", + vec![ + module, + x_aux("idx", "4", None), + arrayed("offs", "E", &[("1", "0"), ("2", "1"), ("3", "2")]), + a2a("picked", "E", picked_eqn), + x_aux("probe", probe_eqn, None), + ], + ); + let mut models = vec![main]; + models.extend(sub_models); + let mut project = x_project( + datamodel::SimSpecs { + start: 0.0, + stop: 1.0, + dt: datamodel::Dt::Dt(1.0), + save_step: Some(datamodel::Dt::Dt(1.0)), + sim_method: datamodel::SimMethod::Euler, + time_units: Some("Month".to_string()), + }, + &models, + ); + project.dimensions = vec![ + datamodel::Dimension::indexed("D".to_string(), 4), + datamodel::Dimension::indexed("E".to_string(), 3), + ]; + TestProject::from_datamodel(project) + } + + /// One hop: `main` -> instance `sub` of a model holding the two sources. + fn xmod_project(picked_eqn: &str, probe_eqn: &str) -> TestProject { + project_with( + x_module("sub", &[], None), + vec![x_model("sub", source_vars())], + picked_eqn, + probe_eqn, + ) + } + + /// TWO hops: `main` -> instance `m` of `mid` -> instance `inr` of `inner`. + /// + /// `mid` holds a scalar `aa` at slot 0 and `inr` at slot 1, so `inner`'s + /// block does NOT start at `m`'s base and the sources sit at `m`-relative + /// slots 1 (`avals`) and 5 (`bvals`). Reaching them needs the offsets + /// ACCUMULATED down the chain -- `aa`'s slot plus the source's own -- which + /// is the arithmetic a one-hop fixture cannot distinguish from ignoring the + /// intermediate offset entirely, because at one hop that offset is 0. + fn nested_xmod_project(picked_eqn: &str, probe_eqn: &str) -> TestProject { + // `compute_layout` orders a model's variables alphabetically, so `aa` + // takes slot 0 and `inr`'s 8-slot block starts at 1. + let mid = x_model( + "mid", + vec![ + x_aux("aa", "1", None), + x_module_named("inr", "inner", &[], None), + ], + ); + project_with( + x_module_named("m", "mid", &[], None), + vec![mid, x_model("inner", source_vars())], + picked_eqn, + probe_eqn, + ) + } + + /// A VECTOR ELM MAP whose source is a DYNAMICALLY subscripted cross-module + /// array must be bounded by that sub-model variable's own extent. + /// + /// `idx = 4` puts the base at `avals`'s last element, so offsets 1 and 2 map + /// outside `avals` and are `:NA:`. Bounded by the module instance's 8 slots + /// instead, they read `bvals[0]` and `bvals[1]` -- 100 and 200, a silent + /// wrong answer with no diagnostic anywhere. + /// + /// No wasm-backend twin: `wasmgen::vector` rejects a dynamically-subscripted + /// ELM MAP source outright, so this shape never reaches wasm lowering. The + /// static shapes below share the operand with wasm by construction -- it is + /// baked into the resolved bytecode both backends read -- so they cannot + /// diverge either. + #[test] + fn elm_map_cross_module_dynamic_source_uses_the_submodel_variable_extent() { + let project = xmod_project("VECTOR ELM MAP(sub.avals[idx], offs[E])", "0"); + project.assert_compiles_incremental(); + let got = project.vm_result_incremental("picked"); + assert_eq!(got[0], 40.0, "in range: avals[3]"); + assert!( + got[1].is_nan() && got[2].is_nan(), + "past avals's end must be :NA:, not bvals[0]/bvals[1]; got {got:?}" + ); + } + + /// The same rule for a source that is NOT at the instance's base: `bvals` + /// starts at slot 4, so the reference is `VarRef { sub, 4 }` with the element + /// in the view. + /// + /// The source is a COLLAPSED static element here, not a dynamic subscript, + /// which is what makes this case load-bearing rather than a restatement of + /// the one above: a collapsed view carries no bounds to fall back on, so the + /// extent must come from the table or not at all. `sub.bvals[2]` is flat + /// base 1 within `bvals`, and offsets 0..2 read `bvals[1..4]`. A reference + /// below the instance's base failed the old whole-variable test outright, + /// the extent fell back to the view's single element, and the ELM MAP then + /// treated the source as a full one-element array -- base 0, everything past + /// it `:NA:` -- so it returned `[200, NaN, NaN]`. + #[test] + fn elm_map_cross_module_source_below_the_instance_base_uses_its_own_extent() { + let project = xmod_project("VECTOR ELM MAP(sub.bvals[2], offs[E])", "0"); + project.assert_compiles_incremental(); + project.assert_vm_result_incremental("picked", &[200.0, 300.0, 400.0]); + } + + /// The lowering-side twin: a collapsed literal element source with a + /// per-element CONSTANT offset is folded to a direct slot read at compile + /// time (GH #578), and that fold reads the extent through + /// `Context::full_var_len_for_base`. + /// + /// `sub.avals[4]` is flat base 3 and `E - 1` is 0, 1, 2, so only the first + /// read is in range. The assertion catches BOTH ways the extent can be + /// wrong, which is why the base sits at the end of the source rather than + /// in the middle: bounded at one element -- what asking the dimensionless + /// module variable for its dimensions returned -- the fold emits a constant + /// NaN for every element; bounded at the instance's 8 slots, it folds the + /// out-of-range reads to `bvals[0]` and `bvals[1]`. + #[test] + fn elm_map_cross_module_constant_offset_fold_uses_the_submodel_variable_extent() { + let project = xmod_project("VECTOR ELM MAP(sub.avals[4], E - 1)", "0"); + project.assert_compiles_incremental(); + let got = project.vm_result_incremental("picked"); + assert_eq!(got[0], 40.0, "in range: avals[3]"); + assert!( + got[1].is_nan() && got[2].is_nan(), + "past avals's end must fold to :NA:, not to bvals[0]/bvals[1]; got {got:?}" + ); + } + + /// A wildcard subscript on a cross-module array must compile: the reference + /// is an array, and the Expr2 -> Expr3 lowering has to see its dimensions to + /// resolve the wildcard. + /// + /// It did not, so `SUM(m.arr[*])` -- the ordinary way to reduce over a + /// sub-model's arrayed output, and the shape `db::stages_tests`' + /// `arrayed_module_project` fixture is built from -- was rejected as + /// `CantSubscriptScalar` and the whole variable failed to compile. That + /// fixture never noticed because it stops at Stage1 lowering, which uses + /// `ast::ArrayContext`, whose own `get_dimensions` always followed the + /// module variable. + #[test] + fn a_wildcard_subscript_on_a_cross_module_array_compiles_and_reduces() { + let project = xmod_project("0", "SUM(sub.avals[*])"); + project.assert_compiles_incremental(); + project.assert_vm_result_incremental("probe", &[100.0, 100.0]); + } + + /// ...and the wildcard also works as a whole-array ELM MAP source, where the + /// extent lookup and the dimension lookup both run on the same reference. + #[test] + fn elm_map_over_a_whole_cross_module_array_matches_the_in_model_shape() { + let project = xmod_project("VECTOR ELM MAP(sub.avals[*], offs[E])", "0"); + project.assert_compiles_incremental(); + project.assert_vm_result_incremental("picked", &[10.0, 20.0, 30.0]); + } + + // ──────────────────────────────────────────────────────────────────── + // TWO hops. A one-hop fixture cannot see the offset ACCUMULATION down a + // module chain, because the only offset it has to accumulate is 0: change + // the recursion's `base + offset` to `base` and every one-hop assertion + // above still holds. `nested_xmod_project` interposes a scalar at slot 0 of + // the middle model so the inner block starts at 1, and the two tests below + // are the only thing in the repository that distinguishes the two. + // ──────────────────────────────────────────────────────────────────── + + /// A source reached through TWO module hops resolves at the accumulated + /// slot: `avals` is at `inner`-relative 0, `inner` at `mid`-relative 1, so + /// the reference is `VarRef { m, 1 }` and its extent is `avals`'s 4. + /// + /// Base at `avals`'s last element, so offsets 1 and 2 map outside it. + #[test] + fn elm_map_through_two_module_hops_uses_the_leaf_variables_extent() { + let project = nested_xmod_project("VECTOR ELM MAP(m.inr.avals[4], offs[E])", "0"); + project.assert_compiles_incremental(); + let got = project.vm_result_incremental("picked"); + assert_eq!(got[0], 40.0, "in range: avals[3]"); + assert!( + got[1].is_nan() && got[2].is_nan(), + "past avals's end must be :NA:, not bvals[0]/bvals[1]; got {got:?}" + ); + } + + /// The same chain to the SECOND leaf variable, at `inner`-relative 4 and so + /// `m`-relative 5 -- the accumulation has to carry both hops' offsets, not + /// just the last one. + #[test] + fn elm_map_through_two_module_hops_below_the_inner_base_uses_its_own_extent() { + let project = nested_xmod_project("VECTOR ELM MAP(m.inr.bvals[2], offs[E])", "0"); + project.assert_compiles_incremental(); + project.assert_vm_result_incremental("picked", &[200.0, 300.0, 400.0]); + } + + /// The whole-array form through two hops. This one is a COMPILES-AT-ALL + /// check and nothing more: its lowered view carries the source's full + /// dimensions, so the view fallback coincides with the extent the table + /// reports and the assertion holds whether or not the chain resolved. It is + /// here because the shape should not regress into a compile error, not + /// because it can detect a wrong offset -- the two tests above do that. + #[test] + fn a_whole_array_source_two_module_hops_away_still_compiles() { + let project = nested_xmod_project("VECTOR ELM MAP(m.inr.avals[*], offs[E])", "0"); + project.assert_compiles_incremental(); + project.assert_vm_result_incremental("picked", &[10.0, 20.0, 30.0]); + } } diff --git a/src/simlin-engine/src/bytecode.rs b/src/simlin-engine/src/bytecode.rs index b96fefa11..7bb7a7c0a 100644 --- a/src/simlin-engine/src/bytecode.rs +++ b/src/simlin-engine/src/bytecode.rs @@ -2,7 +2,7 @@ // Use of this source code is governed by the Apache License, // Version 2.0, that can be found in the LICENSE file. -use std::collections::{BTreeSet, HashMap}; +use std::collections::BTreeSet; use std::sync::Arc; use smallvec::SmallVec; @@ -44,10 +44,16 @@ pub type DimListId = u16; // Index into dim_lists table (for [DimId; 4] or [u16; /// The stack resets to 0 after every assignment opcode, so depth depends only on /// expression complexity, not on model size. /// -/// `ByteCodeBuilder::finish()` validates at compile time that no bytecode -/// sequence exceeds this capacity, making the VM's unsafe unchecked stack -/// access provably safe. The `#![deny(unsafe_code)]` crate attribute ensures -/// no other unsafe code can be added without explicit opt-in. +/// [`crate::compiler::symbolic::resolve_bytecode`] validates at compile time +/// that no bytecode sequence exceeds this capacity, making the VM's unsafe +/// unchecked stack access provably safe. That is the single place concrete +/// bytecode is produced, so the check cannot be bypassed; it answers with a +/// compile `Err` rather than an assertion, so an over-deep or underflowing +/// program is rejected instead of executed. (It lived in the per-fragment +/// `ByteCodeBuilder::finish()` until GH #964 moved emission into the symbolic +/// domain, taking the builder with it.) The `#![deny(unsafe_code)]` crate +/// attribute ensures no other unsafe code can be added without explicit +/// opt-in. pub(crate) const STACK_CAPACITY: usize = 64; /// Lookup interpolation mode for graphical function tables. @@ -613,9 +619,9 @@ pub(crate) enum Op2 { /// - Variable access (LoadVar, LoadGlobalVar, etc.) /// - Control flow (SetCond, If, Ret) /// - Module operations (EvalModule, LoadModuleInput) -/// - Assignment (AssignCurr, AssignNext) +/// - Assignment (AssignCurr, and the fused BinOpAssignNext for stock updates) /// - Builtins and lookups (Apply, Lookup) -/// - Array view stack operations (PushVarView, ViewSubscript*, etc.) +/// - Array view stack operations (PushStaticView, ViewSubscript*, etc.) /// - Array iteration (BeginIter, LoadIterElement, etc.) /// - Array reductions (ArraySum, ArrayMax, etc.) #[cfg_attr(feature = "debug-derive", derive(Debug))] @@ -677,9 +683,6 @@ pub(crate) enum Opcode { AssignCurr { off: VariableOffset, }, - AssignNext { - off: VariableOffset, - }, // === BUILTINS & LOOKUPS === Apply { @@ -714,8 +717,15 @@ pub(crate) enum Opcode { off: VariableOffset, }, - /// Fused Op2 + AssignNext. - /// Pops two values, applies binary op, assigns result to next[module_off + off]. + /// The next-value assignment: pops two values, applies the binary op, and + /// assigns the result to `next[module_off + off]`. + /// + /// There is no un-fused counterpart. Every write to `next[]` is a stock + /// update, and a stock update's operand walk always ends in an `Op2` + /// (`Context::build_stock_update_expr`), so codegen emits this form + /// directly via + /// `compiler::symbolic::SymbolicByteCodeBuilder::fuse_trailing_op2_into_assign_next` + /// and `resolve_bytecode` carries it through unchanged. BinOpAssignNext { op: Op2, off: VariableOffset, @@ -1002,14 +1012,6 @@ pub(crate) enum Opcode { // ========================================================================= // === VIEW STACK: Building views dynamically === - /// Push a view for a variable's full array onto the view stack. - /// Looks up dimension info to compute strides. - /// The dim_list_id references a (n_dims, [DimId; 4]) entry in ByteCodeContext.dim_lists. - PushVarView { - base_off: VariableOffset, - dim_list_id: DimListId, - }, - /// Push a view for a temp array onto the view stack. /// The dim_list_id references a (n_dims, [DimId; 4]) entry in ByteCodeContext.dim_lists. PushTempView { @@ -1319,7 +1321,7 @@ impl Opcode { Opcode::EvalModule { n_inputs, .. } => (*n_inputs, 0), // Assignment: pops 1 (the value to assign) - Opcode::AssignCurr { .. } | Opcode::AssignNext { .. } => (1, 0), + Opcode::AssignCurr { .. } => (1, 0), // Builtins always take 3 args (actual + padding), push 1 result Opcode::Apply { .. } => (3, 1), @@ -1386,8 +1388,7 @@ impl Opcode { | Opcode::AssignStackConstNext { .. } => (1, 0), // View stack ops don't touch arithmetic stack - Opcode::PushVarView { .. } - | Opcode::PushTempView { .. } + Opcode::PushTempView { .. } | Opcode::PushStaticView { .. } | Opcode::PushVarViewDirect { .. } | Opcode::ViewSubscriptConst { .. } @@ -1469,7 +1470,6 @@ impl Opcode { Opcode::LoadModuleInput { .. } => "LoadModuleInput", Opcode::EvalModule { .. } => "EvalModule", Opcode::AssignCurr { .. } => "AssignCurr", - Opcode::AssignNext { .. } => "AssignNext", Opcode::Apply { .. } => "Apply", Opcode::Lookup { .. } => "Lookup", Opcode::AssignConstCurr { .. } => "AssignConstCurr", @@ -1515,7 +1515,6 @@ impl Opcode { Opcode::AssignStackVarNext { .. } => "AssignStackVarNext", Opcode::AssignStackConstCurr { .. } => "AssignStackConstCurr", Opcode::AssignStackConstNext { .. } => "AssignStackConstNext", - Opcode::PushVarView { .. } => "PushVarView", Opcode::PushTempView { .. } => "PushTempView", Opcode::PushStaticView { .. } => "PushStaticView", Opcode::PushVarViewDirect { .. } => "PushVarViewDirect", @@ -1773,174 +1772,34 @@ impl ByteCode { /// SD expressions are straight-line (no conditional jumps that could create /// divergent stack depths -- backward jumps from iteration opcodes always /// return to the same stack depth), a single linear pass is sufficient. - pub(crate) fn max_stack_depth(&self) -> usize { + /// + /// `Err` on an underflow, which means an opcode's [`Opcode::stack_effect`] + /// metadata is wrong -- a compiler bug, not a modelling error. It is + /// reported rather than asserted because the sole production caller, + /// [`crate::compiler::symbolic::resolve_bytecode`], is the one place + /// concrete bytecode is born and already answers with a `Result`: an abort + /// there would take down a `libsimlin` host (release builds are + /// `panic = abort`) for a defect the host can neither cause nor fix, and it + /// would leave that function with one failure mode it reports and one it + /// does not. `checked_sub` rather than `saturating_sub` is the load-bearing + /// part either way -- a silent clamp would invalidate the VM's stack-safety + /// proof without saying so. + pub(crate) fn max_stack_depth(&self) -> Result { let mut depth: usize = 0; let mut max_depth: usize = 0; for (pc, op) in self.code.iter().enumerate() { let (pops, pushes) = op.stack_effect(); - // Use checked_sub rather than saturating_sub: an underflow here - // means stack_effect() metadata is wrong for some opcode, which - // would silently invalidate our safety proof. Panicking surfaces - // the bug immediately in tests. - depth = depth.checked_sub(pops as usize).unwrap_or_else(|| { - panic!("stack_effect underflow at pc {pc}: {pops} pops but depth is {depth}") - }); + depth = depth.checked_sub(pops as usize).ok_or_else(|| { + format!("stack_effect underflow at pc {pc}: {pops} pops but depth is {depth}") + })?; depth += pushes as usize; max_depth = max_depth.max(depth); } - max_depth - } -} - -#[cfg_attr(feature = "debug-derive", derive(Debug))] -#[derive(Clone, Default)] -pub struct ByteCodeBuilder { - bytecode: ByteCode, - // keyed on the literal's bit pattern: interning only needs Eq + Hash, and - // bit-exact deduplication is the right semantic for codegen (it never - // conflates distinct values; at worst -0.0 and 0.0 get separate slots) - interned_literals: HashMap, -} - -impl ByteCodeBuilder { - pub(crate) fn intern_literal(&mut self, lit: f64) -> LiteralId { - let key = lit.to_bits(); - if self.interned_literals.contains_key(&key) { - return self.interned_literals[&key]; - } - self.bytecode.literals.push(lit); - let literal_id = (self.bytecode.literals.len() - 1) as u16; - self.interned_literals.insert(key, literal_id); - literal_id - } - - /// Allocate a new literal slot without deduplication. - /// Used for named constants so each variable gets its own slot, - /// preventing shared-literal corruption when overriding via set_value. - pub(crate) fn push_named_literal(&mut self, lit: f64) -> LiteralId { - self.bytecode.literals.push(lit); - (self.bytecode.literals.len() - 1) as u16 - } - - pub(crate) fn push_opcode(&mut self, op: Opcode) { - self.bytecode.code.push(op) - } - - /// Returns the current number of opcodes in the bytecode - pub(crate) fn len(&self) -> usize { - self.bytecode.code.len() - } - - pub(crate) fn finish(self) -> ByteCode { - let mut bc = self.bytecode; - bc.peephole_optimize(); - - // Validate that the compiled bytecode cannot overflow the VM's - // fixed-size stack. This makes the unsafe unchecked stack access - // in the VM provably safe for this bytecode. - let depth = bc.max_stack_depth(); - assert!( - depth < STACK_CAPACITY, - "compiled bytecode requires stack depth {depth}, exceeding VM capacity {STACK_CAPACITY}" - ); - - bc + Ok(max_depth) } } impl ByteCode { - /// Peephole optimization pass: fuse common opcode sequences into - /// superinstructions to reduce dispatch overhead. - /// - /// Only fuses adjacent instructions when neither is a jump target. - /// Jump offsets are recalculated after fusion using an old->new PC map. - fn peephole_optimize(&mut self) { - if self.code.is_empty() { - return; - } - - // 1. Build set of PCs that are jump targets - let mut jump_targets = vec![false; self.code.len()]; - for (pc, op) in self.code.iter().enumerate() { - if let Some(offset) = op.jump_offset() { - let target = (pc as isize + offset as isize) as usize; - assert!( - target < jump_targets.len(), - "jump at pc {pc} targets {target}, which is out of bounds (code length: {})", - self.code.len() - ); - jump_targets[target] = true; - } - } - - // 2. Build old_pc -> new_pc mapping and fused output. - // pc_map has one entry per original instruction so that jump fixup - // can index by the original PC directly. - let mut optimized: Vec = Vec::with_capacity(self.code.len()); - let mut pc_map: Vec = Vec::with_capacity(self.code.len() + 1); - let mut i = 0; - while i < self.code.len() { - let new_pc = optimized.len(); - pc_map.push(new_pc); - - // Only try fusion if the next instruction is not a jump target. - // We intentionally don't check whether instruction i itself is a - // jump target: the fused instruction replaces both i and i+1 at the - // same PC, so jumps to i still land on the correct (fused) opcode. - let can_fuse = i + 1 < self.code.len() && !jump_targets[i + 1]; - - if can_fuse { - let fused = match (&self.code[i], &self.code[i + 1]) { - // Pattern: LoadConstant + AssignCurr -> AssignConstCurr - (Opcode::LoadConstant { id }, Opcode::AssignCurr { off }) => { - Some(Opcode::AssignConstCurr { - off: *off, - literal_id: *id, - }) - } - // Pattern: Op2 + AssignCurr -> BinOpAssignCurr - (Opcode::Op2 { op }, Opcode::AssignCurr { off }) => { - Some(Opcode::BinOpAssignCurr { op: *op, off: *off }) - } - // Pattern: Op2 + AssignNext -> BinOpAssignNext - (Opcode::Op2 { op }, Opcode::AssignNext { off }) => { - Some(Opcode::BinOpAssignNext { op: *op, off: *off }) - } - _ => None, - }; - - if let Some(op) = fused { - optimized.push(op); - // Both old PCs map to the same new PC - pc_map.push(new_pc); - i += 2; - continue; - } - } - - // No pattern matched - copy opcode as-is - optimized.push(self.code[i]); - i += 1; - } - // Sentinel for instructions past the end - pc_map.push(optimized.len()); - - // 3. Fix up jump offsets. Iterate original code to find jumps, - // then use pc_map (indexed by old_pc) for O(1) translation. - for (old_pc, op) in self.code.iter().enumerate() { - let Some(jump_back) = op.jump_offset() else { - continue; - }; - let new_pc = pc_map[old_pc]; - let old_target = (old_pc as isize + jump_back as isize) as usize; - let new_target = pc_map[old_target]; - let new_jump_back = (new_target as isize - new_pc as isize) as PcOffset; - *optimized[new_pc].jump_offset_mut().unwrap() = new_jump_back; - } - - self.code = optimized; - } - /// Late 3-address fusion pass (R2): fold the leaf operand load(s) of a /// binary op into the op itself. /// @@ -2224,42 +2083,6 @@ impl ByteCode { mod tests { use super::*; - // ========================================================================= - // ByteCode Builder Tests - // ========================================================================= - - #[test] - fn test_memoizing_interning() { - let mut bytecode = ByteCodeBuilder::default(); - let a1 = bytecode.intern_literal(1.0); - let b1 = bytecode.intern_literal(1.01); - let b2 = bytecode.intern_literal(1.01); - let b3 = bytecode.intern_literal(1.01); - let a2 = bytecode.intern_literal(1.0); - let b4 = bytecode.intern_literal(1.01); - - assert_eq!(a1, a2); - assert_eq!(b1, b2); - assert_eq!(b1, b3); - assert_eq!(b1, b4); - assert_ne!(a1, b1); - - let bytecode = bytecode.finish(); - assert_eq!(2, bytecode.literals.len()); - } - - #[test] - fn test_push_named_literal_no_dedup() { - let mut builder = ByteCodeBuilder::default(); - let a = builder.push_named_literal(0.1); - let b = builder.push_named_literal(0.1); - let c = builder.push_named_literal(0.1); - - assert_ne!(a, b); - assert_ne!(b, c); - assert_ne!(a, c); - } - // ========================================================================= // Stack Effect Tests // ========================================================================= @@ -2289,7 +2112,6 @@ mod tests { #[test] fn test_stack_effect_assignments() { assert_eq!((Opcode::AssignCurr { off: 0 }).stack_effect(), (1, 0)); - assert_eq!((Opcode::AssignNext { off: 0 }).stack_effect(), (1, 0)); } #[test] @@ -2369,7 +2191,7 @@ mod tests { #[test] fn test_stack_effect_view_ops_dont_affect_arithmetic_stack() { assert_eq!( - (Opcode::PushVarView { + (Opcode::PushVarViewDirect { base_off: 0, dim_list_id: 0, }) @@ -2459,7 +2281,29 @@ mod tests { #[test] fn test_max_stack_depth_empty() { let bc = ByteCode::default(); - assert_eq!(bc.max_stack_depth(), 0); + assert_eq!(bc.max_stack_depth().unwrap(), 0); + } + + /// A pop with nothing on the stack means some opcode's `stack_effect` + /// metadata is wrong, which would silently invalidate the VM's stack-safety + /// proof. It must be REPORTED, not clamped -- and the message must name the + /// pc so the wrong arm is findable. + /// + /// This was a `#[should_panic]` test until the underflow became a structured + /// error (it is reached from `symbolic::resolve_bytecode`, which answers + /// with a `Result` and must not abort a host process). It is the only + /// negative test this function has; the rest assert depths on well-formed + /// streams and would all still pass against a `saturating_sub`. + #[test] + fn test_max_stack_depth_reports_underflow() { + let bc = ByteCode { + literals: vec![], + code: vec![Opcode::Op2 { op: Op2::Add }], + }; + let err = bc + .max_stack_depth() + .expect_err("an Op2 with an empty stack must be reported, not clamped"); + assert!(err.contains("stack_effect underflow at pc 0"), "got: {err}"); } #[test] @@ -2472,7 +2316,7 @@ mod tests { Opcode::AssignCurr { off: 0 }, ], }; - assert_eq!(bc.max_stack_depth(), 1); + assert_eq!(bc.max_stack_depth().unwrap(), 1); } #[test] @@ -2487,7 +2331,7 @@ mod tests { Opcode::AssignCurr { off: 2 }, ], }; - assert_eq!(bc.max_stack_depth(), 2); + assert_eq!(bc.max_stack_depth().unwrap(), 2); } #[test] @@ -2508,7 +2352,7 @@ mod tests { Opcode::AssignCurr { off: 4 }, // depth: 0 ], }; - assert_eq!(bc.max_stack_depth(), 3); + assert_eq!(bc.max_stack_depth().unwrap(), 3); } #[test] @@ -2526,7 +2370,7 @@ mod tests { Opcode::AssignCurr { off: 1 }, ], }; - assert_eq!(bc.max_stack_depth(), 3); + assert_eq!(bc.max_stack_depth().unwrap(), 3); } #[test] @@ -2543,7 +2387,7 @@ mod tests { Opcode::AssignCurr { off: 3 }, // depth: 0 ], }; - assert_eq!(bc.max_stack_depth(), 2); + assert_eq!(bc.max_stack_depth().unwrap(), 2); } #[test] @@ -2556,7 +2400,7 @@ mod tests { literal_id: 0, }], }; - assert_eq!(bc.max_stack_depth(), 0); + assert_eq!(bc.max_stack_depth().unwrap(), 0); } #[test] @@ -2574,7 +2418,7 @@ mod tests { Opcode::AssignCurr { off: 4 }, ], }; - assert_eq!(bc.max_stack_depth(), 2); + assert_eq!(bc.max_stack_depth().unwrap(), 2); } #[test] @@ -2594,7 +2438,7 @@ mod tests { Opcode::EndIter {}, ], }; - assert_eq!(bc.max_stack_depth(), 1); + assert_eq!(bc.max_stack_depth().unwrap(), 1); } #[test] @@ -2614,40 +2458,7 @@ mod tests { Opcode::AssignCurr { off: 20 }, // depth: 0 ], }; - assert_eq!(bc.max_stack_depth(), 1); - } - - #[test] - fn test_finish_validates_stack_depth() { - // Build bytecode that fits within STACK_CAPACITY -- should succeed - let mut builder = ByteCodeBuilder::default(); - let id = builder.intern_literal(1.0); - builder.push_opcode(Opcode::LoadConstant { id }); - builder.push_opcode(Opcode::AssignCurr { off: 0 }); - let _bc = builder.finish(); // should not panic - } - - #[test] - #[should_panic(expected = "stack_effect underflow at pc 0")] - fn test_max_stack_depth_catches_underflow() { - // An Op2 at the start with nothing on the stack should panic, - // catching bugs in stack_effect metadata - let bc = ByteCode { - literals: vec![], - code: vec![Opcode::Op2 { op: Op2::Add }], - }; - bc.max_stack_depth(); - } - - #[test] - #[should_panic(expected = "jump at pc 0 targets")] - fn test_peephole_panics_on_out_of_bounds_jump_target() { - // A jump that targets beyond the code length indicates a compiler bug - let mut bc = ByteCode { - literals: vec![], - code: vec![Opcode::NextIterOrJump { jump_back: 10 }], - }; - bc.peephole_optimize(); + assert_eq!(bc.max_stack_depth().unwrap(), 1); } // ========================================================================= @@ -3475,504 +3286,6 @@ mod tests { assert_eq!(view.offset_for_iter_index(0), 5); } - // ========================================================================= - // Peephole Optimizer Tests - // ========================================================================= - - #[test] - fn test_peephole_empty_bytecode() { - let mut bc = ByteCode { - code: vec![], - literals: vec![], - }; - bc.peephole_optimize(); - assert!(bc.code.is_empty()); - } - - #[test] - fn test_peephole_single_instruction() { - let mut bc = ByteCode { - code: vec![Opcode::Ret], - literals: vec![], - }; - bc.peephole_optimize(); - assert_eq!(bc.code.len(), 1); - assert!(matches!(bc.code[0], Opcode::Ret)); - } - - #[test] - fn test_peephole_no_fusible_patterns() { - let mut bc = ByteCode { - code: vec![ - Opcode::LoadVar { off: 0 }, - Opcode::LoadVar { off: 1 }, - Opcode::Not {}, - Opcode::Ret, - ], - literals: vec![], - }; - bc.peephole_optimize(); - assert_eq!(bc.code.len(), 4); - assert!(matches!(bc.code[0], Opcode::LoadVar { off: 0 })); - assert!(matches!(bc.code[1], Opcode::LoadVar { off: 1 })); - assert!(matches!(bc.code[2], Opcode::Not {})); - assert!(matches!(bc.code[3], Opcode::Ret)); - } - - #[test] - fn test_peephole_load_constant_assign_curr_fusion() { - let mut bc = ByteCode { - code: vec![ - Opcode::LoadConstant { id: 0 }, - Opcode::AssignCurr { off: 5 }, - ], - literals: vec![42.0], - }; - bc.peephole_optimize(); - - assert_eq!(bc.code.len(), 1); - match &bc.code[0] { - Opcode::AssignConstCurr { off, literal_id } => { - assert_eq!(*off, 5); - assert_eq!(*literal_id, 0); - } - _ => panic!("expected AssignConstCurr"), - } - } - - #[test] - fn test_peephole_op2_assign_curr_fusion() { - let mut bc = ByteCode { - code: vec![ - Opcode::LoadVar { off: 0 }, - Opcode::LoadVar { off: 1 }, - Opcode::Op2 { op: Op2::Add }, - Opcode::AssignCurr { off: 2 }, - ], - literals: vec![], - }; - bc.peephole_optimize(); - - // LoadVar, LoadVar stay; Op2+AssignCurr fuse into BinOpAssignCurr - assert_eq!(bc.code.len(), 3); - assert!(matches!(bc.code[0], Opcode::LoadVar { off: 0 })); - assert!(matches!(bc.code[1], Opcode::LoadVar { off: 1 })); - match &bc.code[2] { - Opcode::BinOpAssignCurr { op, off } => { - assert!(matches!(op, Op2::Add)); - assert_eq!(*off, 2); - } - _ => panic!("expected BinOpAssignCurr"), - } - } - - #[test] - fn test_peephole_op2_assign_next_fusion() { - let mut bc = ByteCode { - code: vec![ - Opcode::LoadVar { off: 0 }, - Opcode::LoadVar { off: 1 }, - Opcode::Op2 { op: Op2::Mul }, - Opcode::AssignNext { off: 3 }, - ], - literals: vec![], - }; - bc.peephole_optimize(); - - assert_eq!(bc.code.len(), 3); - match &bc.code[2] { - Opcode::BinOpAssignNext { op, off } => { - assert!(matches!(op, Op2::Mul)); - assert_eq!(*off, 3); - } - _ => panic!("expected BinOpAssignNext"), - } - } - - #[test] - fn test_peephole_all_op2_variants_fuse() { - // Verify every Op2 variant can be fused with AssignCurr - let ops = [ - Op2::Add, - Op2::Sub, - Op2::Mul, - Op2::Div, - Op2::Exp, - Op2::Mod, - Op2::Gt, - Op2::Gte, - Op2::Lt, - Op2::Lte, - Op2::Eq, - Op2::And, - Op2::Or, - ]; - for op in ops { - let mut bc = ByteCode { - code: vec![Opcode::Op2 { op }, Opcode::AssignCurr { off: 10 }], - literals: vec![], - }; - bc.peephole_optimize(); - assert_eq!(bc.code.len(), 1, "failed for op variant"); - assert!(matches!(bc.code[0], Opcode::BinOpAssignCurr { .. })); - } - } - - #[test] - fn test_peephole_multiple_fusions() { - // Two independent fusion opportunities in sequence - let mut bc = ByteCode { - code: vec![ - Opcode::LoadConstant { id: 0 }, - Opcode::AssignCurr { off: 0 }, - Opcode::LoadVar { off: 1 }, - Opcode::LoadVar { off: 2 }, - Opcode::Op2 { op: Op2::Sub }, - Opcode::AssignCurr { off: 3 }, - ], - literals: vec![1.0], - }; - bc.peephole_optimize(); - - // LoadConstant+AssignCurr -> AssignConstCurr - // LoadVar, LoadVar stay - // Op2+AssignCurr -> BinOpAssignCurr - assert_eq!(bc.code.len(), 4); - assert!(matches!(bc.code[0], Opcode::AssignConstCurr { .. })); - assert!(matches!(bc.code[1], Opcode::LoadVar { off: 1 })); - assert!(matches!(bc.code[2], Opcode::LoadVar { off: 2 })); - assert!(matches!(bc.code[3], Opcode::BinOpAssignCurr { .. })); - } - - #[test] - fn test_peephole_mixed_fusible_and_nonfusible() { - let mut bc = ByteCode { - code: vec![ - Opcode::LoadVar { off: 0 }, - Opcode::Not {}, - Opcode::LoadConstant { id: 0 }, - Opcode::AssignCurr { off: 1 }, - Opcode::LoadVar { off: 2 }, - Opcode::Ret, - ], - literals: vec![0.0], - }; - bc.peephole_optimize(); - - // LoadVar, Not stay; LoadConstant+AssignCurr fuse; LoadVar, Ret stay - assert_eq!(bc.code.len(), 5); - assert!(matches!(bc.code[0], Opcode::LoadVar { off: 0 })); - assert!(matches!(bc.code[1], Opcode::Not {})); - assert!(matches!(bc.code[2], Opcode::AssignConstCurr { .. })); - assert!(matches!(bc.code[3], Opcode::LoadVar { off: 2 })); - assert!(matches!(bc.code[4], Opcode::Ret)); - } - - #[test] - fn test_peephole_jump_target_prevents_fusion() { - // If instruction i+1 is a jump target, don't fuse i with i+1. - // Layout (before optimization): - // 0: LoadConstant { id: 0 } <- loop body start (jump target) - // 1: AssignCurr { off: 0 } - // 2: NextIterOrJump { jump_back: -2 } (target = 2 + (-2) = 0) - // 3: Ret - // - // Instruction 0 is a jump target, so even though 0 is LoadConstant - // and 1 is AssignCurr, we should NOT fuse them because instruction 0 - // is a jump target. Wait -- actually the check is whether i+1 is a - // jump target. Here instruction 0 IS a jump target. The optimizer checks - // `!jump_targets[i + 1]` to decide whether to fuse i with i+1. - // - // For i=0: jump_targets[1] is false, so fusion IS allowed. - // The jump target protection matters when the SECOND instruction of a - // potential pair is a jump target. Let's build that scenario: - // - // 0: Ret <- something before the loop - // 1: LoadVar { off: 5 } <- jump target (loop body start) - // 2: NextIterOrJump { jump_back: -1 } (target = 2 + (-1) = 1) - // 3: Ret - // - // For i=0 (Ret): can_fuse checks jump_targets[1] = true -> no fusion. - // This prevents fusing Ret with LoadVar, which is correct. - // - // A more realistic scenario: Op2 followed by AssignCurr where the - // AssignCurr is a jump target. - let mut bc = ByteCode { - code: vec![ - Opcode::Op2 { op: Op2::Add }, // 0 - Opcode::AssignCurr { off: 0 }, // 1 -- jump target - Opcode::NextIterOrJump { jump_back: -1 }, // 2 -> target = 2-1 = 1 - Opcode::Ret, // 3 - ], - literals: vec![], - }; - bc.peephole_optimize(); - - // Fusion of 0+1 should be prevented because instruction 1 is a jump target - assert_eq!(bc.code.len(), 4); - assert!(matches!(bc.code[0], Opcode::Op2 { op: Op2::Add })); - assert!(matches!(bc.code[1], Opcode::AssignCurr { off: 0 })); - assert!(matches!(bc.code[2], Opcode::NextIterOrJump { .. })); - assert!(matches!(bc.code[3], Opcode::Ret)); - } - - #[test] - fn test_peephole_jump_target_only_blocks_specific_pair() { - // Verify that a jump target only blocks fusion of the pair where - // the second instruction is the target, not other pairs. - // - // 0: LoadConstant { id: 0 } - // 1: AssignCurr { off: 0 } <- NOT a jump target, so 0+1 CAN fuse - // 2: LoadVar { off: 5 } <- jump target - // 3: NextIterOrJump { jump_back: -1 } (target = 3-1 = 2) - // 4: Ret - let mut bc = ByteCode { - code: vec![ - Opcode::LoadConstant { id: 0 }, - Opcode::AssignCurr { off: 0 }, - Opcode::LoadVar { off: 5 }, - Opcode::NextIterOrJump { jump_back: -1 }, - Opcode::Ret, - ], - literals: vec![1.0], - }; - bc.peephole_optimize(); - - // 0+1 should fuse (neither target), 2 stays (it's a jump target, but - // the previous instruction was AssignCurr which doesn't match any pattern - // anyway), 3 stays, 4 stays - assert_eq!(bc.code.len(), 4); - assert!(matches!( - bc.code[0], - Opcode::AssignConstCurr { - off: 0, - literal_id: 0 - } - )); - assert!(matches!(bc.code[1], Opcode::LoadVar { off: 5 })); - assert!(matches!(bc.code[2], Opcode::NextIterOrJump { .. })); - assert!(matches!(bc.code[3], Opcode::Ret)); - } - - #[test] - fn test_peephole_jump_offset_recalculation_next_iter() { - // When fusion shrinks the code, jump offsets must be recalculated. - // This test places a fusion BEFORE the loop (outside the jump target - // to jump instruction range) so the fixup works correctly. - // - // Before optimization: - // 0: LoadConstant { id: 0 } \ - // 1: AssignCurr { off: 0 } / -> fuse - // 2: LoadVar { off: 1 } <- jump target - // 3: AssignCurr { off: 2 } - // 4: NextIterOrJump { jump_back: -2 } target = 4+(-2) = 2 - // 5: Ret - // - // After optimization: - // 0: AssignConstCurr (fused 0+1) - // 1: LoadVar { off: 1 } (jump target) - // 2: AssignCurr { off: 2 } - // 3: NextIterOrJump { jump_back: -2 } (loop body unchanged) - // 4: Ret - let mut bc = ByteCode { - code: vec![ - Opcode::LoadConstant { id: 0 }, // 0 - Opcode::AssignCurr { off: 0 }, // 1 - Opcode::LoadVar { off: 1 }, // 2 (jump target) - Opcode::AssignCurr { off: 2 }, // 3 - Opcode::NextIterOrJump { jump_back: -2 }, // 4, target=2 - Opcode::Ret, // 5 - ], - literals: vec![1.0], - }; - bc.peephole_optimize(); - - assert_eq!(bc.code.len(), 5); - assert!(matches!(bc.code[0], Opcode::AssignConstCurr { .. })); - assert!(matches!(bc.code[1], Opcode::LoadVar { off: 1 })); - assert!(matches!(bc.code[2], Opcode::AssignCurr { off: 2 })); - match &bc.code[3] { - Opcode::NextIterOrJump { jump_back } => { - assert_eq!(*jump_back, -2, "jump_back should remain -2"); - } - _ => panic!("expected NextIterOrJump"), - } - assert!(matches!(bc.code[4], Opcode::Ret)); - } - - #[test] - fn test_peephole_fusion_inside_loop_body() { - let mut bc = ByteCode { - code: vec![ - Opcode::LoadVar { off: 0 }, // 0 (jump target) - Opcode::Op2 { op: Op2::Add }, // 1 \ - Opcode::AssignCurr { off: 1 }, // 2 / fuse - Opcode::NextIterOrJump { jump_back: -3 }, // 3, target=0 - Opcode::Ret, // 4 - ], - literals: vec![], - }; - bc.peephole_optimize(); - - // 1+2 fuse -> BinOpAssignCurr - // Result: [LoadVar, BinOpAssignCurr, NextIterOrJump, Ret] - assert_eq!(bc.code.len(), 4); - assert!(matches!(bc.code[0], Opcode::LoadVar { off: 0 })); - assert!(matches!( - bc.code[1], - Opcode::BinOpAssignCurr { - op: Op2::Add, - off: 1 - } - )); - match &bc.code[2] { - Opcode::NextIterOrJump { jump_back } => { - // new PC 2, target should be new PC 0 -> jump_back = -2 - assert_eq!(*jump_back, -2); - } - other => panic!( - "expected NextIterOrJump, got {:?}", - std::mem::discriminant(other) - ), - } - assert!(matches!(bc.code[3], Opcode::Ret)); - } - - #[test] - fn test_peephole_jump_offset_recalculation_next_broadcast() { - // Same as above but with NextBroadcastOrJump - let mut bc = ByteCode { - code: vec![ - Opcode::LoadConstant { id: 0 }, // 0 - Opcode::AssignCurr { off: 0 }, // 1 - Opcode::LoadVar { off: 1 }, // 2 (jump target) - Opcode::NextBroadcastOrJump { jump_back: -1 }, // 3, target=2 - Opcode::Ret, // 4 - ], - literals: vec![1.0], - }; - bc.peephole_optimize(); - - // 0+1 fuse -> AssignConstCurr at new PC 0 - // 2 -> new PC 1 (jump target) - // 3 -> new PC 2 - // 4 -> new PC 3 - assert_eq!(bc.code.len(), 4); - assert!(matches!(bc.code[0], Opcode::AssignConstCurr { .. })); - assert!(matches!(bc.code[1], Opcode::LoadVar { off: 1 })); - match &bc.code[2] { - Opcode::NextBroadcastOrJump { jump_back } => { - // new PC 2, target should be new PC 1 - assert_eq!(*jump_back, -1, "jump_back should be -1"); - } - _ => panic!("expected NextBroadcastOrJump"), - } - assert!(matches!(bc.code[3], Opcode::Ret)); - } - - #[test] - fn test_peephole_no_fusion_when_patterns_dont_match() { - // Op2 followed by something other than AssignCurr/AssignNext - let mut bc = ByteCode { - code: vec![Opcode::Op2 { op: Op2::Add }, Opcode::Not {}, Opcode::Ret], - literals: vec![], - }; - bc.peephole_optimize(); - - assert_eq!(bc.code.len(), 3); - assert!(matches!(bc.code[0], Opcode::Op2 { op: Op2::Add })); - assert!(matches!(bc.code[1], Opcode::Not {})); - } - - #[test] - fn test_peephole_load_constant_not_followed_by_assign_curr() { - // LoadConstant not followed by AssignCurr should not fuse - let mut bc = ByteCode { - code: vec![Opcode::LoadConstant { id: 0 }, Opcode::Not {}, Opcode::Ret], - literals: vec![1.0], - }; - bc.peephole_optimize(); - - assert_eq!(bc.code.len(), 3); - assert!(matches!(bc.code[0], Opcode::LoadConstant { id: 0 })); - } - - #[test] - fn test_peephole_via_builder() { - // Verify that ByteCodeBuilder::finish() runs peephole_optimize - let mut builder = ByteCodeBuilder::default(); - let lit_id = builder.intern_literal(3.125); - builder.push_opcode(Opcode::LoadConstant { id: lit_id }); - builder.push_opcode(Opcode::AssignCurr { off: 7 }); - builder.push_opcode(Opcode::Ret); - - let bc = builder.finish(); - assert_eq!(bc.code.len(), 2); - match &bc.code[0] { - Opcode::AssignConstCurr { off, literal_id } => { - assert_eq!(*off, 7); - assert_eq!(*literal_id, lit_id); - } - _ => panic!("expected AssignConstCurr after builder finish"), - } - assert!(matches!(bc.code[1], Opcode::Ret)); - } - - #[test] - fn test_peephole_consecutive_fusions_chain() { - // Three consecutive fusible pairs - let mut bc = ByteCode { - code: vec![ - Opcode::LoadConstant { id: 0 }, - Opcode::AssignCurr { off: 0 }, - Opcode::LoadConstant { id: 1 }, - Opcode::AssignCurr { off: 1 }, - Opcode::Op2 { op: Op2::Div }, - Opcode::AssignNext { off: 2 }, - ], - literals: vec![1.0, 2.0], - }; - bc.peephole_optimize(); - - assert_eq!(bc.code.len(), 3); - assert!(matches!( - bc.code[0], - Opcode::AssignConstCurr { - off: 0, - literal_id: 0 - } - )); - assert!(matches!( - bc.code[1], - Opcode::AssignConstCurr { - off: 1, - literal_id: 1 - } - )); - match &bc.code[2] { - Opcode::BinOpAssignNext { op, off } => { - assert!(matches!(op, Op2::Div)); - assert_eq!(*off, 2); - } - _ => panic!("expected BinOpAssignNext"), - } - } - - #[test] - fn test_peephole_last_instruction_not_fused_alone() { - // If the fusible first instruction is the very last one, no fusion happens - let mut bc = ByteCode { - code: vec![Opcode::Ret, Opcode::LoadConstant { id: 0 }], - literals: vec![1.0], - }; - bc.peephole_optimize(); - - assert_eq!(bc.code.len(), 2); - assert!(matches!(bc.code[0], Opcode::Ret)); - assert!(matches!(bc.code[1], Opcode::LoadConstant { id: 0 })); - } - - // ========================================================================= // DimList Side Table Tests // ========================================================================= @@ -4398,13 +3711,13 @@ mod tests { }, ], }; - let before = bc.max_stack_depth(); + let before = bc.max_stack_depth().unwrap(); bc.fuse_three_address(); // Fusion folds loads into ops, so depth can only stay equal or shrink. - assert!(bc.max_stack_depth() <= before); + assert!(bc.max_stack_depth().unwrap() <= before); // And the leaf-assign collapsed to a (0,0) op, the stack-leaf assign to // (1,0): the whole stream's peak is now 1 (the BinVarVar push). - assert_eq!(bc.max_stack_depth(), 1); + assert_eq!(bc.max_stack_depth().unwrap(), 1); } #[test] @@ -4465,9 +3778,9 @@ mod tests { Opcode::AssignCurr { off: 4 }, ], }; - let before = bc.max_stack_depth(); + let before = bc.max_stack_depth().unwrap(); bc.fuse_three_address(); - assert!(bc.max_stack_depth() <= before); + assert!(bc.max_stack_depth().unwrap() <= before); } #[test] @@ -4781,9 +4094,9 @@ mod tests { Opcode::AssignCurr { off: 5 }, ], }; - let before = bc.max_stack_depth(); + let before = bc.max_stack_depth().unwrap(); bc.fuse_three_address(); - assert!(bc.max_stack_depth() <= before); + assert!(bc.max_stack_depth().unwrap() <= before); } } diff --git a/src/simlin-engine/src/common.rs b/src/simlin-engine/src/common.rs index 0d46b3adb..baf097042 100644 --- a/src/simlin-engine/src/common.rs +++ b/src/simlin-engine/src/common.rs @@ -360,6 +360,14 @@ pub enum ErrorCode { NotSimulatable, BadTable, BadSimSpecs, + /// No producer since GH #568 deleted `model.rs`'s second dependency walk, + /// which was the only site that raised it (on a `\·`-prefixed dependency). + /// The salsa dependency extraction never had an equivalent check, so this + /// was already unreachable from every production compile; deleting the walk + /// made it unreachable from every path. Kept because `ErrorCode` is mapped + /// to a numbered FFI enum (`src/core/errors.ts`, `src/engine/src/errors.ts`) + /// and removing a discriminant renumbers its successors. Reinstating the + /// check belongs in `db::var_fragment`'s dependency walk, not here. NoAbsoluteReferences, CircularDependency, ArraysNotImplemented, @@ -382,6 +390,9 @@ pub enum ErrorCode { ExpectedInteger, ExpectedIntegerOne, DuplicateUnit, + /// No producer, for the same reason and with the same caveat as + /// [`ErrorCode::NoAbsoluteReferences`]: the deleted walk raised it when a + /// dotted `submodel·output` dependency named a non-module variable. ExpectedModule, ExpectedIdent, UnitMismatch, diff --git a/src/simlin-engine/src/compiler/codegen.rs b/src/simlin-engine/src/compiler/codegen.rs index b4e5ba029..2e4651ef1 100644 --- a/src/simlin-engine/src/compiler/codegen.rs +++ b/src/simlin-engine/src/compiler/codegen.rs @@ -3,34 +3,91 @@ // Version 2.0, that can be found in the LICENSE file. use std::collections::HashMap; -use std::sync::Arc; use smallvec::SmallVec; use crate::ast::{ArrayView, BinaryOp}; use crate::bytecode::{ - BuiltinId, ByteCode, ByteCodeBuilder, ByteCodeContext, CompiledInitial, CompiledModule, DimId, - DimListId, DimensionInfo, GraphicalFunctionId, LookupMode, ModuleDeclaration, ModuleId, - ModuleInputOffset, NameId, Op2, Opcode, RuntimeSparseMapping, StaticArrayView, - SubdimensionRelation, TempId, VariableOffset, ViewId, + BuiltinId, DimId, DimListId, DimensionInfo, GraphicalFunctionId, LookupMode, ModuleId, + ModuleInputOffset, NameId, Op2, RuntimeSparseMapping, SubdimensionRelation, TempId, + VariableOffset, ViewId, }; use crate::common::{Canonical, ErrorCode, ErrorKind, Ident, Result, canonicalize}; use crate::dimensions::Dimension; use crate::sim_err; use crate::vm::{DT_OFF, FINAL_TIME_OFF, INITIAL_TIME_OFF, TIME_OFF}; -use super::Module; use super::dimensions::UnaryOp; -use super::expr::{BuiltinFn, Expr, SubscriptIndex}; +use super::expr::{BuiltinFn, Expr, SubscriptIndex, Table, VarRef}; +use super::symbolic::{ + SymStaticViewBase, SymbolicByteCode, SymbolicByteCodeBuilder, SymbolicCompiledInitial, + SymbolicCompiledModule, SymbolicModuleDecl, SymbolicOpcode, SymbolicStaticView, +}; +use super::{Var, VarSizes}; + +/// Everything `Compiler` reads, borrowed for the duration of one emission. +/// +/// This is the compiler's *whole* input contract, and there is exactly one +/// codegen behind it. Two very different callers build one: +/// +/// * [`super::Module::compile`] -- the test-only monolithic whole-model path, +/// which borrows every field straight off its owned `Module` and then +/// resolves the emitted symbolic module against its own layout; +/// * `db::assemble::compile_phase_to_per_var_bytecodes` -- the production +/// per-variable fragment compiler, which borrows the salsa-cached +/// project-global dimension context and converted dimensions plus the +/// variable's own lowered expressions, and keeps the emitted fragment +/// symbolic until assembly. +/// +/// Borrowing rather than owning is the point (GH #964 / #655): the fragment +/// compiler runs once per variable *per phase* -- tens of thousands of times +/// on an LTM-heavy model -- and the stand-in one-variable `Module` it used to +/// build by struct literal deep-cloned `dimensions_ctx`, `dimensions`, +/// `tables`, `offsets` and the phase's whole lowered `Vec` on every one +/// of them. Nothing in codegen mutates any of it, so a reference is the +/// honest type. +/// +/// The field set is deliberately minimal: it is what codegen actually reads. +/// In particular there is no offset map and no slot count. Codegen emits +/// `VarRef`s straight through from the lowered expressions, so the only thing +/// it still needs from the symbol table is `var_sizes` -- the *extent* of a +/// VECTOR ELM MAP source variable, which is a property of the variable and not +/// of where it lives. +#[derive(Clone, Copy)] +pub(crate) struct ModuleCtx<'a> { + pub(crate) ident: &'a Ident, + /// The module-instance input set, consulted only by `isModuleInput(x)`. + pub(crate) inputs: &'a std::collections::BTreeSet>, + /// Per-temp element counts, indexed by `TempId`. Its length is the temp + /// count; there is no separate `n_temps`, which could only ever disagree. + pub(crate) temp_sizes: &'a [usize], + pub(crate) runlist_initials_by_var: &'a [Var], + pub(crate) runlist_flows: &'a [Expr], + pub(crate) runlist_stocks: &'a [Expr], + /// Reference -> the extent of the variable it addresses in whole. The sole + /// reader is [`Compiler::full_source_len`]; the sole producer is + /// `context::whole_variable_extents`, shared with lowering. + pub(crate) var_sizes: &'a VarSizes, + pub(crate) tables: &'a HashMap, Vec>, + pub(crate) dimensions: &'a [Dimension], + pub(crate) dimensions_ctx: &'a crate::dimensions::DimensionsContext, +} + +impl<'a> ModuleCtx<'a> { + /// Emit this unit's bytecode. The single entry point into codegen. + pub(crate) fn compile(self) -> Result { + Compiler::new(self).compile() + } +} pub(super) struct Compiler<'module> { - module: &'module Module, - module_decls: Vec, + module: ModuleCtx<'module>, + module_decls: Vec, graphical_functions: Vec>, /// Maps table variable names to their base index in graphical_functions. /// For subscripted lookups, the actual table is at base_id + element_offset. table_base_ids: HashMap, GraphicalFunctionId>, - curr_code: ByteCodeBuilder, + curr_code: SymbolicByteCodeBuilder, // Array support fields pub(super) dimensions: Vec, pub(super) subdim_relations: Vec, @@ -41,26 +98,54 @@ pub(super) struct Compiler<'module> { /// element name up front -- with a linear-scan intern that was O(D^2) /// string comparisons per fragment (GH #655). name_ids: HashMap, - static_views: Vec, + static_views: Vec, dim_lists: Vec<(u8, [u16; 4])>, // Iteration context - set when compiling inside AssignTemp in_iteration: bool, /// When in optimized iteration mode, maps pre-pushed views to their stack offset. - /// Each entry is (StaticArrayView, stack_offset) where stack_offset is 1-based from top. + /// Each entry is (SymbolicStaticView, stack_offset) where stack_offset is 1-based from top. /// The output view is always at offset (n_source_views + 1). - iter_source_views: Option>, + iter_source_views: Option>, } impl<'module> Compiler<'module> { - pub(super) fn new(module: &'module Module) -> Compiler<'module> { - // Pre-populate graphical_functions with all tables and record base IDs + pub(super) fn new(module: ModuleCtx<'module>) -> Compiler<'module> { + // Pre-populate graphical_functions with all tables and record base IDs. + // + // Iterated in sorted ident order, NOT `HashMap` order, and that is + // load-bearing rather than cosmetic. This loop assigns both the layout + // of `graphical_functions` and every `base_gf` operand the emitted + // `Lookup`/`LookupArray` opcodes carry, so with `HashMap` order a model + // whose fragment holds two or more table-bearing variables compiled to + // a *different* (still self-consistent, still numerically correct) + // bytecode on every run. `PerVarBytecodes` is a salsa-cached value with + // a derived `PartialEq`, so that defeats backdating exactly as an + // unordered `temp_sizes` did (`db::assemble::temp_sizes_by_id`), and the + // assembled `CompiledModule` was not reproducible: measured differing on + // 18 of 23 fresh-database repeats. It reaches shipped models -- + // `test/metasd/theil-statistics/Theil_2011.mdl` compiles a fragment + // holding `["dummy_data", "dummy_simulation"]`. + // + // Nothing downstream depends on WHICH order is chosen, only that a + // fragment's `base_gf` operands agree with its own + // `graphical_functions` vector and that distinct variables' blocks stay + // disjoint -- both of which sorting preserves. Checked: the VM reads + // `graphical_functions[base_gf + element_offset]` self-relatively; + // `resolve` passes `base_gf`/`table_count` through untouched (they + // are table indices, not layout offsets); and + // `symbolic::gf_blocks_of_fragment` derives its blocks from the + // fragment's own opcode runs (sorting them itself) while + // `FragmentMerger::absorb_gf` dedups on block CONTENT, so #582's + // cross-fragment GF dedup is order-insensitive by construction. let mut graphical_functions = Vec::new(); let mut table_base_ids = HashMap::new(); - for (ident, tables) in &module.tables { + let mut table_idents: Vec<&Ident> = module.tables.keys().collect(); + table_idents.sort_unstable(); + for ident in table_idents { let base_gf = graphical_functions.len() as GraphicalFunctionId; table_base_ids.insert(ident.clone(), base_gf); - for table in tables { + for table in &module.tables[ident] { graphical_functions.push(table.data.clone()); } } @@ -70,7 +155,7 @@ impl<'module> Compiler<'module> { module_decls: vec![], graphical_functions, table_base_ids, - curr_code: ByteCodeBuilder::default(), + curr_code: SymbolicByteCodeBuilder::default(), dimensions: vec![], subdim_relations: vec![], names: vec![], @@ -92,7 +177,7 @@ impl<'module> Compiler<'module> { /// Note: Subdimension relations are populated lazily via `get_or_add_subdim_relation` /// when ViewStarRange bytecode is emitted, rather than pre-computing all pairs. fn populate_dimension_metadata(&mut self) { - for dim in &self.module.dimensions { + for dim in self.module.dimensions { let dim_name = dim.name(); let name_id = self.intern_name(dim_name); @@ -200,7 +285,7 @@ impl<'module> Compiler<'module> { } /// Add a static view and return its ViewId - fn add_static_view(&mut self, view: StaticArrayView) -> ViewId { + fn add_static_view(&mut self, view: SymbolicStaticView) -> ViewId { self.static_views.push(view); (self.static_views.len() - 1) as ViewId } @@ -209,37 +294,57 @@ impl<'module> Compiler<'module> { /// expression, i.e. the product of its full declared dimensions. This is /// the genuine-Vensim VECTOR ELM MAP out-of-range bound (`:NA:` is /// returned for an offset that would map outside the source variable's - /// full storage). Each model variable owns a unique `[offset, offset+size)` - /// slot range (offsets are assigned by `i += size`), so the base offset - /// uniquely identifies the variable and its full `size`. Falls back to - /// the lowered view's element count when the source is not a plain - /// variable/subscript reference (e.g. a scalar `Var`), which is the - /// correct full extent for those non-sliced shapes. + /// full storage). + /// + /// This is a direct [`VarSizes`] lookup keyed by the reference itself, + /// because the reference does not always name the variable being asked + /// about: a cross-module `m·x` names the module INSTANCE `m` with `x`'s slot + /// inside it. `VarSizes` holds one entry per variable a reference can + /// address in whole -- a sub-model's variables among them, at their slots + /// within the instance -- so both shapes are one lookup. + /// + /// A reference the table does not hold starts mid-variable: it names one + /// element of a bigger array, and the array's extent is not that element's. + /// Those fall back to the lowered view's element count, which is the correct + /// full extent for the non-sliced shapes. fn full_source_len(&self, source: &Expr) -> u32 { - let (base_off, view_len) = match source { - Expr::StaticSubscript(off, view, _) => { - (Some(*off), view.dims.iter().product::().max(1)) + let (base, view_len) = match source { + Expr::StaticSubscript(base, view, _) => { + (Some(base), view.dims.iter().product::().max(1)) } - Expr::Var(off, _) => (Some(*off), 1usize), + // A *dynamic* subscript (`arr[i]`, `i` a variable) is the same + // shape as `StaticSubscript` for this purpose -- the source is + // still the whole `arr` variable, only the base element is chosen + // at runtime. Without this arm it fell to the `_` case and reported + // a full extent of 1, so `VECTOR ELM MAP(arr[i], offsets)` returned + // NaN for every element whenever `i` selected anything but `arr`'s + // first element, and for any non-zero offset at all. + // + // No wasm-backend parity test accompanies the VM regression test + // (`array_tests::…::elm_map_dynamic_source_subscript_uses_full_variable_extent_vm`) + // because the backends cannot diverge here: `wasmgen::vector` + // rejects a dynamically-subscripted ELM MAP source outright + // (`source_view.runtime_off_local.is_some()` => + // `WasmGenError::Unsupported`), so this shape never reaches wasm + // lowering at all. + Expr::Subscript(base, _, bounds, _) => { + (Some(base), bounds.iter().product::().max(1)) + } + Expr::Var(base, _) => (Some(base), 1usize), Expr::TempArray(_, view, _) => (None, view.dims.iter().product::().max(1)), _ => (None, 1usize), }; - if let Some(base_off) = base_off { - let model_offsets = &self.module.offsets[&self.module.ident]; - if let Some(size) = model_offsets - .values() - .find(|(off, _)| *off == base_off) - .map(|(_, size)| *size) - { - return size as u32; - } + if let Some(base) = base + && let Some(size) = self.module.var_sizes.get(base) + { + return *size as u32; } view_len as u32 } - /// Convert an ArrayView to a StaticArrayView for a variable - fn array_view_to_static(&mut self, base_off: usize, view: &ArrayView) -> StaticArrayView { + /// Convert an ArrayView to a SymbolicStaticView for a variable + fn array_view_to_static(&mut self, base: &VarRef, view: &ArrayView) -> SymbolicStaticView { // Convert sparse info let sparse: SmallVec<[RuntimeSparseMapping; 2]> = view .sparse @@ -265,9 +370,8 @@ impl<'module> Compiler<'module> { }) .collect(); - StaticArrayView { - base_off: base_off as u32, - is_temp: false, + SymbolicStaticView { + base: SymStaticViewBase::Var(base.clone()), dims: view.dims.iter().map(|&d| d as u16).collect(), strides: view.strides.iter().map(|&s| s as i32).collect(), offset: view.offset as u32, @@ -276,8 +380,8 @@ impl<'module> Compiler<'module> { } } - /// Convert an ArrayView to a StaticArrayView for a temp array - fn array_view_to_static_temp(&mut self, temp_id: u32, view: &ArrayView) -> StaticArrayView { + /// Convert an ArrayView to a SymbolicStaticView for a temp array + fn array_view_to_static_temp(&mut self, temp_id: u32, view: &ArrayView) -> SymbolicStaticView { // Look up or create DimIds for each dimension using the dim_names let dim_ids: SmallVec<[DimId; 4]> = view .dim_names @@ -306,9 +410,8 @@ impl<'module> Compiler<'module> { stride *= view.dims[i] as i32; } - StaticArrayView { - base_off: temp_id, - is_temp: true, + SymbolicStaticView { + base: SymStaticViewBase::Temp(temp_id), dims: view.dims.iter().map(|&d| d as u16).collect(), strides, offset: 0, @@ -331,12 +434,10 @@ impl<'module> Compiler<'module> { /// `BadTable` (loud-safe: an un-reconstructable arrayed-GF dependency must /// never become a silent stub -- GH #580 / AC7.5). fn arrayed_lookup_table_info(&self, table_expr: &Expr) -> Result<(GraphicalFunctionId, u16)> { - let module_offsets = &self.module.offsets[&self.module.ident]; - let base_off = match table_expr { - // Whole-array static subscript: the var's storage starts at `off` - // and the view spans the full array (offset 0). - Expr::StaticSubscript(off, _, _) => *off, - Expr::Var(off, _) => *off, + let base = match table_expr { + // Whole-array reference: the view spans the full array, so the + // reference sits at the variable's base. + Expr::StaticSubscript(base, _, _) | Expr::Var(base, _) => base, other => { return sim_err!( BadTable, @@ -348,17 +449,18 @@ impl<'module> Compiler<'module> { ); } }; - let table_ident = module_offsets - .iter() - .find(|(_, (base, _))| *base == base_off) - .map(|(k, _)| k.clone()) - .ok_or_else(|| { - crate::Error::new( - ErrorKind::Simulation, - ErrorCode::BadTable, - Some("could not find arrayed lookup table variable".to_string()), - ) - })?; + // A reference into the middle of a variable is not a whole-array base, + // and never was: the previous exact-base offset scan rejected it too, + // because a variable's slot range contains no other variable's base. + if !base.is_whole_var() { + return sim_err!( + BadTable, + "arrayed graphical-function apply expected a whole-array base, got an \ + element reference" + .to_string() + ); + } + let table_ident = base.name.clone(); let base_gf = *self.table_base_ids.get(&table_ident).ok_or_else(|| { crate::Error::new( ErrorKind::Simulation, @@ -381,30 +483,30 @@ impl<'module> Compiler<'module> { /// This is used for array operations that need to iterate over arrays. fn walk_expr_as_view(&mut self, expr: &Expr) -> Result<()> { match expr { - Expr::StaticSubscript(off, view, _) => { + Expr::StaticSubscript(base, view, _) => { // Create a static view and push it - let static_view = self.array_view_to_static(*off, view); + let static_view = self.array_view_to_static(base, view); let view_id = self.add_static_view(static_view); - self.push(Opcode::PushStaticView { view_id }); + self.push(SymbolicOpcode::PushStaticView { view_id }); Ok(()) } Expr::TempArray(id, view, _) => { // Create a static view for the temp array and push it let static_view = self.array_view_to_static_temp(*id, view); let view_id = self.add_static_view(static_view); - self.push(Opcode::PushStaticView { view_id }); + self.push(SymbolicOpcode::PushStaticView { view_id }); Ok(()) } - Expr::Var(off, _) => { + Expr::Var(base, _) => { // A bare variable reference used as an array - create a scalar view // This shouldn't normally happen for array operations, but handle it let view = ArrayView::contiguous(vec![1]); - let static_view = self.array_view_to_static(*off, &view); + let static_view = self.array_view_to_static(base, &view); let view_id = self.add_static_view(static_view); - self.push(Opcode::PushStaticView { view_id }); + self.push(SymbolicOpcode::PushStaticView { view_id }); Ok(()) } - Expr::Subscript(off, indices, bounds, _) => { + Expr::Subscript(base, indices, bounds, _) => { // Dynamic subscript with potential range indices // First, push a full view for the base array using explicit bounds let n_dims = bounds.len().min(4) as u8; @@ -414,8 +516,8 @@ impl<'module> Compiler<'module> { } let dim_list_id = self.dim_lists.len() as DimListId; self.dim_lists.push((n_dims, dims)); - self.push(Opcode::PushVarViewDirect { - base_off: *off as u16, + self.push(SymbolicOpcode::PushVarViewDirect { + var: base.clone(), dim_list_id, }); @@ -445,7 +547,7 @@ impl<'module> Compiler<'module> { // is a genuine compiler invariant violation, never // reachable from a real subscript index. self.walk_expr(expr)?.unwrap(); - self.push(Opcode::ViewSubscriptDynamic { + self.push(SymbolicOpcode::ViewSubscriptDynamic { dim_idx: effective_dim, }); singles_processed += 1; // Track collapse for subsequent indices @@ -455,7 +557,7 @@ impl<'module> Compiler<'module> { // a range bound can carry a recoverable lowering Err. self.walk_expr(start)?.unwrap(); self.walk_expr(end)?.unwrap(); - self.push(Opcode::ViewRangeDynamic { + self.push(SymbolicOpcode::ViewRangeDynamic { dim_idx: effective_dim, }); } @@ -477,18 +579,18 @@ impl<'module> Compiler<'module> { /// Emit the array-reduce pattern: push view, emit reduction opcode, pop view. /// Used by SUM, SIZE, STDDEV, MIN (1-arg), MAX (1-arg), and MEAN (1-arg). - fn emit_array_reduce(&mut self, arg: &Expr, opcode: Opcode) -> Result> { + fn emit_array_reduce(&mut self, arg: &Expr, opcode: SymbolicOpcode) -> Result> { self.walk_expr_as_view(arg)?; self.push(opcode); - self.push(Opcode::PopView {}); + self.push(SymbolicOpcode::PopView {}); Ok(Some(())) } - fn walk(&mut self, exprs: &[Expr]) -> Result { + fn walk(&mut self, exprs: &[Expr]) -> Result { for expr in exprs.iter() { self.walk_expr(expr)?; } - self.push(Opcode::Ret); + self.push(SymbolicOpcode::Ret); let curr = std::mem::take(&mut self.curr_code); @@ -499,16 +601,14 @@ impl<'module> Compiler<'module> { let result = match expr { Expr::Const(value, _) => { let id = self.curr_code.intern_literal(*value); - self.push(Opcode::LoadConstant { id }); + self.push(SymbolicOpcode::LoadConstant { id }); Some(()) } - Expr::Var(off, _) => { - self.push(Opcode::LoadVar { - off: *off as VariableOffset, - }); + Expr::Var(var, _) => { + self.push(SymbolicOpcode::LoadVar { var: var.clone() }); Some(()) } - Expr::Subscript(off, indices, bounds, _) => { + Expr::Subscript(base, indices, bounds, _) => { // For scalar access (old-style Subscript), all indices must be Single for (i, idx) in indices.iter().enumerate() { match idx { @@ -523,7 +623,7 @@ impl<'module> Compiler<'module> { // LTM synthetic fragment), never escalate to a panic (#363). self.walk_expr(expr)?.unwrap(); let bounds = bounds[i] as VariableOffset; - self.push(Opcode::PushSubscriptIndex { bounds }); + self.push(SymbolicOpcode::PushSubscriptIndex { bounds }); } SubscriptIndex::Range(_, _) => { // Range subscripts should be handled via walk_expr_as_view @@ -537,15 +637,13 @@ impl<'module> Compiler<'module> { } } assert!(indices.len() == bounds.len()); - self.push(Opcode::LoadSubscript { - off: *off as VariableOffset, - }); + self.push(SymbolicOpcode::LoadSubscript { var: base.clone() }); Some(()) } - Expr::StaticSubscript(off, view, _) => { + Expr::StaticSubscript(base, view, _) => { if self.in_iteration { // In iteration context with optimized view hoisting - let static_view = self.array_view_to_static(*off, view); + let static_view = self.array_view_to_static(base, view); let offset = self.find_iter_view_offset(&static_view).unwrap_or_else(|| { unreachable!( @@ -553,12 +651,14 @@ impl<'module> Compiler<'module> { collect_iter_source_views_impl and walk_expr should visit same nodes" ) }); - self.push(Opcode::LoadIterViewAt { offset }); + self.push(SymbolicOpcode::LoadIterViewAt { offset }); Some(()) } else if view.dims.iter().product::() == 1 { - // Scalar result - compute final offset and load - let final_off = (*off + view.offset) as VariableOffset; - self.push(Opcode::LoadVar { off: final_off }); + // Scalar result - the view has collapsed to one element, so + // read it directly. + self.push(SymbolicOpcode::LoadVar { + var: base.offset_by(view.offset), + }); Some(()) } else { // Non-scalar array outside iteration context - this shouldn't happen @@ -580,57 +680,54 @@ impl<'module> Compiler<'module> { collect_iter_source_views_impl and walk_expr should visit same nodes" ) }); - self.push(Opcode::LoadIterViewAt { offset }); + self.push(SymbolicOpcode::LoadIterViewAt { offset }); Some(()) } else { // Outside iteration - push temp view for subsequent operations (like SUM) let static_view = self.array_view_to_static_temp(*id, view); let view_id = self.add_static_view(static_view); - self.push(Opcode::PushStaticView { view_id }); + self.push(SymbolicOpcode::PushStaticView { view_id }); // Note: caller (like array builtin) will use and pop this view None } } Expr::TempArrayElement(id, _view, idx, _) => { // Load a specific element from a temp array - self.push(Opcode::LoadTempConst { + self.push(SymbolicOpcode::LoadTempConst { temp_id: *id as TempId, index: *idx as u16, }); Some(()) } Expr::Dt(_) => { - self.push(Opcode::LoadGlobalVar { + self.push(SymbolicOpcode::LoadGlobalVar { off: DT_OFF as VariableOffset, }); Some(()) } Expr::App(builtin, _) => { - // Helper to extract table info from table expression - fn extract_table_info( - table_expr: &Expr, - module_offsets: &HashMap, (usize, usize)>, - ) -> Result<(Ident, Expr)> { + // Helper to extract table info from table expression. + // + // The table's identity and the element within it both come + // straight off the reference: `name` is the variable that owns + // the slot and `element_offset` is the element within it -- + // which is what the offset-range scan this used to do + // reconstructed. A reference whose base is not the variable's + // own (a cross-module `m·x`, whose owner in this model is the + // module variable) can only fail the `table_base_ids` lookup + // below, exactly as the scan's exact-base match used to fail. + fn extract_table_info(table_expr: &Expr) -> Result<(Ident, Expr)> { match table_expr { - Expr::Var(off, loc) => { - // Could be a simple scalar table or an element of an arrayed table - // (when subscript was static and compiled to a direct Var reference). - // Find the variable whose range contains this offset. - let (table_ident, base_off) = module_offsets - .iter() - .find(|(_, (base, size))| *off >= *base && *off < *base + *size) - .map(|(k, (base, _))| (k.clone(), *base)) - .ok_or_else(|| { - crate::Error::new( - ErrorKind::Simulation, - ErrorCode::BadTable, - Some("could not find table variable".to_string()), - ) - })?; - let elem_off = *off - base_off; - Ok((table_ident, Expr::Const(elem_off as f64, *loc))) + Expr::Var(var, loc) => { + // Either a scalar table or an element of an arrayed + // table (a static subscript that compiled down to a + // direct element reference). + Ok(( + var.name.clone(), + Expr::Const(var.element_offset as f64, *loc), + )) } - Expr::StaticSubscript(off, view, loc) => { + Expr::StaticSubscript(base, view, loc) => { // Static subscript - element offset is precomputed in the ArrayView // Reject ranges/wildcards - only single element selection is valid if view.size() > 1 { @@ -639,20 +736,12 @@ impl<'module> Compiler<'module> { "range subscripts not supported in lookup tables".to_string() ); } - let table_ident = module_offsets - .iter() - .find(|(_, (base, _))| *off == *base) - .map(|(k, _)| k.clone()) - .ok_or_else(|| { - crate::Error::new( - ErrorKind::Simulation, - ErrorCode::BadTable, - Some("could not find table variable".to_string()), - ) - })?; - Ok((table_ident, Expr::Const(view.offset as f64, *loc))) + Ok(( + base.name.clone(), + Expr::Const((base.element_offset + view.offset) as f64, *loc), + )) } - Expr::Subscript(off, subscript_indices, dim_sizes, _loc) => { + Expr::Subscript(base, subscript_indices, dim_sizes, _loc) => { // Subscripted table reference - compute element_offset // For a multi-dimensional subscript, compute linear offset // offset = sum(index_i * stride_i) where stride_i = product of sizes[i+1..] @@ -710,18 +799,24 @@ impl<'module> Compiler<'module> { stride *= dim_sizes.get(i).copied().unwrap_or(1); } - let table_ident = module_offsets - .iter() - .find(|(_, (base, _))| *off == *base) - .map(|(k, _)| k.clone()) - .ok_or_else(|| { - crate::Error::new( - ErrorKind::Simulation, - ErrorCode::BadTable, - Some("could not find table variable".to_string()), - ) - })?; - Ok((table_ident, offset_expr.unwrap_or(Expr::Const(0.0, *_loc)))) + // A dynamically-subscripted table reference is always + // to the whole table variable: the index expression + // selects the element. `base.element_offset` is + // therefore 0, and a non-zero one (a cross-module + // `m·x`) has no representable table identity here -- + // the previous exact-base offset scan rejected it too. + if !base.is_whole_var() { + return sim_err!( + BadTable, + "subscripted lookup table reference must name a \ + variable of this model" + .to_string() + ); + } + Ok(( + base.name.clone(), + offset_expr.unwrap_or(Expr::Const(0.0, *_loc)), + )) } _ => { sim_err!( @@ -735,9 +830,7 @@ impl<'module> Compiler<'module> { // lookups are special if let BuiltinFn::Lookup(table_expr, index, _loc) = builtin { - let module_offsets = &self.module.offsets[&self.module.ident]; - let (table_ident, element_offset_expr) = - extract_table_info(table_expr, module_offsets)?; + let (table_ident, element_offset_expr) = extract_table_info(table_expr)?; // Look up the base_gf for this table variable let base_gf = *self.table_base_ids.get(&table_ident).ok_or_else(|| { @@ -759,7 +852,7 @@ impl<'module> Compiler<'module> { // Emit: push element_offset, push lookup_index, Lookup { base_gf, table_count, mode } self.walk_expr(&element_offset_expr)?.unwrap(); self.walk_expr(index)?.unwrap(); - self.push(Opcode::Lookup { + self.push(SymbolicOpcode::Lookup { base_gf, table_count, mode: LookupMode::Interpolate, @@ -776,9 +869,7 @@ impl<'module> Compiler<'module> { } else { LookupMode::Backward }; - let module_offsets = &self.module.offsets[&self.module.ident]; - let (table_ident, element_offset_expr) = - extract_table_info(table_expr, module_offsets)?; + let (table_ident, element_offset_expr) = extract_table_info(table_expr)?; let base_gf = *self.table_base_ids.get(&table_ident).ok_or_else(|| { crate::Error::new( @@ -797,7 +888,7 @@ impl<'module> Compiler<'module> { self.walk_expr(&element_offset_expr)?.unwrap(); self.walk_expr(index)?.unwrap(); - self.push(Opcode::Lookup { + self.push(SymbolicOpcode::Lookup { base_gf, table_count, mode, @@ -812,7 +903,7 @@ impl<'module> Compiler<'module> { } else { self.curr_code.intern_literal(0.0) }; - self.push(Opcode::LoadConstant { id }); + self.push(SymbolicOpcode::LoadConstant { id }); return Ok(Some(())); }; @@ -831,11 +922,11 @@ impl<'module> Compiler<'module> { // compile-time constant. Anything else (dynamic indices, // expressions) was rewritten through a helper aux at parse // time, so reaching here with one is a compiler bug. - let static_slot_offset = |arg: &Expr| -> Option { + let static_slot = |arg: &Expr| -> Option { match arg { - Expr::Var(off, _) => Some(*off as VariableOffset), - Expr::StaticSubscript(off, view, _) if view.dims.is_empty() => { - Some((*off + view.offset) as VariableOffset) + Expr::Var(var, _) => Some(var.clone()), + Expr::StaticSubscript(base, view, _) if view.dims.is_empty() => { + Some(base.offset_by(view.offset)) } _ => None, } @@ -843,9 +934,9 @@ impl<'module> Compiler<'module> { match builtin { BuiltinFn::Previous(arg, fallback) => { self.walk_expr(fallback)?.unwrap(); - match static_slot_offset(arg.as_ref()) { - Some(off) => { - self.push(Opcode::LoadPrev { off }); + match static_slot(arg.as_ref()) { + Some(var) => { + self.push(SymbolicOpcode::SymLoadPrev { var }); } None => { return sim_err!( @@ -858,8 +949,8 @@ impl<'module> Compiler<'module> { return Ok(Some(())); } BuiltinFn::Init(arg) => { - let off = match static_slot_offset(arg.as_ref()) { - Some(off) => off, + let var = match static_slot(arg.as_ref()) { + Some(var) => var, None => { return sim_err!( NotSimulatable, @@ -867,7 +958,7 @@ impl<'module> Compiler<'module> { ); } }; - self.push(Opcode::LoadInitial { off }); + self.push(SymbolicOpcode::SymLoadInitial { var }); return Ok(Some(())); } _ => {} @@ -885,7 +976,7 @@ impl<'module> Compiler<'module> { BuiltinFn::FinalTime => FINAL_TIME_OFF, _ => unreachable!(), } as u16; - self.push(Opcode::LoadGlobalVar { off }); + self.push(SymbolicOpcode::LoadGlobalVar { off }); return Ok(Some(())); } BuiltinFn::Lookup(_, _, _) @@ -899,7 +990,7 @@ impl<'module> Compiler<'module> { _ => unreachable!(), }; let id = self.curr_code.intern_literal(lit); - self.push(Opcode::LoadConstant { id }); + self.push(SymbolicOpcode::LoadConstant { id }); return Ok(Some(())); } BuiltinFn::Abs(a) @@ -917,23 +1008,23 @@ impl<'module> Compiler<'module> { | BuiltinFn::Tan(a) => { self.walk_expr(a)?.unwrap(); let id = self.curr_code.intern_literal(0.0); - self.push(Opcode::LoadConstant { id }); - self.push(Opcode::LoadConstant { id }); + self.push(SymbolicOpcode::LoadConstant { id }); + self.push(SymbolicOpcode::LoadConstant { id }); } BuiltinFn::Step(a, b) => { self.walk_expr(a)?.unwrap(); self.walk_expr(b)?.unwrap(); let id = self.curr_code.intern_literal(0.0); - self.push(Opcode::LoadConstant { id }); + self.push(SymbolicOpcode::LoadConstant { id }); } BuiltinFn::Max(a, b) => { if let Some(b) = b { self.walk_expr(a)?.unwrap(); self.walk_expr(b)?.unwrap(); let id = self.curr_code.intern_literal(0.0); - self.push(Opcode::LoadConstant { id }); + self.push(SymbolicOpcode::LoadConstant { id }); } else { - return self.emit_array_reduce(a, Opcode::ArrayMax {}); + return self.emit_array_reduce(a, SymbolicOpcode::ArrayMax {}); } } BuiltinFn::Min(a, b) => { @@ -941,16 +1032,16 @@ impl<'module> Compiler<'module> { self.walk_expr(a)?.unwrap(); self.walk_expr(b)?.unwrap(); let id = self.curr_code.intern_literal(0.0); - self.push(Opcode::LoadConstant { id }); + self.push(SymbolicOpcode::LoadConstant { id }); } else { - return self.emit_array_reduce(a, Opcode::ArrayMin {}); + return self.emit_array_reduce(a, SymbolicOpcode::ArrayMin {}); } } BuiltinFn::Quantum(a, b) => { self.walk_expr(a)?.unwrap(); self.walk_expr(b)?.unwrap(); let id = self.curr_code.intern_literal(0.0); - self.push(Opcode::LoadConstant { id }); + self.push(SymbolicOpcode::LoadConstant { id }); } BuiltinFn::Pulse(a, b, c) => { self.walk_expr(a)?.unwrap(); @@ -959,7 +1050,7 @@ impl<'module> Compiler<'module> { self.walk_expr(c.as_ref().unwrap())?.unwrap() } else { let id = self.curr_code.intern_literal(0.0); - self.push(Opcode::LoadConstant { id }); + self.push(SymbolicOpcode::LoadConstant { id }); }; } BuiltinFn::Ramp(a, b, c) => { @@ -978,7 +1069,7 @@ impl<'module> Compiler<'module> { // root model (where slot 3 IS final_time); inside a // submodule it reads an unrelated body slot (or drops // the fragment when that slot has no symbolic mapping). - self.push(Opcode::LoadGlobalVar { + self.push(SymbolicOpcode::LoadGlobalVar { off: FINAL_TIME_OFF as u16, }); }; @@ -1000,7 +1091,7 @@ impl<'module> Compiler<'module> { }; if c.is_none() { let id = self.curr_code.intern_literal(0.0); - self.push(Opcode::LoadConstant { id }); + self.push(SymbolicOpcode::LoadConstant { id }); } } BuiltinFn::Sshape(a, b, c) => { @@ -1019,7 +1110,8 @@ impl<'module> Compiler<'module> { | Expr::TempArray(..) | Expr::Var(..) | Expr::Subscript(..) => { - return self.emit_array_reduce(&args[0], Opcode::ArrayMean {}); + return self + .emit_array_reduce(&args[0], SymbolicOpcode::ArrayMean {}); } _ => { self.walk_expr(&args[0])?.unwrap(); @@ -1030,16 +1122,16 @@ impl<'module> Compiler<'module> { // Multi-argument scalar mean: (arg1 + arg2 + ... + argN) / N let id = self.curr_code.intern_literal(0.0); - self.push(Opcode::LoadConstant { id }); + self.push(SymbolicOpcode::LoadConstant { id }); for arg in args.iter() { self.walk_expr(arg)?.unwrap(); - self.push(Opcode::Op2 { op: Op2::Add }); + self.push(SymbolicOpcode::Op2 { op: Op2::Add }); } let id = self.curr_code.intern_literal(args.len() as f64); - self.push(Opcode::LoadConstant { id }); - self.push(Opcode::Op2 { op: Op2::Div }); + self.push(SymbolicOpcode::LoadConstant { id }); + self.push(SymbolicOpcode::Op2 { op: Op2::Div }); return Ok(Some(())); } BuiltinFn::Rank(_, _) => { @@ -1049,22 +1141,22 @@ impl<'module> Compiler<'module> { ); } BuiltinFn::Size(arg) => { - return self.emit_array_reduce(arg, Opcode::ArraySize {}); + return self.emit_array_reduce(arg, SymbolicOpcode::ArraySize {}); } BuiltinFn::Stddev(arg) => { - return self.emit_array_reduce(arg, Opcode::ArrayStddev {}); + return self.emit_array_reduce(arg, SymbolicOpcode::ArrayStddev {}); } BuiltinFn::Sum(arg) => { - return self.emit_array_reduce(arg, Opcode::ArraySum {}); + return self.emit_array_reduce(arg, SymbolicOpcode::ArraySum {}); } BuiltinFn::VectorSelect(sel, expr, max_val, action, _err) => { self.walk_expr_as_view(sel)?; self.walk_expr_as_view(expr)?; self.walk_expr(max_val)?.unwrap(); self.walk_expr(action)?.unwrap(); - self.push(Opcode::VectorSelect {}); - self.push(Opcode::PopView {}); - self.push(Opcode::PopView {}); + self.push(SymbolicOpcode::VectorSelect {}); + self.push(SymbolicOpcode::PopView {}); + self.push(SymbolicOpcode::PopView {}); return Ok(Some(())); } BuiltinFn::Previous(_, _) | BuiltinFn::Init(_) => { @@ -1149,7 +1241,7 @@ impl<'module> Compiler<'module> { } }; - self.push(Opcode::Apply { func }); + self.push(SymbolicOpcode::Apply { func }); Some(()) } Expr::EvalModule(ident, model_name, input_set, args) => { @@ -1159,22 +1251,24 @@ impl<'module> Compiler<'module> { // (consistent with every other operand walk in this fn). self.walk_expr(arg)?.unwrap(); } - let module_offsets = &self.module.offsets[&self.module.ident]; - self.module_decls.push(ModuleDeclaration { + // The instance's base slot is the module variable's own first + // slot; naming it is all the declaration needs, and assembly + // resolves it against the final layout like any other reference. + self.module_decls.push(SymbolicModuleDecl { model_name: model_name.clone(), input_set: input_set.clone(), - off: module_offsets[ident].0, + var: VarRef::base(ident.clone()), }); let id = (self.module_decls.len() - 1) as ModuleId; - self.push(Opcode::EvalModule { + self.push(SymbolicOpcode::EvalModule { id, n_inputs: args.len() as u8, }); None } Expr::ModuleInput(off, _) => { - self.push(Opcode::LoadModuleInput { + self.push(SymbolicOpcode::LoadModuleInput { input: *off as ModuleInputOffset, }); Some(()) @@ -1183,23 +1277,23 @@ impl<'module> Compiler<'module> { self.walk_expr(lhs)?.unwrap(); self.walk_expr(rhs)?.unwrap(); let opcode = match op { - BinaryOp::Add => Opcode::Op2 { op: Op2::Add }, - BinaryOp::Sub => Opcode::Op2 { op: Op2::Sub }, - BinaryOp::Exp => Opcode::Op2 { op: Op2::Exp }, - BinaryOp::Mul => Opcode::Op2 { op: Op2::Mul }, - BinaryOp::Div => Opcode::Op2 { op: Op2::Div }, - BinaryOp::Mod => Opcode::Op2 { op: Op2::Mod }, - BinaryOp::Gt => Opcode::Op2 { op: Op2::Gt }, - BinaryOp::Gte => Opcode::Op2 { op: Op2::Gte }, - BinaryOp::Lt => Opcode::Op2 { op: Op2::Lt }, - BinaryOp::Lte => Opcode::Op2 { op: Op2::Lte }, - BinaryOp::Eq => Opcode::Op2 { op: Op2::Eq }, + BinaryOp::Add => SymbolicOpcode::Op2 { op: Op2::Add }, + BinaryOp::Sub => SymbolicOpcode::Op2 { op: Op2::Sub }, + BinaryOp::Exp => SymbolicOpcode::Op2 { op: Op2::Exp }, + BinaryOp::Mul => SymbolicOpcode::Op2 { op: Op2::Mul }, + BinaryOp::Div => SymbolicOpcode::Op2 { op: Op2::Div }, + BinaryOp::Mod => SymbolicOpcode::Op2 { op: Op2::Mod }, + BinaryOp::Gt => SymbolicOpcode::Op2 { op: Op2::Gt }, + BinaryOp::Gte => SymbolicOpcode::Op2 { op: Op2::Gte }, + BinaryOp::Lt => SymbolicOpcode::Op2 { op: Op2::Lt }, + BinaryOp::Lte => SymbolicOpcode::Op2 { op: Op2::Lte }, + BinaryOp::Eq => SymbolicOpcode::Op2 { op: Op2::Eq }, BinaryOp::Neq => { - self.push(Opcode::Op2 { op: Op2::Eq }); - Opcode::Not {} + self.push(SymbolicOpcode::Op2 { op: Op2::Eq }); + SymbolicOpcode::Not {} } - BinaryOp::And => Opcode::Op2 { op: Op2::And }, - BinaryOp::Or => Opcode::Op2 { op: Op2::Or }, + BinaryOp::And => SymbolicOpcode::Op2 { op: Op2::And }, + BinaryOp::Or => SymbolicOpcode::Op2 { op: Op2::Or }, }; self.push(opcode); Some(()) @@ -1207,7 +1301,7 @@ impl<'module> Compiler<'module> { Expr::Op1(op, rhs, _) => { self.walk_expr(rhs)?.unwrap(); match op { - UnaryOp::Not => self.push(Opcode::Not {}), + UnaryOp::Not => self.push(SymbolicOpcode::Not {}), UnaryOp::Transpose => { unreachable!("Transpose should be handled at compile time in lower()"); } @@ -1218,30 +1312,44 @@ impl<'module> Compiler<'module> { self.walk_expr(t)?.unwrap(); self.walk_expr(f)?.unwrap(); self.walk_expr(cond)?.unwrap(); - self.push(Opcode::SetCond {}); - self.push(Opcode::If {}); + self.push(SymbolicOpcode::SetCond {}); + self.push(SymbolicOpcode::If {}); Some(()) } - Expr::AssignCurr(off, rhs) => { + Expr::AssignCurr(dst, rhs) => { if let Expr::Const(value, _) = rhs.as_ref() { let id = self.curr_code.push_named_literal(*value); - self.push(Opcode::AssignConstCurr { - off: *off as VariableOffset, + self.push(SymbolicOpcode::AssignConstCurr { + var: dst.clone(), literal_id: id, }); } else { self.walk_expr(rhs)?.unwrap(); - self.push(Opcode::AssignCurr { - off: *off as VariableOffset, - }); + self.push(SymbolicOpcode::AssignCurr { var: dst.clone() }); } None } - Expr::AssignNext(off, rhs) => { + // A stock update -- the only thing that writes `next[]`. It is + // emitted as the fused `BinOpAssignNext` directly, because + // `build_stock_update_expr` always produces `Op2(Add, curr, net*dt)` + // and so the operand walk always ends in an `Op2`. There is no + // un-fused `Opcode::AssignNext` to fall back to; a stock update + // arriving in any other shape is a compile error here rather than a + // silently different program (see + // `SymbolicByteCodeBuilder::fuse_trailing_op2_into_assign_next`). + Expr::AssignNext(dst, rhs) => { self.walk_expr(rhs)?.unwrap(); - self.push(Opcode::AssignNext { - off: *off as VariableOffset, - }); + if !self.curr_code.fuse_trailing_op2_into_assign_next(dst) { + return sim_err!( + NotSimulatable, + format!( + "stock update for '{}' does not end in a binary \ + operation, so it cannot be emitted as a next-value \ + assignment", + dst.name + ) + ); + } None } Expr::AssignTemp(id, rhs, view) => { @@ -1258,30 +1366,30 @@ impl<'module> Compiler<'module> { let full_source_len = self.full_source_len(source); self.walk_expr_as_view(source)?; self.walk_expr_as_view(offset)?; - self.push(Opcode::VectorElmMap { + self.push(SymbolicOpcode::VectorElmMap { write_temp_id: *id as TempId, full_source_len, }); - self.push(Opcode::PopView {}); - self.push(Opcode::PopView {}); + self.push(SymbolicOpcode::PopView {}); + self.push(SymbolicOpcode::PopView {}); return Ok(None); } BuiltinFn::VectorSortOrder(array, direction) => { self.walk_expr_as_view(array)?; self.walk_expr(direction)?.unwrap(); - self.push(Opcode::VectorSortOrder { + self.push(SymbolicOpcode::VectorSortOrder { write_temp_id: *id as TempId, }); - self.push(Opcode::PopView {}); + self.push(SymbolicOpcode::PopView {}); return Ok(None); } BuiltinFn::Rank(array, direction) => { self.walk_expr_as_view(array)?; self.walk_expr(direction)?.unwrap(); - self.push(Opcode::Rank { + self.push(SymbolicOpcode::Rank { write_temp_id: *id as TempId, }); - self.push(Opcode::PopView {}); + self.push(SymbolicOpcode::PopView {}); return Ok(None); } // Per-element arrayed-GF lookup (GH #580 Bug B): @@ -1306,24 +1414,24 @@ impl<'module> Compiler<'module> { self.arrayed_lookup_table_info(table_expr)?; self.walk_expr_as_view(table_expr)?; self.walk_expr(index)?.unwrap(); - self.push(Opcode::LookupArray { + self.push(SymbolicOpcode::LookupArray { base_gf, table_count, mode, write_temp_id: *id as TempId, }); - self.push(Opcode::PopView {}); + self.push(SymbolicOpcode::PopView {}); return Ok(None); } BuiltinFn::AllocateAvailable(requests, profile, avail) => { self.walk_expr_as_view(requests)?; self.walk_expr_as_view(profile)?; self.walk_expr(avail)?.unwrap(); - self.push(Opcode::AllocateAvailable { + self.push(SymbolicOpcode::AllocateAvailable { write_temp_id: *id as TempId, }); - self.push(Opcode::PopView {}); - self.push(Opcode::PopView {}); + self.push(SymbolicOpcode::PopView {}); + self.push(SymbolicOpcode::PopView {}); return Ok(None); } BuiltinFn::AllocateByPriority(requests, priority, _size, width, supply) => { @@ -1334,11 +1442,11 @@ impl<'module> Compiler<'module> { self.walk_expr_as_view(priority)?; self.walk_expr(width)?.unwrap(); self.walk_expr(supply)?.unwrap(); - self.push(Opcode::AllocateByPriority { + self.push(SymbolicOpcode::AllocateByPriority { write_temp_id: *id as TempId, }); - self.push(Opcode::PopView {}); - self.push(Opcode::PopView {}); + self.push(SymbolicOpcode::PopView {}); + self.push(SymbolicOpcode::PopView {}); return Ok(None); } _ => {} // fall through to existing BeginIter logic @@ -1383,13 +1491,13 @@ impl<'module> Compiler<'module> { // 2. Push the OUTPUT temp's view for iteration size let output_static_view = self.array_view_to_static_temp(*id, view); let output_view_id = self.add_static_view(output_static_view); - self.push(Opcode::PushStaticView { + self.push(SymbolicOpcode::PushStaticView { view_id: output_view_id, }); // 3. Begin iteration - MUST be before source views are pushed // BeginIter captures view_stack.last() as the iteration view - self.push(Opcode::BeginIter { + self.push(SymbolicOpcode::BeginIter { write_temp_id: *id as TempId, has_write_temp: true, }); @@ -1397,12 +1505,12 @@ impl<'module> Compiler<'module> { // 4. Push all source views AFTER BeginIter and record their stack offsets // After this, view_stack looks like: [output_view, src1, src2, ...] // So src1 is at offset n_source_views, src2 at n_source_views-1, etc. - let mut iter_views_with_offsets: Vec<(StaticArrayView, u8)> = + let mut iter_views_with_offsets: Vec<(SymbolicStaticView, u8)> = Vec::with_capacity(n_source_views); for (i, src_view) in source_views.into_iter().enumerate() { let view_id = self.add_static_view(src_view.clone()); - self.push(Opcode::PushStaticView { view_id }); + self.push(SymbolicOpcode::PushStaticView { view_id }); // Offset is counted from top: last pushed is at offset 1 // First pushed source view will be at offset n_source_views after all are pushed let offset = (n_source_views - i) as u8; @@ -1420,22 +1528,22 @@ impl<'module> Compiler<'module> { self.in_iteration = false; // Store the result to temp - self.push(Opcode::StoreIterElement {}); + self.push(SymbolicOpcode::StoreIterElement {}); // Calculate jump offset (negative, back to loop start) let next_iter_pos = self.curr_code.len(); let jump_back = (loop_start as isize - next_iter_pos as isize) as i16; - self.push(Opcode::NextIterOrJump { jump_back }); - self.push(Opcode::EndIter {}); + self.push(SymbolicOpcode::NextIterOrJump { jump_back }); + self.push(SymbolicOpcode::EndIter {}); // 6. Pop all source views (in reverse order of push) for _ in 0..n_source_views { - self.push(Opcode::PopView {}); + self.push(SymbolicOpcode::PopView {}); } // 7. Pop output view - self.push(Opcode::PopView {}); + self.push(SymbolicOpcode::PopView {}); // AssignTemp doesn't produce a value on the stack None @@ -1444,14 +1552,14 @@ impl<'module> Compiler<'module> { Ok(result) } - fn push(&mut self, op: Opcode) { + fn push(&mut self, op: SymbolicOpcode) { self.curr_code.push_opcode(op) } /// Collect all source views referenced in an expression. /// This traverses the expression and collects StaticArrayView data for each /// StaticSubscript and TempArray node, deduplicating identical views. - fn collect_iter_source_views(&mut self, expr: &Expr) -> Vec { + fn collect_iter_source_views(&mut self, expr: &Expr) -> Vec { let mut views = Vec::new(); let mut seen = std::collections::HashSet::new(); self.collect_iter_source_views_impl(expr, &mut views, &mut seen); @@ -1461,12 +1569,12 @@ impl<'module> Compiler<'module> { fn collect_iter_source_views_impl( &mut self, expr: &Expr, - views: &mut Vec, - seen: &mut std::collections::HashSet, + views: &mut Vec, + seen: &mut std::collections::HashSet, ) { match expr { - Expr::StaticSubscript(off, view, _) => { - let static_view = self.array_view_to_static(*off, view); + Expr::StaticSubscript(base, view, _) => { + let static_view = self.array_view_to_static(base, view); // O(1) deduplication using HashSet if seen.insert(static_view.clone()) { views.push(static_view); @@ -1513,8 +1621,8 @@ impl<'module> Compiler<'module> { fn collect_builtin_views( &mut self, builtin: &BuiltinFn, - views: &mut Vec, - seen: &mut std::collections::HashSet, + views: &mut Vec, + seen: &mut std::collections::HashSet, ) { use crate::builtins::BuiltinFn::*; match builtin { @@ -1603,7 +1711,7 @@ impl<'module> Compiler<'module> { /// Find the stack offset for a view that was pre-pushed. /// Returns Some(offset) if found, where offset is 1-based from stack top. - fn find_iter_view_offset(&self, view: &StaticArrayView) -> Option { + fn find_iter_view_offset(&self, view: &SymbolicStaticView) -> Option { self.iter_source_views.as_ref().and_then(|views| { views .iter() @@ -1612,60 +1720,61 @@ impl<'module> Compiler<'module> { }) } - pub(super) fn compile(mut self) -> Result { + pub(super) fn compile(mut self) -> Result { + // The runlists live for `'module`, independent of `&mut self`, so bind + // them out of the (Copy) context before the mutable walks. + let initials_by_var = self.module.runlist_initials_by_var; + let flows = self.module.runlist_flows; + let stocks = self.module.runlist_stocks; + let temp_sizes = self.module.temp_sizes; + // Compile each variable's initials separately - let compiled_initials: Vec = self - .module - .runlist_initials_by_var + let compiled_initials: Vec = initials_by_var .iter() .map(|var_init| { let bytecode = self.walk(&var_init.ast)?; - Ok(CompiledInitial { + Ok(SymbolicCompiledInitial { ident: var_init.ident.clone(), - offsets: var_init.offsets.clone(), bytecode, }) }) .collect::>>()?; - let compiled_initials = Arc::new(compiled_initials); - let compiled_flows = Arc::new(self.walk(&self.module.runlist_flows)?); - let compiled_stocks = Arc::new(self.walk(&self.module.runlist_stocks)?); + let compiled_flows = self.walk(flows)?; + let compiled_stocks = self.walk(stocks)?; // Build temp info from module - let mut temp_offsets = Vec::with_capacity(self.module.n_temps); + let mut temp_offsets = Vec::with_capacity(temp_sizes.len()); let mut offset = 0usize; - for &size in &self.module.temp_sizes { + for &size in temp_sizes { temp_offsets.push(offset); offset += size; } let temp_total_size = offset; - Ok(CompiledModule { + Ok(SymbolicCompiledModule { ident: self.module.ident.clone(), - n_slots: self.module.n_slots, - context: Arc::new(ByteCodeContext { - graphical_functions: self.graphical_functions, - modules: self.module_decls, - arrays: vec![], - dimensions: self.dimensions, - subdim_relations: self.subdim_relations, - names: self.names, - static_views: self.static_views, - temp_offsets, - temp_total_size, - dim_lists: self.dim_lists, - }), compiled_initials, compiled_flows, compiled_stocks, + graphical_functions: self.graphical_functions, + module_decls: self.module_decls, + static_views: self.static_views, + arrays: vec![], + dimensions: self.dimensions, + subdim_relations: self.subdim_relations, + names: self.names, + temp_offsets, + temp_total_size, + dim_lists: self.dim_lists, // The flow-runlist invariant/dynamic partition is decided at module // assembly (the salsa `assemble_module` path, where the whole // root-model runlist is available), NOT here: `Compiler::compile` // runs both per single-variable fragment (split is trivially 0) and // on the test-only monolithic whole-model `Module`. So this is - // always 0; the production split is installed by `resolve_module` - // from the assembled `SymbolicCompiledModule` (GH #712). + // always 0; the production split is installed by `assemble_module` + // on the `SymbolicCompiledModule` it builds from the fragments + // (GH #712). flows_invariant_opcode_len: 0, }) } @@ -1677,27 +1786,57 @@ mod tests { use crate::ast::Loc; use std::collections::HashMap; - /// Build a minimal, dimension-free `Module` sufficient to drive `walk_expr` - /// on a hand-built `Expr`. The runlists are empty -- the test calls - /// `walk_expr` directly, so the only requirement is a well-formed `Module` - /// that `Compiler::new` can populate metadata from. - fn empty_module() -> Module { - Module { - ident: Ident::new("test"), - inputs: Default::default(), - n_slots: 1, - n_temps: 0, - temp_sizes: vec![], - runlist_initials: vec![], - runlist_initials_by_var: vec![], - runlist_flows: vec![], - runlist_stocks: vec![], - offsets: HashMap::new(), - runlist_order: vec![], - tables: HashMap::new(), - dimensions: vec![], - dimensions_ctx: Default::default(), - module_refs: HashMap::new(), + /// Owner for the borrows a minimal, dimension-free [`ModuleCtx`] needs. + /// The runlists are empty -- the tests call `walk_expr` directly, so the + /// only requirement is a well-formed context `Compiler::new` can populate + /// metadata from. + struct EmptyCtxOwner { + ident: Ident, + inputs: std::collections::BTreeSet>, + var_sizes: VarSizes, + tables: HashMap, Vec
>, + dimensions_ctx: crate::dimensions::DimensionsContext, + } + + impl EmptyCtxOwner { + fn new() -> Self { + EmptyCtxOwner { + ident: Ident::new("test"), + inputs: Default::default(), + var_sizes: HashMap::new(), + tables: HashMap::new(), + dimensions_ctx: Default::default(), + } + } + + /// The extent of a whole reference to `name`, as + /// `context::whole_variable_extents` records an ordinary variable. + fn with_var_size(mut self, name: &str, size: usize) -> Self { + self.var_sizes.insert(VarRef::base(Ident::new(name)), size); + self + } + + /// The extent of a SUB-MODEL variable living at slot `slot` of module + /// instance `instance`, as `whole_variable_extents` records one. + fn with_submodel_var_size(mut self, instance: &str, slot: usize, size: usize) -> Self { + self.var_sizes + .insert(VarRef::new(Ident::new(instance), slot), size); + self + } + + fn ctx(&self) -> ModuleCtx<'_> { + ModuleCtx { + ident: &self.ident, + inputs: &self.inputs, + temp_sizes: &[], + runlist_initials_by_var: &[], + runlist_flows: &[], + runlist_stocks: &[], + var_sizes: &self.var_sizes, + tables: &self.tables, + dimensions: &[], + dimensions_ctx: &self.dimensions_ctx, + } } } @@ -1711,15 +1850,15 @@ mod tests { /// (codegen.rs line 494 was a double-`unwrap` on this `Result`). #[test] fn previous_of_non_var_inside_subscript_index_is_err_not_panic() { - let module = empty_module(); - let mut compiler = Compiler::new(&module); + let owner = EmptyCtxOwner::new(); + let mut compiler = Compiler::new(owner.ctx()); // arr[ PREVIOUS(1, 0) ] -- the index is a PREVIOUS of a constant, which // is not a bare variable reference, so the PREVIOUS arm returns // NotSimulatable. Before the fix the enclosing Subscript arm panicked // by unwrapping that Err; after the fix it propagates via `?`. let expr = Expr::Subscript( - 0, + VarRef::base(Ident::new("arr")), vec![SubscriptIndex::Single(Expr::App( BuiltinFn::Previous( Box::new(Expr::Const(1.0, Loc::default())), @@ -1753,14 +1892,14 @@ mod tests { /// the fix it propagates the typed `NotSimulatable` via `?`. #[test] fn previous_of_non_var_inside_view_subscript_index_is_err_not_panic() { - let module = empty_module(); - let mut compiler = Compiler::new(&module); + let owner = EmptyCtxOwner::new(); + let mut compiler = Compiler::new(owner.ctx()); // arr[ PREVIOUS(1, 0) ] driven through the array-view path: the index // is a PREVIOUS of a constant (NotSimulatable). The `Subscript` arm of // `walk_expr_as_view` must return that typed Err, not panic. let expr = Expr::Subscript( - 0, + VarRef::base(Ident::new("arr")), vec![SubscriptIndex::Single(Expr::App( BuiltinFn::Previous( Box::new(Expr::Const(1.0, Loc::default())), @@ -1781,4 +1920,112 @@ mod tests { "expected NotSimulatable, got {err:?}" ); } + + /// `full_source_len` reports a VECTOR ELM MAP source's extent from the + /// [`VarSizes`] entry the reference itself keys, and falls back to the view + /// when the reference addresses no variable in whole. + /// + /// A collapsed element source inside an array-producing builtin keeps the + /// variable's base and carries the element in `view.offset` (that is why + /// `Context::lower_from_expr3` returns a `StaticSubscript` rather than a + /// `Var` there -- GH #578), so it recovers the FULL extent. A reference that + /// starts mid-array names one element of a bigger array and falls back. + #[test] + fn full_source_len_uses_the_whole_variable_only_at_its_base() { + let owner = EmptyCtxOwner::new().with_var_size("d", 6); + let compiler = Compiler::new(owner.ctx()); + + // `d[DimA, DimB]` collapsed to one element: base is `d`, the element + // rides in the view. The genuine-Vensim bound is d's FULL 6 elements. + let scalar_view = { + let mut v = ArrayView::contiguous(vec![]); + v.offset = 4; + v + }; + assert_eq!( + compiler.full_source_len(&Expr::StaticSubscript( + VarRef::base(Ident::new("d")), + scalar_view.clone(), + crate::ast::Loc::default(), + )), + 6 + ); + + // The same view, but the reference starts 4 slots into `d`. It names one + // element of a bigger array, whose extent is not the array's, so the + // view's extent is all this can honestly report. + assert_eq!( + compiler.full_source_len(&Expr::StaticSubscript( + VarRef::new(Ident::new("d"), 4), + scalar_view, + crate::ast::Loc::default(), + )), + 1 + ); + } + + /// A CROSS-MODULE source reports the SUB-MODEL variable's extent, not the + /// module instance's slot count. + /// + /// `m·x` lowers to `VarRef { name: m, element_offset: x's slot inside the + /// instance }`, so a name-keyed lookup answered with the size of the whole + /// instance -- and when `x` happened to sit at slot 0 of its sub-model the + /// reference also passed a `element_offset == 0` whole-variable test, so the + /// wrong answer was returned rather than declined. Reads past `x`'s end then + /// landed on the NEXT sub-model variable instead of yielding `:NA:`. The + /// end-to-end shape is + /// `array_tests::…::elm_map_cross_module_source_uses_the_submodel_variable_extent`; + /// this is the unit-level statement of the same rule. + #[test] + fn full_source_len_of_a_cross_module_source_is_the_submodel_variables_extent() { + // Instance `m` spans 8 slots: `avals[4]` at 0, another 4-slot variable + // at 4. Both are recorded; the instance itself is not. + let owner = EmptyCtxOwner::new() + .with_submodel_var_size("m", 0, 4) + .with_submodel_var_size("m", 4, 4); + let compiler = Compiler::new(owner.ctx()); + + // A COLLAPSED element source: the view carries no dimensions, so the + // table is the only thing that can report an extent and the answer is + // not confounded by a fallback that happens to coincide with it. + let collapsed = |slot: usize| { + let mut view = ArrayView::contiguous(vec![]); + view.offset = 1; + compiler.full_source_len(&Expr::StaticSubscript( + VarRef::new(Ident::new("m"), slot), + view, + Loc::default(), + )) + }; + + assert_eq!( + collapsed(0), + 4, + "the sub-model variable at the instance's base -- its extent, not m's 8 slots" + ); + assert_eq!( + collapsed(4), + 4, + "the second sub-model variable: a reference below the instance's \ + base still addresses a variable in whole" + ); + assert_eq!( + collapsed(2), + 1, + "mid-array: no whole-variable entry, so the collapsed view stands" + ); + + // The dynamic-subscript shape resolves through the same table; its + // lowered bounds agree with it for an in-model source, and only the + // table is right for a cross-module one. + assert_eq!( + compiler.full_source_len(&Expr::Subscript( + VarRef::new(Ident::new("m"), 4), + vec![SubscriptIndex::Single(Expr::Const(0.0, Loc::default()))], + vec![4], + Loc::default(), + )), + 4 + ); + } } diff --git a/src/simlin-engine/src/compiler/context.rs b/src/simlin-engine/src/compiler/context.rs index 96695b503..865d237de 100644 --- a/src/simlin-engine/src/compiler/context.rs +++ b/src/simlin-engine/src/compiler/context.rs @@ -17,7 +17,7 @@ use crate::variable::Variable; use crate::{Error, sim_err}; use super::dimensions::{UnaryOp, find_dimension_reordering, match_dimensions_with_mapping}; -use super::expr::{BuiltinFn, Expr, SubscriptIndex}; +use super::expr::{BuiltinFn, Expr, SubscriptIndex, VarRef}; use super::subscript::{ IndexOp, Subscript3Config, ViewBuildConfig, ViewBuildResult, build_view_from_ops, normalize_subscripts3, @@ -26,11 +26,121 @@ use super::subscript::{ #[cfg_attr(feature = "debug-derive", derive(Debug))] #[derive(Clone, Copy)] pub(crate) struct VariableMetadata<'a> { - pub(crate) offset: usize, + /// The variable's slot offset within **its own model's** layout. + /// + /// `None` means that layout has not been assigned yet, which is the normal + /// case for the model being compiled: the per-variable fragment path defers + /// its model's layout to assembly, so a fragment cannot know (and must not + /// depend on) where its own variables land. + /// + /// The only reader is `Context::submodel_offset_within`, and it only ever + /// walks *sub*-model entries: a cross-module reference `m·x` lowers to + /// `VarRef { name: m, element_offset: }`, + /// and the sub-model's layout IS already fixed (`db::compute_layout`) -- + /// the parent relocates the whole block via `m`'s own name. A `None` there + /// is a loud `DoesNotExist`, never a silent slot 0. + pub(crate) offset: Option, pub(crate) size: usize, pub(crate) var: &'a Variable, } +/// The symbol table lowering builds: `model -> variable -> metadata`. It holds +/// the model being compiled plus every sub-model reachable from it, which is +/// what makes a cross-module name resolvable. +pub(crate) type MetadataByModel<'a> = + HashMap, HashMap, VariableMetadata<'a>>>; + +/// Project [`super::VarSizes`] out of the symbol table: the extent of every +/// variable a reference in `model` can address **in whole**. +/// +/// This is the single statement of "which reference addresses which variable's +/// full storage", and both consumers of that question read it -- lowering's +/// GH #578 scalar-source ELM MAP fold (`Context::full_var_len_for_base`) and +/// emission's `codegen::full_source_len`. They used to answer it separately, +/// each from its own view of the symbol table, and both got a cross-module +/// source wrong in a different direction. +/// +/// An ordinary variable contributes one entry, at its base. A MODULE INSTANCE +/// contributes none of its own -- its slot count is the whole sub-model block, +/// which is the extent of nothing a reference can name -- and instead +/// contributes one entry per sub-model variable at that variable's slot within +/// the instance, recursively through nested instances so every entry names a +/// leaf variable. A reference landing mid-array is absent from the result, +/// which is the same answer it has always had: the extent of one element of a +/// bigger array is not the array's extent, so the caller falls back to what the +/// lowered view says. +/// +/// A sub-model that is not in `metadata` contributes nothing. That is the +/// loud-safe direction: the reference falls back to its view's extent, exactly +/// as an unresolvable one always did, rather than borrowing a neighbour's size. +pub(crate) fn whole_variable_extents( + metadata: &MetadataByModel<'_>, + model: &Ident, +) -> super::VarSizes { + let mut extents = super::VarSizes::new(); + let Some(vars) = metadata.get(model) else { + return extents; + }; + for (name, md) in vars { + if let Variable::Module { + model_name: sub_model, + .. + } = md.var + { + let mut path = vec![model.clone()]; + collect_instance_extents(metadata, sub_model, name, 0, &mut path, &mut extents); + } else { + extents.insert(VarRef::base(name.clone()), md.size); + } + } + extents +} + +/// One module instance's contribution to [`whole_variable_extents`]. +/// +/// `base` is the slot the instance's copy of `sub_model` starts at, measured +/// from the instance's own base, so a nested instance's variables are reached +/// by accumulating the offsets on the way down -- the same arithmetic +/// `Context::submodel_offset_within` performs when it lowers `m·n·x`. +/// +/// `path` is the chain of models being walked, and a model already on it is +/// declined rather than descended into. `db::project_module_graph` rejects a +/// cyclic project before any of this runs, so the guard exists to bound the +/// walk, not to gate the project. +fn collect_instance_extents( + metadata: &MetadataByModel<'_>, + sub_model: &Ident, + instance: &Ident, + base: usize, + path: &mut Vec>, + extents: &mut super::VarSizes, +) { + if path.iter().any(|seen| seen == sub_model) { + return; + } + let Some(vars) = metadata.get(sub_model) else { + return; + }; + path.push(sub_model.clone()); + for md in vars.values() { + // A sub-model's layout is always already assigned (`db::compute_layout` + // runs before any fragment of the parent compiles); an entry without one + // is the model being compiled, which cannot also be its own sub-model. + let Some(offset) = md.offset else { + continue; + }; + if let Variable::Module { + model_name: nested, .. + } = md.var + { + collect_instance_extents(metadata, nested, instance, base + offset, path, extents); + } else { + extents.insert(VarRef::new(instance.clone(), base + offset), md.size); + } + } + path.pop(); +} + #[cfg_attr(feature = "debug-derive", derive(Debug))] #[derive(Clone)] pub(crate) struct Context<'a> { @@ -57,8 +167,12 @@ pub(crate) struct ContextCore<'a> { #[allow(dead_code)] pub(crate) dimensions_ctx: &'a DimensionsContext, pub(crate) model_name: &'a Ident, - pub(crate) metadata: - &'a HashMap, HashMap, VariableMetadata<'a>>>, + pub(crate) metadata: &'a MetadataByModel<'a>, + /// The extents [`whole_variable_extents`] projects out of `metadata`, so + /// lowering and codegen answer "how big is the variable this reference + /// addresses?" from one table rather than from two readings of the symbol + /// table. Derived, never authored: build it with `whole_variable_extents`. + pub(crate) var_sizes: &'a super::VarSizes, pub(crate) module_models: &'a HashMap, HashMap, Ident>>, pub(crate) inputs: &'a BTreeSet>, @@ -121,13 +235,18 @@ impl Context<'_> { ) } - pub(super) fn get_offset(&self, ident: &Ident) -> Result { - self.get_submodel_offset(self.model_name, ident, false) + /// The reference `ident` denotes in the model being compiled, resolving an + /// arrayed variable's *implicit* subscripts (the active A2A element) to an + /// element offset. + pub(super) fn get_ref(&self, ident: &Ident) -> Result { + self.var_ref(ident, false) } - /// get_base_offset ignores arrays and should only be used from Var::new and Expr::Subscript - pub(super) fn get_base_offset(&self, ident: &Ident) -> Result { - self.get_submodel_offset(self.model_name, ident, true) + /// `get_base_ref` ignores arrays -- it yields the variable's first slot -- + /// and should only be used from `Var::new` and `Expr::Subscript`, where the + /// element is selected by an explicit view or index expression instead. + pub(super) fn get_base_ref(&self, ident: &Ident) -> Result { + self.var_ref(ident, true) } pub(super) fn get_metadata(&self, ident: &Ident) -> Result<&VariableMetadata<'_>> { @@ -402,7 +521,83 @@ impl Context<'_> { } } - fn get_submodel_offset( + /// Resolve `ident` to a layout-independent [`VarRef`] in the model being + /// compiled. + /// + /// A plain name resolves to itself, so the reference is just + /// `(ident, implicit element offset)` -- the model's own layout is never + /// consulted. A cross-module name `m·x` resolves to the *module* variable + /// `m`, because the enclosing model's layout has one entry spanning the + /// whole sub-model instance and none for `m·x`; the element offset is `x`'s + /// position inside that block, which the sub-model's already-fixed layout + /// supplies (see [`Self::submodel_offset_within`]). + fn var_ref(&self, ident: &Ident, ignore_arrays: bool) -> Result { + let metadata = self + .metadata + .get(self.model_name) + .ok_or_else(|| Error::new(ErrorKind::Simulation, ErrorCode::DoesNotExist, None))?; + let ident_str = ident.as_str(); + if let Some(pos) = ident_str.find('\u{00B7}') { + let submodel_module_name = &ident_str[..pos]; + let module_key = Ident::::from_str_unchecked(submodel_module_name); + // The module variable may have been deleted while dependent + // equations still reference it (e.g. `module.output`). + let model_modules = self + .module_models + .get(self.model_name) + .ok_or_else(|| Error::new(ErrorKind::Simulation, ErrorCode::DoesNotExist, None))?; + let submodel_name = model_modules + .get(&module_key) + .ok_or_else(|| Error::new(ErrorKind::Simulation, ErrorCode::DoesNotExist, None))?; + // The module variable must still be in scope; the sub-model's + // internal offset is what indexes into its block. + if !metadata.contains_key(&module_key) { + return Err(Error::new( + ErrorKind::Simulation, + ErrorCode::DoesNotExist, + None, + )); + } + let submodel_var = &ident_str[pos + '\u{00B7}'.len_utf8()..]; + let element_offset = self.submodel_offset_within( + submodel_name, + &Ident::::from_str_unchecked(submodel_var), + ignore_arrays, + )?; + Ok(VarRef::new(module_key, element_offset)) + } else { + // The lookup is load-bearing even when `ignore_arrays` discards the + // metadata: a reference to a variable that is not in scope must be + // `DoesNotExist`, and naming it is no longer enough to prove it + // exists. + let var_meta = metadata + .get(ident) + .ok_or_else(|| Error::new(ErrorKind::Simulation, ErrorCode::DoesNotExist, None))?; + if ignore_arrays { + return Ok(VarRef::base(ident.clone())); + } + match var_meta.var.get_dimensions() { + Some(dims) => { + let off = self.get_implicit_subscript_off(dims, ident.as_str())?; + Ok(VarRef::new(ident.clone(), off)) + } + None => Ok(VarRef::base(ident.clone())), + } + } + } + + /// The absolute slot offset of `ident` within the *sub*-model `model`'s own + /// layout. + /// + /// This is the one place a concrete offset is still computed during + /// lowering, and it is sound because a sub-model's layout is already fixed + /// (`db::compute_layout`) before any fragment of the parent is compiled: the + /// parent relocates the whole block through the module variable's name. A + /// metadata entry with no offset (the model being compiled, whose layout is + /// assigned at assembly) is a loud `DoesNotExist` rather than a silent 0 -- + /// reaching one here would mean a model instantiated itself, which the + /// module-graph cycle gate rejects first. + fn submodel_offset_within( &self, model: &Ident, ident: &Ident, @@ -416,8 +611,6 @@ impl Context<'_> { if let Some(pos) = ident_str.find('\u{00B7}') { let submodel_module_name = &ident_str[..pos]; let module_key = Ident::::from_str_unchecked(submodel_module_name); - // The module variable may have been deleted while dependent - // equations still reference it (e.g. `module.output`). let model_modules = self .module_models .get(model) @@ -428,29 +621,31 @@ impl Context<'_> { let submodel_var = &ident_str[pos + '\u{00B7}'.len_utf8()..]; let submodel_off = metadata .get(&module_key) - .ok_or_else(|| Error::new(ErrorKind::Simulation, ErrorCode::DoesNotExist, None))? - .offset; + .and_then(|m| m.offset) + .ok_or_else(|| Error::new(ErrorKind::Simulation, ErrorCode::DoesNotExist, None))?; Ok(submodel_off - + self.get_submodel_offset( + + self.submodel_offset_within( submodel_name, &Ident::::from_str_unchecked(submodel_var), ignore_arrays, )?) - } else if !ignore_arrays { + } else { let var_meta = metadata .get(ident) .ok_or_else(|| Error::new(ErrorKind::Simulation, ErrorCode::DoesNotExist, None))?; - if let Some(dims) = var_meta.var.get_dimensions() { - let off = self.get_implicit_subscript_off(dims, ident.as_str())?; - Ok(var_meta.offset + off) - } else { - Ok(var_meta.offset) + let base = var_meta + .offset + .ok_or_else(|| Error::new(ErrorKind::Simulation, ErrorCode::DoesNotExist, None))?; + if ignore_arrays { + return Ok(base); + } + match var_meta.var.get_dimensions() { + Some(dims) => { + let off = self.get_implicit_subscript_off(dims, ident.as_str())?; + Ok(base + off) + } + None => Ok(base), } - } else { - metadata - .get(ident) - .map(|m| m.offset) - .ok_or_else(|| Error::new(ErrorKind::Simulation, ErrorCode::DoesNotExist, None)) } } @@ -848,10 +1043,7 @@ impl Context<'_> { let loads: Result> = flows .iter() - .map(|flow| { - self.get_offset(flow) - .map(|off| Expr::Var(off, Loc::default())) - }) + .map(|flow| self.get_ref(flow).map(|var| Expr::Var(var, Loc::default()))) .collect(); let mut loads = loads?.into_iter(); @@ -874,10 +1066,10 @@ impl Context<'_> { // Check if this is a simple variable or static subscript that we can reorder directly match &expr { - Expr::Var(off, _) => { + Expr::Var(var, _) => { // This is a bare array variable - create a StaticSubscript with reordered view // First, get the variable metadata to get dimensions - if let Ok(metadata) = self.get_variable_metadata_by_offset(*off) + if let Some(metadata) = self.whole_var_metadata(var) && let Some(dims) = metadata.var.get_dimensions() { let orig_dims: Vec = dims.iter().map(|d| d.len()).collect(); @@ -887,16 +1079,16 @@ impl Context<'_> { // Create a contiguous view with names and apply reordering let view = ArrayView::contiguous_with_names(orig_dims, orig_dim_names); return Ok(Expr::StaticSubscript( - *off, + var.clone(), view.reorder_dimensions(&reordering), loc, )); } } - Expr::StaticSubscript(off, view, _) => { + Expr::StaticSubscript(var, view, _) => { // Apply reordering to existing view return Ok(Expr::StaticSubscript( - *off, + var.clone(), view.reorder_dimensions(&reordering), loc, )); @@ -916,43 +1108,45 @@ impl Context<'_> { } } - /// Helper to get variable metadata by offset - fn get_variable_metadata_by_offset(&self, offset: usize) -> Result<&VariableMetadata<'_>> { - let metadata = self.metadata.get(self.model_name).ok_or_else(|| { - use crate::common::{Error, ErrorCode, ErrorKind}; - Error { - kind: ErrorKind::Simulation, - code: ErrorCode::BadModelName, - details: Some("Model not found".to_string()), - } - })?; - - // Find the variable with the matching offset - for (_, var_metadata) in metadata.iter() { - if var_metadata.offset == offset { - return Ok(var_metadata); - } + /// Metadata for a reference that addresses a variable's *whole* storage, in + /// the model being compiled. + /// + /// `None` for a reference into the middle of a variable, which is the exact + /// answer the previous offset-scan gave ("no variable's storage begins at + /// this offset"): a mid-variable reference names an element, not the array, + /// so neither the array's dimensions nor its extent apply to it. + /// + /// The sole reader is [`Self::apply_dimension_reordering`], which wants a + /// bare array reference's declared DIMENSIONS. Note that a CROSS-MODULE + /// reference gets the module instance's entry here (a `Variable::Module`, + /// which declares no dimensions), so the reorder falls through to its + /// generic path rather than specializing -- correct, if unspecialized. Do + /// not reach for this to answer a question about a reference's EXTENT: + /// [`super::VarSizes`] is where that lives, precisely because it resolves + /// the cross-module case. + fn whole_var_metadata(&self, var: &VarRef) -> Option<&VariableMetadata<'_>> { + if !var.is_whole_var() { + return None; } - - sim_err!(DoesNotExist, "Variable not found by offset".to_string()) + self.metadata.get(self.model_name)?.get(&var.name) } - /// Full element count of the variable whose storage *begins* at `base_off` - /// (the product of its declared dimensions; 1 for a scalar). Returns `None` - /// if no variable owns that exact base offset. This is the compile-time - /// twin of `codegen::full_source_len`, used by the GH #578 scalar-source - /// VECTOR ELM MAP fold to bound the per-element static read. - pub(super) fn full_var_len_for_base(&self, base_off: usize) -> Option { - let md = self.get_variable_metadata_by_offset(base_off).ok()?; - Some( - md.var - .get_dimensions() - .map(|dims| dims.iter().map(|d| d.len()).product::().max(1)) - .unwrap_or(1), - ) + /// Full element count of the variable `base` addresses in whole. `None` for + /// a reference that does not start at a variable's base. Used by the GH #578 + /// scalar-source VECTOR ELM MAP fold to bound the per-element static read. + /// + /// This reads the SAME table `codegen::full_source_len` reads, which is the + /// point: the fold and the opcode must agree about where a source's storage + /// ends, and they are on opposite sides of lowering. Answering from the + /// variable's own declared dimensions -- what this did before -- silently + /// reported 1 for a cross-module source, whose metadata entry here is the + /// dimensionless module instance rather than the sub-model variable the + /// reference actually names. + pub(super) fn full_var_len_for_base(&self, base: &VarRef) -> Option { + self.var_sizes.get(base).copied() } - pub(super) fn build_stock_update_expr(&self, stock_off: usize, var: &Variable) -> Result { + pub(super) fn build_stock_update_expr(&self, stock: &VarRef, var: &Variable) -> Result { if let Variable::Stock { inflows, outflows, .. } = var @@ -978,7 +1172,7 @@ impl Context<'_> { Ok(Expr::Op2( BinaryOp::Add, - Box::new(Expr::Var(stock_off, Loc::default())), + Box::new(Expr::Var(stock.clone(), Loc::default())), Box::new(dt_update), Loc::default(), )) @@ -993,9 +1187,39 @@ impl Context<'_> { // Implement Expr3LowerContext for Context to enable Expr2 -> Expr3 conversion impl Expr3LowerContext for Context<'_> { + /// Resolved through [`Self::get_metadata`], so a CROSS-MODULE name + /// (`m·arr`) reports the sub-model variable's dimensions rather than + /// nothing. A direct `metadata[model][ident]` lookup -- what this did + /// before -- has no entry for the dotted name, and the `None` it returned + /// made `Expr3::from_expr2` treat every cross-module array as a scalar, so + /// a WILDCARD subscript on one (`SUM(m.arr[*])`) was rejected as + /// `CantSubscriptScalar`. `ast::ArrayContext::get_dimensions`, the + /// Expr2-stage twin of this trait method, has always followed the module + /// variable into the sub-model; this is the same rule, one stage later. + /// + /// The wildcard rejection was the whole of the user-visible consequence. + /// A BARE cross-module array reference was unaffected: it skipped the + /// implicit-subscript expansion here, but `lower_from_expr3`'s `Var` arm + /// resolves it one layer down through `var_ref` -> + /// `submodel_offset_within(.., ignore_arrays: false)`, which reads the + /// sub-model's own metadata and gets the dimensions this method did not. + /// Bare references, whole-array copies and transposes all produced correct + /// results before the fix. fn get_dimensions(&self, ident: &str) -> Option> { - let metadata = self.metadata.get(self.model_name)?; - let var_metadata = metadata.get(&*canonicalize(ident))?; + let canonical = canonicalize(ident); + let vars = self.metadata.get(self.model_name)?; + // A plain name is its own key, so the direct lookup answers it without + // interning an `Ident`; only a name the model does not hold -- every + // dotted cross-module one among them -- pays for the module-following + // resolution. The two agree on a plain name by construction: + // `get_submodel_metadata` reduces to this same lookup when there is no + // module separator to follow. + let var_metadata = match vars.get(&*canonical) { + Some(md) => md, + None => self + .get_metadata(&Ident::from_unchecked(canonical.into_owned())) + .ok()?, + }; var_metadata.var.get_dimensions().map(|dims| dims.to_vec()) } @@ -1076,8 +1300,8 @@ impl Context<'_> { match expr { // Handle Expr3-specific variants directly Expr3::StaticSubscript(id, view, _, loc) => { - let off = self.get_base_offset(id)?; - Ok(Expr::StaticSubscript(off, view.clone(), *loc)) + let base = self.get_base_ref(id)?; + Ok(Expr::StaticSubscript(base, view.clone(), *loc)) } Expr3::TempArray(id, view, loc) => Ok(Expr::TempArray(*id, view.clone(), *loc)), @@ -1180,8 +1404,8 @@ impl Context<'_> { } // Check if it's a regular variable - match self.get_offset(id) { - Ok(off) => Ok(Expr::Var(off, *loc)), + match self.get_ref(id) { + Ok(var) => Ok(Expr::Var(var, *loc)), Err(err) => { // If get_offset fails because it's an array without implicit subscripts, // try to create a full array view @@ -1190,7 +1414,7 @@ impl Context<'_> { && let Some(source_dims) = metadata.var.get_dimensions() { // This is an array variable - check if we need dimension reordering - let off = self.get_base_offset(id)?; + let base = self.get_base_ref(id)?; // Check if we're in an A2A context and need to reorder dimensions if let Some(target_dims) = &self.active_dimension { @@ -1246,7 +1470,7 @@ impl Context<'_> { dim_names: target_dim_names.clone(), }; - return Ok(Expr::StaticSubscript(off, view, *loc)); + return Ok(Expr::StaticSubscript(base, view, *loc)); } } } @@ -1257,7 +1481,7 @@ impl Context<'_> { let dim_names: Vec = source_dims.iter().map(|d| d.name().to_string()).collect(); let view = ArrayView::contiguous_with_names(orig_dims, dim_names); - return Ok(Expr::StaticSubscript(off, view, *loc)); + return Ok(Expr::StaticSubscript(base, view, *loc)); } Err(err) } @@ -1266,7 +1490,7 @@ impl Context<'_> { Expr3::Subscript(id, indices, _bounds, loc) => { // Handle subscript directly without converting to Expr2 - let off = self.get_base_offset(id)?; + let base = self.get_base_ref(id)?; let metadata = self.get_metadata(id)?; let dims = metadata.var.get_dimensions().ok_or_else(|| { Error::new( @@ -1399,7 +1623,7 @@ impl Context<'_> { &orig_strides, &view_config, )?; - return Ok(Expr::StaticSubscript(off, preserved_result.view, *loc)); + return Ok(Expr::StaticSubscript(base, preserved_result.view, *loc)); } else { if view.dims.is_empty() { // Inside an array-producing builtin a fully- @@ -1420,9 +1644,9 @@ impl Context<'_> { if self.promote_active_dim_ref && operations.iter().any(|op| matches!(op, IndexOp::Single(_))) { - return Ok(Expr::StaticSubscript(off, view, *loc)); + return Ok(Expr::StaticSubscript(base, view, *loc)); } - return Ok(Expr::Var(off + view.offset, *loc)); + return Ok(Expr::Var(base.offset_by(view.offset), *loc)); } // For broadcasting: source array may have fewer dimensions than output. @@ -1805,16 +2029,16 @@ impl Context<'_> { result_index += rel_offset * (*stride as usize); } - return Ok(Expr::Var(off + view.offset + result_index, *loc)); + return Ok(Expr::Var(base.offset_by(view.offset + result_index), *loc)); } if !has_dim_positions { - return Ok(Expr::StaticSubscript(off, view, *loc)); + return Ok(Expr::StaticSubscript(base, view, *loc)); } // has_dim_positions is true - fall through to dynamic handling } else { // Not in A2A context - return StaticSubscript for the full view - return Ok(Expr::StaticSubscript(off, view, *loc)); + return Ok(Expr::StaticSubscript(base, view, *loc)); } } @@ -1889,7 +2113,7 @@ impl Context<'_> { // Load the element via dynamic single subscript let load_elem = Expr::Subscript( - off, + base.clone(), vec![SubscriptIndex::Single(computed_index)], vec![dims[0].len()], *loc, @@ -1911,7 +2135,7 @@ impl Context<'_> { .enumerate() .map(|(i, arg)| self.lower_index_expr3(arg, id, i, dims, &orig_dims, *loc)) .collect(); - Ok(Expr::Subscript(off, args?, orig_dims, *loc)) + Ok(Expr::Subscript(base, args?, orig_dims, *loc)) } Expr3::App(builtin, _bounds, loc) => { @@ -1939,7 +2163,7 @@ impl Context<'_> { } else { // Not in A2A context - create a wildcard subscript to get the full array // then apply transpose - let off = self.get_base_offset(id)?; + let base = self.get_base_ref(id)?; let orig_dims: Vec = dims.iter().map(|d| d.len()).collect(); let orig_dim_names: Vec = @@ -1957,7 +2181,7 @@ impl Context<'_> { }; return Ok(Expr::StaticSubscript( - off, + base, view.transpose(), *var_loc, )); @@ -1974,8 +2198,8 @@ impl Context<'_> { let lowered = self.lower_from_expr3(inner)?; // Transpose reverses the dimensions of an array match lowered { - Expr::StaticSubscript(off, view, expr_loc) => { - Ok(Expr::StaticSubscript(off, view.transpose(), expr_loc)) + Expr::StaticSubscript(base, view, expr_loc) => { + Ok(Expr::StaticSubscript(base, view.transpose(), expr_loc)) } _ => { // For other expressions, wrap in a transpose operation @@ -2337,7 +2561,7 @@ impl Context<'_> { // simultaneous allocation. Any explicit subscripts that restricted // dimensions are intentionally overridden: the allocator requires // the full array regardless of the calling element's context. - let base = self.get_base_offset(var_ident)?; + let base = self.get_base_ref(var_ident)?; let dim_sizes: Vec = full_dims.iter().map(|d| d.len()).collect(); let dim_names: Vec = full_dims.iter().map(|d| d.name().to_string()).collect(); let view = ArrayView::contiguous_with_names(dim_sizes, dim_names); @@ -2675,7 +2899,7 @@ fn test_lower() { metadata.insert( Ident::new("true_input"), VariableMetadata { - offset: 7, + offset: Some(7), size: 1, var: &true_var, }, @@ -2683,7 +2907,7 @@ fn test_lower() { metadata.insert( Ident::new("false_input"), VariableMetadata { - offset: 8, + offset: Some(8), size: 1, var: &false_var, }, @@ -2693,12 +2917,14 @@ fn test_lower() { let test_ident = Ident::new("test"); metadata2.insert(main_ident.clone(), metadata); let dims_ctx = DimensionsContext::default(); + let var_sizes = whole_variable_extents(&metadata2, &main_ident); let context = Context::new( ContextCore { dimensions: &[], dimensions_ctx: &dims_ctx, model_name: &main_ident, metadata: &metadata2, + var_sizes: &var_sizes, module_models: &module_models, inputs, }, @@ -2708,8 +2934,14 @@ fn test_lower() { let expected = Expr::If( Box::new(Expr::Op2( BinaryOp::And, - Box::new(Expr::Var(7, Loc::default())), - Box::new(Expr::Var(8, Loc::default())), + Box::new(Expr::Var( + VarRef::base(Ident::new("true_input")), + Loc::default(), + )), + Box::new(Expr::Var( + VarRef::base(Ident::new("false_input")), + Loc::default(), + )), Loc::default(), )), Box::new(Expr::Const(1.0, Loc::default())), @@ -2774,7 +3006,7 @@ fn test_lower() { metadata.insert( Ident::new("true_input"), VariableMetadata { - offset: 7, + offset: Some(7), size: 1, var: &true_var, }, @@ -2782,7 +3014,7 @@ fn test_lower() { metadata.insert( Ident::new("false_input"), VariableMetadata { - offset: 8, + offset: Some(8), size: 1, var: &false_var, }, @@ -2792,12 +3024,14 @@ fn test_lower() { let test_ident = Ident::new("test"); metadata2.insert(main_ident.clone(), metadata); let dims_ctx = DimensionsContext::default(); + let var_sizes = whole_variable_extents(&metadata2, &main_ident); let context = Context::new( ContextCore { dimensions: &[], dimensions_ctx: &dims_ctx, model_name: &main_ident, metadata: &metadata2, + var_sizes: &var_sizes, module_models: &module_models, inputs, }, @@ -2807,8 +3041,14 @@ fn test_lower() { let expected = Expr::If( Box::new(Expr::Op2( BinaryOp::Or, - Box::new(Expr::Var(7, Loc::default())), - Box::new(Expr::Var(8, Loc::default())), + Box::new(Expr::Var( + VarRef::base(Ident::new("true_input")), + Loc::default(), + )), + Box::new(Expr::Var( + VarRef::base(Ident::new("false_input")), + Loc::default(), + )), Loc::default(), )), Box::new(Expr::Const(1.0, Loc::default())), @@ -2840,12 +3080,14 @@ fn test_with_active_subscripts_reuses_dimension_storage() { HashMap::new(); let inputs = BTreeSet::new(); + let var_sizes = whole_variable_extents(&metadata, &model_name); let base = Context::new( ContextCore { dimensions: &dims, dimensions_ctx: &dims_ctx, model_name: &model_name, metadata: &metadata, + var_sizes: &var_sizes, module_models: &module_models, inputs: &inputs, }, @@ -2895,12 +3137,14 @@ fn test_get_implicit_subscript_off_translates_through_mapping_parent() { let model_name = Ident::new("main"); let ident = Ident::new("test_var"); + let var_sizes = whole_variable_extents(&metadata, &model_name); let base = Context::new( ContextCore { dimensions: &all_dims, dimensions_ctx: &dims_ctx, model_name: &model_name, metadata: &metadata, + var_sizes: &var_sizes, module_models: &module_models, inputs: &inputs, }, @@ -2963,7 +3207,7 @@ fn test_positional_fallback_ignores_unrelated_mapping() { model_metadata.insert( Ident::new("source_var"), VariableMetadata { - offset: 10, + offset: Some(10), size: 2, var: &source_var, }, @@ -2978,12 +3222,14 @@ fn test_positional_fallback_ignores_unrelated_mapping() { HashMap::new(); let inputs = BTreeSet::new(); let ident = Ident::new("test_var"); + let var_sizes = whole_variable_extents(&metadata, &model_name); let base = Context::new( ContextCore { dimensions: &all_dims, dimensions_ctx: &dims_ctx, model_name: &model_name, metadata: &metadata, + var_sizes: &var_sizes, module_models: &module_models, inputs: &inputs, }, @@ -3008,7 +3254,7 @@ fn test_positional_fallback_ignores_unrelated_mapping() { .expect("positional fallback should resolve source[*] in target context"); assert_eq!( lowered, - Expr::Var(11, Loc::default()), + Expr::Var(VarRef::new(Ident::new("source_var"), 1), Loc::default()), "target element t2 should select the second source element" ); } diff --git a/src/simlin-engine/src/compiler/dimensions.rs b/src/simlin-engine/src/compiler/dimensions.rs index 6d1ce8d29..23c7cbeb3 100644 --- a/src/simlin-engine/src/compiler/dimensions.rs +++ b/src/simlin-engine/src/compiler/dimensions.rs @@ -685,7 +685,7 @@ mod tests { .expect("Module creation should succeed"); // Create a Compiler directly to test lazy subdim_relation population - let mut compiler = super::super::codegen::Compiler::new(&module); + let mut compiler = super::super::codegen::Compiler::new(module.as_ctx()); // Initially, subdim_relations should be empty (lazy population) assert!( diff --git a/src/simlin-engine/src/compiler/expr.rs b/src/simlin-engine/src/compiler/expr.rs index 072bff4d6..96cb7aaa8 100644 --- a/src/simlin-engine/src/compiler/expr.rs +++ b/src/simlin-engine/src/compiler/expr.rs @@ -10,6 +10,94 @@ use crate::sim_err; use super::dimensions::UnaryOp; +/// A reference to one slot of a model variable, **by name**. +/// +/// This is the compiler's only variable-address representation. Lowering +/// produces it, codegen emits it into `symbolic::SymbolicOpcode` / +/// `SymbolicStaticView` / `SymbolicModuleDecl` operands unchanged, and +/// `symbolic::resolve_module` turns it into a concrete slot exactly once, at +/// assembly, against the model's final `VariableLayout`. Nothing in between +/// needs to know where the variable lives, which is what lets one salsa cache +/// entry per variable serve both the diagnostic pass and assembly and survive +/// unrelated variables being added, removed, or renamed. +/// +/// `name` is the canonical name of the variable that owns the slot **in the +/// model being compiled**. A cross-module reference (`m·x`) resolves to the +/// *module* variable `m` with `element_offset` indexing into that module +/// instance's slot block, because the enclosing model's layout has one entry +/// for `m` spanning the whole sub-model and none for `m·x`. That is the one +/// place a sub-model's own (already fixed) layout is consulted during +/// lowering; see `Context::submodel_offset_within`. +#[cfg_attr(feature = "debug-derive", derive(Debug))] +#[derive(Clone, PartialEq, Eq, Hash)] +pub struct VarRef { + /// Canonical name of the variable that owns the referenced slot. + pub name: Ident, + /// Offset of the slot within that variable's storage: 0 for a scalar, + /// `0..size` for an array element or a slot inside a module instance. + pub element_offset: usize, +} + +// A `VarRef` rides on `PerVarBytecodes`, whose `Debug` is unconditional (the +// fragment characterization goldens and the salsa artifacts print it), so this +// cannot hang off the optional `debug-derive` feature the way `Expr` does. +#[cfg(not(feature = "debug-derive"))] +impl std::fmt::Debug for VarRef { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "{self}") + } +} + +/// `name@element` -- the spelling the fragment characterization goldens and the +/// compiler's diagnostics both use for a reference. +impl std::fmt::Display for VarRef { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "{}@{}", self.name, self.element_offset) + } +} + +impl VarRef { + /// A reference to the variable's first slot. + pub(crate) fn base(name: Ident) -> Self { + VarRef { + name, + element_offset: 0, + } + } + + pub(crate) fn new(name: Ident, element_offset: usize) -> Self { + VarRef { + name, + element_offset, + } + } + + /// The reference `delta` slots further into the same variable. + pub(crate) fn offset_by(&self, delta: usize) -> Self { + VarRef { + name: self.name.clone(), + element_offset: self.element_offset + delta, + } + } + + /// Whether this reference is to the variable's *whole* storage, i.e. its + /// first slot. + /// + /// Two codegen/lowering lookups are only meaningful for a whole-variable + /// reference: the dimensions of a transposed bare array, and the identity of + /// an arrayed lookup table. Both used to ask "does any variable's storage + /// *begin* at this offset?"; asking whether the reference sits at its + /// owner's base is the same question, and answers it without a reverse scan. + /// + /// A VECTOR ELM MAP source's extent is deliberately NOT one of them. That + /// question is keyed on the whole reference (`super::VarSizes`), because a + /// cross-module reference sits at its module INSTANCE's base while naming a + /// sub-model variable, and this predicate cannot tell those apart. + pub(crate) fn is_whole_var(&self) -> bool { + self.element_offset == 0 + } +} + #[cfg_attr(feature = "debug-derive", derive(Debug))] #[derive(Clone, PartialEq)] pub struct Table { @@ -61,12 +149,12 @@ impl SubscriptIndex { #[allow(dead_code)] pub enum Expr { Const(f64, Loc), - Var(usize, Loc), // offset + Var(VarRef, Loc), /// Dynamic subscript with possible range indices - /// (offset, subscript indices, dimension sizes, location) - Subscript(usize, Vec, Vec, Loc), - StaticSubscript(usize, ArrayView, Loc), // offset, precomputed view, location - TempArray(u32, ArrayView, Loc), // temp id, view into temp array, location + /// (base reference, subscript indices, dimension sizes, location) + Subscript(VarRef, Vec, Vec, Loc), + StaticSubscript(VarRef, ArrayView, Loc), // base reference, precomputed view, location + TempArray(u32, ArrayView, Loc), // temp id, view into temp array, location TempArrayElement(u32, ArrayView, usize, Loc), // temp id, view, element index, location Dt(Loc), App(BuiltinFn, Loc), @@ -82,8 +170,8 @@ pub enum Expr { Op2(BinaryOp, Box, Box, Loc), Op1(UnaryOp, Box, Loc), If(Box, Box, Box, Loc), - AssignCurr(usize, Box), - AssignNext(usize, Box), + AssignCurr(VarRef, Box), + AssignNext(VarRef, Box), AssignTemp(u32, Box, ArrayView), // temp id, expression to evaluate, view info } diff --git a/src/simlin-engine/src/compiler/fold.rs b/src/simlin-engine/src/compiler/fold.rs index 456a3479b..50ceafbfe 100644 --- a/src/simlin-engine/src/compiler/fold.rs +++ b/src/simlin-engine/src/compiler/fold.rs @@ -159,8 +159,15 @@ mod tests { Expr::Op2(op, Box::new(l), Box::new(r), Loc::default()) } - fn var(off: usize) -> Expr { - Expr::Var(off, Loc::default()) + fn var(n: usize) -> Expr { + Expr::Var(vref(n), Loc::default()) + } + + /// A distinct variable reference per test slot: the folder never inspects + /// a reference beyond copying it, so the tests only need distinguishable + /// operands. + fn vref(n: usize) -> crate::compiler::VarRef { + crate::compiler::VarRef::base(crate::common::Ident::new(&format!("v{n}"))) } fn assert_folds_to(expr: Expr, expected: f64) { @@ -246,7 +253,7 @@ mod tests { assert!(matches!(*r, Expr::Const(v, _) if v == 3.0)); match *l { Expr::Op2(BinaryOp::Mul, ll, lr, _) => { - assert!(matches!(*ll, Expr::Var(0, _))); + assert!(*ll == var(0)); assert!(matches!(*lr, Expr::Const(v, _) if v == 2.0)); } _ => panic!("inner Mul must be preserved"), @@ -263,7 +270,7 @@ mod tests { match fold_constants(expr) { Expr::Op2(BinaryOp::Mul, l, r, _) => { assert!(matches!(*l, Expr::Const(v, _) if v == 6.0)); - assert!(matches!(*r, Expr::Var(0, _))); + assert!(*r == var(0)); } _ => panic!("expected Mul(Const(6), Var)"), } @@ -285,7 +292,7 @@ mod tests { Box::new(var(2)), Loc::default(), ); - assert!(matches!(fold_constants(expr), Expr::Var(1, _))); + assert_eq!(fold_constants(expr), var(1)); let expr = Expr::If( Box::new(c(0.0)), @@ -306,9 +313,9 @@ mod tests { ); match fold_constants(expr) { Expr::If(cond, t, f, _) => { - assert!(matches!(*cond, Expr::Var(0, _))); + assert!(*cond == var(0)); assert!(matches!(*t, Expr::Const(v, _) if v == 2.0)); - assert!(matches!(*f, Expr::Var(2, _))); + assert!(*f == var(2)); } _ => panic!("If with dynamic condition must be preserved"), } @@ -328,7 +335,7 @@ mod tests { match fold_constants(expr) { Expr::App(BuiltinFn::Step(a, b), _) => { assert!(matches!(*a, Expr::Const(v, _) if v == 3.0)); - assert!(matches!(*b, Expr::Var(0, _))); + assert!(*b == var(0)); } _ => panic!("Step must be preserved with folded args"), } @@ -336,17 +343,17 @@ mod tests { #[test] fn folds_inside_assignments_and_module_args() { - let expr = Expr::AssignCurr(7, Box::new(op2(BinaryOp::Mul, c(2.0), c(3.0)))); + let expr = Expr::AssignCurr(vref(7), Box::new(op2(BinaryOp::Mul, c(2.0), c(3.0)))); match fold_constants(expr) { - Expr::AssignCurr(7, rhs) => { + Expr::AssignCurr(dst, rhs) if dst == vref(7) => { assert!(matches!(*rhs, Expr::Const(v, _) if v == 6.0)); } _ => panic!("AssignCurr must be preserved"), } - let expr = Expr::AssignNext(3, Box::new(op2(BinaryOp::Add, c(1.0), c(1.0)))); + let expr = Expr::AssignNext(vref(3), Box::new(op2(BinaryOp::Add, c(1.0), c(1.0)))); match fold_constants(expr) { - Expr::AssignNext(3, rhs) => { + Expr::AssignNext(dst, rhs) if dst == vref(3) => { assert!(matches!(*rhs, Expr::Const(v, _) if v == 2.0)); } _ => panic!("AssignNext must be preserved"), @@ -361,7 +368,7 @@ mod tests { match fold_constants(expr) { Expr::EvalModule(_, _, _, args) => { assert!(matches!(args[0], Expr::Const(v, _) if v == 2.0)); - assert!(matches!(args[1], Expr::Var(0, _))); + assert_eq!(args[1], var(0)); } _ => panic!("EvalModule must be preserved"), } @@ -370,7 +377,7 @@ mod tests { #[test] fn folds_subscript_indices() { let expr = Expr::Subscript( - 0, + vref(0), vec![ SubscriptIndex::Single(op2(BinaryOp::Add, c(1.0), c(1.0))), SubscriptIndex::Range(c(1.0), op2(BinaryOp::Add, c(1.0), c(2.0))), @@ -379,7 +386,7 @@ mod tests { Loc::default(), ); match fold_constants(expr) { - Expr::Subscript(0, indices, _, _) => { + Expr::Subscript(base, indices, _, _) if base == vref(0) => { assert!( matches!(&indices[0], SubscriptIndex::Single(Expr::Const(v, _)) if *v == 2.0) ); diff --git a/src/simlin-engine/src/compiler/invariance.rs b/src/simlin-engine/src/compiler/invariance.rs index 3012e27fc..ef77bc22d 100644 --- a/src/simlin-engine/src/compiler/invariance.rs +++ b/src/simlin-engine/src/compiler/invariance.rs @@ -15,13 +15,12 @@ //! definition and the soundness argument. //! //! This is the **functional core**: a pure walk over the lowered `Expr` tree -//! parameterized by an offset-classification callback. The callback resolves a -//! slot offset to the run-invariance verdict of its owning variable (invariant, -//! or variant because it is a dynamic var / a stock / a module-instance slot / -//! a time-global other than DT/INITIAL/FINAL). The two compile paths supply -//! different callbacks (the salsa per-fragment path resolves mini-offsets to -//! dependency names; the monolithic test path resolves model-global offsets via -//! the metadata map), but share this walk so they classify identically. +//! parameterized by a reference-classification callback. The callback resolves a +//! [`VarRef`] to the run-invariance verdict of the variable it names (invariant, +//! or variant because it is a dynamic var / a stock / a module instance / a +//! time-global other than DT/INITIAL/FINAL). Both compile paths look the +//! variable up by the name the reference carries, and share this walk, so they +//! classify identically. //! //! The walk is **exhaustive** over every `Expr` and `BuiltinFn` variant with //! explicit arms and is **default-variant**: anything not positively recognized @@ -30,23 +29,22 @@ use crate::builtins::BuiltinFn; -use super::expr::{Expr, SubscriptIndex}; +use super::expr::{Expr, SubscriptIndex, VarRef}; -/// The run-invariance verdict for the variable owning a referenced slot offset. +/// The run-invariance verdict for the variable a reference names. /// -/// The offset-classification callback returns this for an `Expr::Var(off)` / -/// `Expr::Subscript(off, ..)` / `Expr::StaticSubscript(off, ..)` base offset. -/// `Variant` covers every non-invariant source: a dynamic variable, a stock, a -/// slot inside a module instance's range, and a time-global other than -/// DT/INITIAL/FINAL (those three reach the lowered Expr as builtins, not as -/// `Var` offsets, but the callback rejects a stray `Var` reference to any -/// implicit-global slot defensively). +/// The classification callback returns this for an `Expr::Var` / +/// `Expr::Subscript` / `Expr::StaticSubscript` base reference. `Variant` covers +/// every non-invariant source: a dynamic variable, a stock, a module instance, +/// and a time-global other than DT/INITIAL/FINAL (those three reach the lowered +/// Expr as builtins, not as variable references, but the callback rejects a +/// stray reference to any implicit-global slot defensively). #[derive(Clone, Copy, PartialEq, Eq, Debug)] -pub(crate) enum OffsetClass { - /// The offset resolves to a run-invariant variable. +pub(crate) enum RefClass { + /// The reference resolves to a run-invariant variable. Invariant, - /// The offset resolves to a variant source (dynamic var, stock, module - /// instance slot, or a time-global other than DT/INITIAL/FINAL). + /// The reference resolves to a variant source (dynamic var, stock, module + /// instance, or a time-global other than DT/INITIAL/FINAL). /// /// In production, `compute_flow_invariance_support` always returns /// `Invariant` (to check structural purity only), so `Variant` is only @@ -57,28 +55,28 @@ pub(crate) enum OffsetClass { /// Returns true iff every expression in `exprs` (one variable's lowered /// flow-phase statement list) is run-invariant, given the dependency verdicts -/// from `classify_offset`. +/// from `classify_ref`. /// /// `exprs` is the variable's own `Var.ast`: a list of statements that ends in /// `AssignCurr`/`AssignNext`/`AssignTemp` writes plus any `AssignTemp` /// scratch-array precomputations. A variable is invariant iff ALL of its /// statements are invariant. -pub(crate) fn exprs_are_invariant(exprs: &[Expr], classify_offset: &F) -> bool +pub(crate) fn exprs_are_invariant(exprs: &[Expr], classify_ref: &F) -> bool where - F: Fn(usize) -> OffsetClass, + F: Fn(&VarRef) -> RefClass, { exprs .iter() - .all(|expr| expr_is_invariant(expr, classify_offset)) + .all(|expr| expr_is_invariant(expr, classify_ref)) } /// Returns true iff a single lowered expression is run-invariant. /// /// Exhaustive over every `Expr` variant. Default-variant: a variant is /// invariant only if positively matched here. -fn expr_is_invariant(expr: &Expr, classify_offset: &F) -> bool +fn expr_is_invariant(expr: &Expr, classify_ref: &F) -> bool where - F: Fn(usize) -> OffsetClass, + F: Fn(&VarRef) -> RefClass, { match expr { // Literals and DT are run-invariant by definition. @@ -89,13 +87,13 @@ where // time-globals other than DT/INITIAL/FINAL). Dynamic-subscript index // exprs must ALSO be invariant -- a variant index changes which element // is read each step even when the base array is invariant. - Expr::Var(off, _) => classify_offset(*off) == OffsetClass::Invariant, - Expr::StaticSubscript(off, _, _) => classify_offset(*off) == OffsetClass::Invariant, - Expr::Subscript(off, indices, _, _) => { - classify_offset(*off) == OffsetClass::Invariant + Expr::Var(var, _) => classify_ref(var) == RefClass::Invariant, + Expr::StaticSubscript(base, _, _) => classify_ref(base) == RefClass::Invariant, + Expr::Subscript(base, indices, _, _) => { + classify_ref(base) == RefClass::Invariant && indices .iter() - .all(|idx| subscript_index_is_invariant(idx, classify_offset)) + .all(|idx| subscript_index_is_invariant(idx, classify_ref)) } // Temp arrays are intra-statement scratch: a `TempArray`/ @@ -114,37 +112,37 @@ where Expr::EvalModule(_, _, _, _) | Expr::ModuleInput(_, _) => false, // Builtins: see `builtin_is_invariant`. - Expr::App(builtin, _) => builtin_is_invariant(builtin, classify_offset), + Expr::App(builtin, _) => builtin_is_invariant(builtin, classify_ref), // Compound exprs: invariant iff all operands are. Expr::Op2(_, l, r, _) => { - expr_is_invariant(l, classify_offset) && expr_is_invariant(r, classify_offset) + expr_is_invariant(l, classify_ref) && expr_is_invariant(r, classify_ref) } - Expr::Op1(_, operand, _) => expr_is_invariant(operand, classify_offset), + Expr::Op1(_, operand, _) => expr_is_invariant(operand, classify_ref), Expr::If(cond, t, f, _) => { // The VM evaluates BOTH branches every step, so all three must be // invariant. - expr_is_invariant(cond, classify_offset) - && expr_is_invariant(t, classify_offset) - && expr_is_invariant(f, classify_offset) + expr_is_invariant(cond, classify_ref) + && expr_is_invariant(t, classify_ref) + && expr_is_invariant(f, classify_ref) } // Assignments: invariant iff the assigned expression is. Expr::AssignCurr(_, rhs) | Expr::AssignNext(_, rhs) | Expr::AssignTemp(_, rhs, _) => { - expr_is_invariant(rhs, classify_offset) + expr_is_invariant(rhs, classify_ref) } } } /// A dynamic-subscript index component is invariant iff its index expr(s) are. -fn subscript_index_is_invariant(idx: &SubscriptIndex, classify_offset: &F) -> bool +fn subscript_index_is_invariant(idx: &SubscriptIndex, classify_ref: &F) -> bool where - F: Fn(usize) -> OffsetClass, + F: Fn(&VarRef) -> RefClass, { match idx { - SubscriptIndex::Single(e) => expr_is_invariant(e, classify_offset), + SubscriptIndex::Single(e) => expr_is_invariant(e, classify_ref), SubscriptIndex::Range(start, end) => { - expr_is_invariant(start, classify_offset) && expr_is_invariant(end, classify_offset) + expr_is_invariant(start, classify_ref) && expr_is_invariant(end, classify_ref) } } } @@ -157,13 +155,13 @@ where /// lookup with an invariant index, or a pure function whose every argument is /// invariant. `TIME`, `PULSE`/`RAMP`/`STEP` (time-dependent even with constant /// args), and `PREVIOUS` (reads `prev_values`) are variant. -fn builtin_is_invariant(builtin: &BuiltinFn, classify_offset: &F) -> bool +fn builtin_is_invariant(builtin: &BuiltinFn, classify_ref: &F) -> bool where - F: Fn(usize) -> OffsetClass, + F: Fn(&VarRef) -> RefClass, { use BuiltinFn::*; // Walk a slice of subexpressions, all of which must be invariant. - let all = |args: &[&Expr]| args.iter().all(|e| expr_is_invariant(e, classify_offset)); + let all = |args: &[&Expr]| args.iter().all(|e| expr_is_invariant(e, classify_ref)); match builtin { // Fixed time-globals and nullary constants. Inf | Pi | TimeStep | StartTime | FinalTime | IsModuleInput(_, _) => true, @@ -191,17 +189,17 @@ where Abs(a) | Arccos(a) | Arcsin(a) | Arctan(a) | Cos(a) | Exp(a) | Int(a) | Ln(a) | Log10(a) | Sign(a) | Sin(a) | Sqrt(a) | Tan(a) => all(&[a]), Max(a, b) | Min(a, b) => { - expr_is_invariant(a, classify_offset) + expr_is_invariant(a, classify_ref) && b.as_ref() - .is_none_or(|b| expr_is_invariant(b, classify_offset)) + .is_none_or(|b| expr_is_invariant(b, classify_ref)) } - Mean(args) => args.iter().all(|e| expr_is_invariant(e, classify_offset)), + Mean(args) => args.iter().all(|e| expr_is_invariant(e, classify_ref)), Quantum(a, b) => all(&[a, b]), SafeDiv(a, b, c) => { - expr_is_invariant(a, classify_offset) - && expr_is_invariant(b, classify_offset) + expr_is_invariant(a, classify_ref) + && expr_is_invariant(b, classify_ref) && c.as_ref() - .is_none_or(|c| expr_is_invariant(c, classify_offset)) + .is_none_or(|c| expr_is_invariant(c, classify_ref)) } Sshape(a, b, c) => all(&[a, b, c]), @@ -224,25 +222,30 @@ mod tests { use crate::ast::{BinaryOp, Loc}; use crate::compiler::dimensions::UnaryOp; - /// An offset callback where a fixed set of offsets is invariant and all - /// others are variant. Mirrors the per-model verdict map both production - /// paths supply. - fn classifier(invariant_offsets: &[usize]) -> impl Fn(usize) -> OffsetClass + '_ { - move |off| { - if invariant_offsets.contains(&off) { - OffsetClass::Invariant + /// A callback where a fixed set of variables is invariant and all others + /// are variant. Mirrors the per-model verdict map both production paths + /// supply. + fn classifier(invariant: &[usize]) -> impl Fn(&VarRef) -> RefClass + '_ { + move |var: &VarRef| { + if invariant.iter().any(|n| vref(*n).name == var.name) { + RefClass::Invariant } else { - OffsetClass::Variant + RefClass::Variant } } } + /// A distinct variable reference per test slot. + fn vref(n: usize) -> VarRef { + VarRef::base(crate::common::Ident::new(&format!("v{n}"))) + } + fn lit(n: f64) -> Expr { Expr::Const(n, Loc::default()) } - fn var(off: usize) -> Expr { - Expr::Var(off, Loc::default()) + fn var(n: usize) -> Expr { + Expr::Var(vref(n), Loc::default()) } fn add(l: Expr, r: Expr) -> Expr { @@ -252,21 +255,21 @@ mod tests { /// A bare constant assignment: `dst = 3` lowered as `AssignCurr(0, Const)`. #[test] fn constant_is_invariant() { - let exprs = vec![Expr::AssignCurr(0, Box::new(lit(3.0)))]; + let exprs = vec![Expr::AssignCurr(vref(0), Box::new(lit(3.0)))]; assert!(exprs_are_invariant(&exprs, &classifier(&[]))); } - /// `dst = a + 2` where `a` (offset 1) is an invariant variable. + /// `dst = a + 2` where `a` (`v1`) is an invariant variable. #[test] fn const_derived_chain_is_invariant() { - let exprs = vec![Expr::AssignCurr(0, Box::new(add(var(1), lit(2.0))))]; + let exprs = vec![Expr::AssignCurr(vref(0), Box::new(add(var(1), lit(2.0))))]; assert!(exprs_are_invariant(&exprs, &classifier(&[1]))); } - /// `dst = a + 2` where `a` (offset 1) is a *variant* variable. + /// `dst = a + 2` where `a` (`v1`) is a *variant* variable. #[test] fn variant_dependency_is_variant() { - let exprs = vec![Expr::AssignCurr(0, Box::new(add(var(1), lit(2.0))))]; + let exprs = vec![Expr::AssignCurr(vref(0), Box::new(add(var(1), lit(2.0))))]; assert!(!exprs_are_invariant(&exprs, &classifier(&[]))); } @@ -274,7 +277,7 @@ mod tests { #[test] fn time_builtin_is_variant() { let exprs = vec![Expr::AssignCurr( - 0, + vref(0), Box::new(Expr::App(BuiltinFn::Time, Loc::default())), )]; assert!(!exprs_are_invariant(&exprs, &classifier(&[]))); @@ -283,7 +286,10 @@ mod tests { /// `dst = DT` is invariant (DT is fixed for the run). #[test] fn dt_is_invariant() { - let exprs = vec![Expr::AssignCurr(0, Box::new(Expr::Dt(Loc::default())))]; + let exprs = vec![Expr::AssignCurr( + vref(0), + Box::new(Expr::Dt(Loc::default())), + )]; assert!(exprs_are_invariant(&exprs, &classifier(&[]))); } @@ -297,7 +303,10 @@ mod tests { BuiltinFn::Pi, BuiltinFn::Inf, ] { - let exprs = vec![Expr::AssignCurr(0, Box::new(Expr::App(bf, Loc::default())))]; + let exprs = vec![Expr::AssignCurr( + vref(0), + Box::new(Expr::App(bf, Loc::default())), + )]; assert!(exprs_are_invariant(&exprs, &classifier(&[]))); } } @@ -306,7 +315,7 @@ mod tests { #[test] fn pulse_is_variant() { let exprs = vec![Expr::AssignCurr( - 0, + vref(0), Box::new(Expr::App( BuiltinFn::Pulse(Box::new(lit(1.0)), Box::new(lit(2.0)), None), Loc::default(), @@ -319,7 +328,7 @@ mod tests { #[test] fn ramp_is_variant() { let exprs = vec![Expr::AssignCurr( - 0, + vref(0), Box::new(Expr::App( BuiltinFn::Ramp(Box::new(lit(1.0)), Box::new(lit(2.0)), None), Loc::default(), @@ -332,7 +341,7 @@ mod tests { #[test] fn step_is_variant() { let exprs = vec![Expr::AssignCurr( - 0, + vref(0), Box::new(Expr::App( BuiltinFn::Step(Box::new(lit(1.0)), Box::new(lit(2.0))), Loc::default(), @@ -346,7 +355,7 @@ mod tests { #[test] fn previous_is_variant() { let exprs = vec![Expr::AssignCurr( - 0, + vref(0), Box::new(Expr::App( BuiltinFn::Previous(Box::new(var(1)), Box::new(lit(0.0))), Loc::default(), @@ -360,10 +369,10 @@ mod tests { #[test] fn init_of_any_variable_is_invariant() { let exprs = vec![Expr::AssignCurr( - 0, + vref(0), Box::new(Expr::App(BuiltinFn::Init(Box::new(var(1))), Loc::default())), )]; - // offset 1 is NOT invariant, yet INIT(a) is still invariant. + // `v1` is NOT invariant, yet INIT(a) is still invariant. assert!(exprs_are_invariant(&exprs, &classifier(&[]))); } @@ -373,13 +382,13 @@ mod tests { #[test] fn lookup_of_constant_is_invariant() { let exprs = vec![Expr::AssignCurr( - 0, + vref(0), Box::new(Expr::App( BuiltinFn::Lookup(Box::new(var(5)), Box::new(lit(3.0)), Loc::default()), Loc::default(), )), )]; - // offset 5 (the static table holder) is invariant. + // `v5` (the static table holder) is invariant. assert!(exprs_are_invariant(&exprs, &classifier(&[5]))); } @@ -387,7 +396,7 @@ mod tests { #[test] fn lookup_of_time_is_variant() { let exprs = vec![Expr::AssignCurr( - 0, + vref(0), Box::new(Expr::App( BuiltinFn::Lookup( Box::new(var(5)), @@ -400,21 +409,21 @@ mod tests { assert!(!exprs_are_invariant(&exprs, &classifier(&[5]))); } - /// A stock dependency is variant (the callback classifies the stock's - /// offset as Variant). `dst = stock + 1`. + /// A stock dependency is variant (the callback classifies the stock as + /// Variant). `dst = stock + 1`. #[test] fn stock_dependency_is_variant() { - let exprs = vec![Expr::AssignCurr(0, Box::new(add(var(1), lit(1.0))))]; - // offset 1 is a stock -> Variant. + let exprs = vec![Expr::AssignCurr(vref(0), Box::new(add(var(1), lit(1.0))))]; + // `v1` is a stock -> Variant. assert!(!exprs_are_invariant(&exprs, &classifier(&[]))); } - /// False-positive class 1: a module-output read via a plain `Var` offset is - /// variant (the callback classifies the module-instance slot as Variant). + /// False-positive class 1: a module-output read via a plain `Var` reference + /// is variant (the callback classifies the module instance as Variant). #[test] fn module_output_read_is_variant() { - let exprs = vec![Expr::AssignCurr(0, Box::new(var(7)))]; - // offset 7 is a slot inside a module instance -> Variant. + let exprs = vec![Expr::AssignCurr(vref(0), Box::new(var(7)))]; + // `v7` is a slot inside a module instance -> Variant. assert!(!exprs_are_invariant(&exprs, &classifier(&[]))); } @@ -424,10 +433,10 @@ mod tests { fn static_subscript_of_variant_array_is_variant() { let view = crate::ast::ArrayView::contiguous(vec![3]); let exprs = vec![Expr::AssignCurr( - 0, - Box::new(Expr::StaticSubscript(2, view, Loc::default())), + vref(0), + Box::new(Expr::StaticSubscript(vref(2), view, Loc::default())), )]; - // offset 2 (the array base) is variant. + // `v2` (the array) is variant. assert!(!exprs_are_invariant(&exprs, &classifier(&[]))); } @@ -436,8 +445,8 @@ mod tests { fn static_subscript_of_invariant_array_is_invariant() { let view = crate::ast::ArrayView::contiguous(vec![3]); let exprs = vec![Expr::AssignCurr( - 0, - Box::new(Expr::StaticSubscript(2, view, Loc::default())), + vref(0), + Box::new(Expr::StaticSubscript(vref(2), view, Loc::default())), )]; assert!(exprs_are_invariant(&exprs, &classifier(&[2]))); } @@ -447,7 +456,7 @@ mod tests { #[test] fn module_input_is_variant() { let exprs = vec![Expr::AssignCurr( - 0, + vref(0), Box::new(Expr::ModuleInput(0, Loc::default())), )]; assert!(!exprs_are_invariant(&exprs, &classifier(&[]))); @@ -472,9 +481,9 @@ mod tests { #[test] fn dynamic_subscript_invariant_base_and_index_is_invariant() { let exprs = vec![Expr::AssignCurr( - 0, + vref(0), Box::new(Expr::Subscript( - 2, + vref(2), vec![SubscriptIndex::Single(lit(1.0))], vec![3], Loc::default(), @@ -488,15 +497,15 @@ mod tests { #[test] fn dynamic_subscript_variant_index_is_variant() { let exprs = vec![Expr::AssignCurr( - 0, + vref(0), Box::new(Expr::Subscript( - 2, + vref(2), vec![SubscriptIndex::Single(var(9))], vec![3], Loc::default(), )), )]; - // base offset 2 is invariant, but index var 9 is variant. + // the base `v2` is invariant, but the index `v9` is variant. assert!(!exprs_are_invariant(&exprs, &classifier(&[2]))); } @@ -506,10 +515,10 @@ mod tests { fn sum_reducer_tracks_argument() { let view = crate::ast::ArrayView::contiguous(vec![3]); let inv = vec![Expr::AssignCurr( - 0, + vref(0), Box::new(Expr::App( BuiltinFn::Sum(Box::new(Expr::StaticSubscript( - 2, + vref(2), view.clone(), Loc::default(), ))), @@ -519,9 +528,13 @@ mod tests { assert!(exprs_are_invariant(&inv, &classifier(&[2]))); let variant = vec![Expr::AssignCurr( - 0, + vref(0), Box::new(Expr::App( - BuiltinFn::Sum(Box::new(Expr::StaticSubscript(2, view, Loc::default()))), + BuiltinFn::Sum(Box::new(Expr::StaticSubscript( + vref(2), + view, + Loc::default(), + ))), Loc::default(), )), )]; @@ -538,17 +551,17 @@ mod tests { // temp 0 = invariant array (base var 3, which is invariant) Expr::AssignTemp( 0, - Box::new(Expr::StaticSubscript(3, view.clone(), Loc::default())), + Box::new(Expr::StaticSubscript(vref(3), view.clone(), Loc::default())), view.clone(), ), // dst[0] = temp0[0] Expr::AssignCurr( - 0, + vref(0), Box::new(Expr::TempArrayElement(0, view.clone(), 0, Loc::default())), ), // dst[1] = temp0[1] Expr::AssignCurr( - 1, + vref(1), Box::new(Expr::TempArrayElement(0, view, 1, Loc::default())), ), ]; @@ -562,11 +575,11 @@ mod tests { let exprs = vec![ Expr::AssignTemp( 0, - Box::new(Expr::StaticSubscript(3, view.clone(), Loc::default())), + Box::new(Expr::StaticSubscript(vref(3), view.clone(), Loc::default())), view.clone(), ), Expr::AssignCurr( - 0, + vref(0), Box::new(Expr::TempArrayElement(0, view, 0, Loc::default())), ), ]; @@ -579,7 +592,7 @@ mod tests { #[test] fn if_invariant_when_all_branches_invariant() { let inv = vec![Expr::AssignCurr( - 0, + vref(0), Box::new(Expr::If( Box::new(lit(1.0)), Box::new(var(1)), @@ -590,7 +603,7 @@ mod tests { assert!(exprs_are_invariant(&inv, &classifier(&[1]))); let variant = vec![Expr::AssignCurr( - 0, + vref(0), Box::new(Expr::If( Box::new(lit(1.0)), Box::new(var(1)), @@ -606,7 +619,7 @@ mod tests { #[test] fn op1_tracks_operand() { let exprs = vec![Expr::AssignCurr( - 0, + vref(0), Box::new(Expr::Op1(UnaryOp::Not, Box::new(var(1)), Loc::default())), )]; assert!(exprs_are_invariant(&exprs, &classifier(&[1]))); diff --git a/src/simlin-engine/src/compiler/mod.rs b/src/simlin-engine/src/compiler/mod.rs index cc6f890b5..2a7403435 100644 --- a/src/simlin-engine/src/compiler/mod.rs +++ b/src/simlin-engine/src/compiler/mod.rs @@ -12,15 +12,18 @@ pub mod pretty; pub mod subscript; pub(crate) mod symbolic; -use std::collections::{BTreeSet, HashMap, HashSet}; +use std::collections::{BTreeSet, HashMap}; use std::sync::Arc; -use crate::ast::{ArrayView, Ast, Loc}; +use crate::ast::{ArrayView, Ast, BinaryOp, Loc}; +#[cfg(test)] use crate::bytecode::CompiledModule; use crate::common::{Canonical, CanonicalElementName, Ident, Result}; #[cfg(test)] use crate::common::{Error, ErrorCode, ErrorKind}; -use crate::dimensions::{Dimension, DimensionsContext, SubscriptIterator}; +#[cfg(test)] +use crate::dimensions::DimensionsContext; +use crate::dimensions::{Dimension, SubscriptIterator}; #[cfg(test)] use crate::model::ModelStage1; #[cfg(test)] @@ -29,16 +32,40 @@ use crate::sim_err; use crate::variable::Variable; #[cfg(test)] use crate::vm::IMPLICIT_VAR_COUNT; -use crate::vm::ModuleKey; // Re-exports for crate-internal API -pub(crate) use self::context::{Context, ContextCore, VariableMetadata}; -pub(crate) use self::expr::{BuiltinFn, Expr, SubscriptIndex, Table}; +pub(crate) use self::codegen::ModuleCtx; +pub(crate) use self::context::{Context, ContextCore, VariableMetadata, whole_variable_extents}; +pub(crate) use self::expr::{BuiltinFn, Expr, SubscriptIndex, Table, VarRef}; -use self::codegen::Compiler; - -// Type alias to reduce complexity -type VariableOffsetMap = HashMap, HashMap, (usize, usize)>>; +/// The total slot count of the variable a reference addresses **in whole**, +/// keyed by that reference. +/// +/// This is all codegen needs from the symbol table now that references carry +/// names: the *extent* of a VECTOR ELM MAP's source variable +/// (`codegen::full_source_len`). Every other lookup that used to run backwards +/// through a `name -> (offset, size)` map -- the identity of a lookup table, +/// the base slot of a module instance -- reads the name off the reference +/// directly. Offsets are assigned once, at assembly, and never appear here. +/// +/// Keyed by the whole `VarRef` rather than by name, because a reference does +/// not always name the variable whose extent it asks about. A CROSS-MODULE +/// reference `m·x` lowers to `VarRef { name: m, element_offset: x's slot +/// inside the instance }`, and `m`'s own slot count -- the whole sub-model +/// block -- is the extent of nothing a reference can name. Each instance +/// therefore contributes one entry per sub-model variable, at that variable's +/// slot; a reference sitting at a sub-model variable's base reports THAT +/// variable's extent, and one landing mid-array is simply absent. Absence is +/// the same answer an in-model mid-array reference gets, so the caller's +/// fallback ("all the view can honestly report") is unchanged. +/// +/// Built once per emission unit by [`context::whole_variable_extents`]. That is +/// the only place the rule is DERIVED -- every production site calls it, so +/// lowering and emission cannot disagree about what a reference addresses. +/// (`codegen`'s unit tests hand-build a table instead, in order to state the +/// lookup's contract against inputs of their own choosing; they exercise the +/// reader, not the rule.) +pub(crate) type VarSizes = HashMap; #[cfg_attr(feature = "debug-derive", derive(Debug))] #[derive(PartialEq, Clone)] @@ -69,7 +96,7 @@ fn test_fold_flows() { metadata.insert( Ident::new("a"), VariableMetadata { - offset: 1, + offset: Some(1), size: 1, var: &dummy_var, }, @@ -77,7 +104,7 @@ fn test_fold_flows() { metadata.insert( Ident::new("b"), VariableMetadata { - offset: 2, + offset: Some(2), size: 1, var: &dummy_var, }, @@ -85,7 +112,7 @@ fn test_fold_flows() { metadata.insert( Ident::new("c"), VariableMetadata { - offset: 3, + offset: Some(3), size: 1, var: &dummy_var, }, @@ -93,7 +120,7 @@ fn test_fold_flows() { metadata.insert( Ident::new("d"), VariableMetadata { - offset: 4, + offset: Some(4), size: 1, var: &dummy_var, }, @@ -103,12 +130,14 @@ fn test_fold_flows() { let test_ident = Ident::new("test"); metadata2.insert(main_ident.clone(), metadata); let dims_ctx = DimensionsContext::default(); + let var_sizes = whole_variable_extents(&metadata2, &main_ident); let ctx = Context::new( ContextCore { dimensions: &[], dimensions_ctx: &dims_ctx, model_name: &main_ident, metadata: &metadata2, + var_sizes: &var_sizes, module_models: &module_models, inputs, }, @@ -118,14 +147,17 @@ fn test_fold_flows() { assert_eq!(Ok(None), ctx.fold_flows(&[])); assert_eq!( - Ok(Some(Expr::Var(1, Loc::default()))), + Ok(Some(Expr::Var( + VarRef::base(Ident::new("a")), + Loc::default() + ))), ctx.fold_flows(&[Ident::new("a")]) ); assert_eq!( Ok(Some(Expr::Op2( crate::ast::BinaryOp::Add, - Box::new(Expr::Var(1, Loc::default())), - Box::new(Expr::Var(4, Loc::default())), + Box::new(Expr::Var(VarRef::base(Ident::new("a")), Loc::default())), + Box::new(Expr::Var(VarRef::base(Ident::new("d")), Loc::default())), Loc::default(), ))), ctx.fold_flows(&[Ident::new("a"), Ident::new("d")]) @@ -174,7 +206,7 @@ fn test_module_var_new_missing_input_source_returns_error() { metadata.insert( module_ident.clone(), VariableMetadata { - offset: IMPLICIT_VAR_COUNT, + offset: Some(IMPLICIT_VAR_COUNT), size: 1, var: &module_var, }, @@ -183,12 +215,14 @@ fn test_module_var_new_missing_input_source_returns_error() { metadata2.insert(main_ident.clone(), metadata); let dims_ctx = DimensionsContext::default(); + let var_sizes = whole_variable_extents(&metadata2, &main_ident); let ctx = Context::new( ContextCore { dimensions: &[], dimensions_ctx: &dims_ctx, model_name: &main_ident, metadata: &metadata2, + var_sizes: &var_sizes, module_models: &module_models, inputs, }, @@ -236,7 +270,7 @@ fn test_build_stock_update_expr_inflows_only() { metadata.insert( Ident::new("stock"), VariableMetadata { - offset: 0, + offset: Some(0), size: 1, var: &dummy_var, }, @@ -244,7 +278,7 @@ fn test_build_stock_update_expr_inflows_only() { metadata.insert( Ident::new("inflow"), VariableMetadata { - offset: 1, + offset: Some(1), size: 1, var: &dummy_var, }, @@ -254,12 +288,14 @@ fn test_build_stock_update_expr_inflows_only() { let test_ident = Ident::new("test"); metadata2.insert(main_ident.clone(), metadata); let dims_ctx = DimensionsContext::default(); + let var_sizes = whole_variable_extents(&metadata2, &main_ident); let ctx = Context::new( ContextCore { dimensions: &[], dimensions_ctx: &dims_ctx, model_name: &main_ident, metadata: &metadata2, + var_sizes: &var_sizes, module_models: &module_models, inputs, }, @@ -267,16 +303,24 @@ fn test_build_stock_update_expr_inflows_only() { false, ); - let result = ctx.build_stock_update_expr(0, &stock_var).unwrap(); + let result = ctx + .build_stock_update_expr(&VarRef::base(Ident::new("stock")), &stock_var) + .unwrap(); // stock + (inflow - 0.0) * dt // outflows should be Const(0.0) since there are none if let Expr::Op2(crate::ast::BinaryOp::Add, stock_box, dt_update_box, _) = &result { - assert!(matches!(stock_box.as_ref(), Expr::Var(0, _))); + assert_eq!( + stock_box.as_ref(), + &Expr::Var(VarRef::base(Ident::new("stock")), Loc::default()) + ); if let Expr::Op2(crate::ast::BinaryOp::Mul, sub_box, dt_box, _) = dt_update_box.as_ref() { assert!(matches!(dt_box.as_ref(), Expr::Dt(_))); if let Expr::Op2(crate::ast::BinaryOp::Sub, in_box, out_box, _) = sub_box.as_ref() { - assert!(matches!(in_box.as_ref(), Expr::Var(1, _))); + assert_eq!( + in_box.as_ref(), + &Expr::Var(VarRef::base(Ident::new("inflow")), Loc::default()) + ); assert!( matches!(out_box.as_ref(), Expr::Const(v, _) if *v == 0.0), "outflows should be Const(0.0) when empty" @@ -325,7 +369,7 @@ fn test_build_stock_update_expr_outflows_only() { metadata.insert( Ident::new("stock"), VariableMetadata { - offset: 0, + offset: Some(0), size: 1, var: &dummy_var, }, @@ -333,7 +377,7 @@ fn test_build_stock_update_expr_outflows_only() { metadata.insert( Ident::new("outflow"), VariableMetadata { - offset: 1, + offset: Some(1), size: 1, var: &dummy_var, }, @@ -343,12 +387,14 @@ fn test_build_stock_update_expr_outflows_only() { let test_ident = Ident::new("test"); metadata2.insert(main_ident.clone(), metadata); let dims_ctx = DimensionsContext::default(); + let var_sizes = whole_variable_extents(&metadata2, &main_ident); let ctx = Context::new( ContextCore { dimensions: &[], dimensions_ctx: &dims_ctx, model_name: &main_ident, metadata: &metadata2, + var_sizes: &var_sizes, module_models: &module_models, inputs, }, @@ -356,19 +402,27 @@ fn test_build_stock_update_expr_outflows_only() { false, ); - let result = ctx.build_stock_update_expr(0, &stock_var).unwrap(); + let result = ctx + .build_stock_update_expr(&VarRef::base(Ident::new("stock")), &stock_var) + .unwrap(); // stock + (0.0 - outflow) * dt // inflows should be Const(0.0) since there are none if let Expr::Op2(crate::ast::BinaryOp::Add, stock_box, dt_update_box, _) = &result { - assert!(matches!(stock_box.as_ref(), Expr::Var(0, _))); + assert_eq!( + stock_box.as_ref(), + &Expr::Var(VarRef::base(Ident::new("stock")), Loc::default()) + ); if let Expr::Op2(crate::ast::BinaryOp::Mul, sub_box, _, _) = dt_update_box.as_ref() { if let Expr::Op2(crate::ast::BinaryOp::Sub, in_box, out_box, _) = sub_box.as_ref() { assert!( matches!(in_box.as_ref(), Expr::Const(v, _) if *v == 0.0), "inflows should be Const(0.0) when empty" ); - assert!(matches!(out_box.as_ref(), Expr::Var(1, _))); + assert_eq!( + out_box.as_ref(), + &Expr::Var(VarRef::base(Ident::new("outflow")), Loc::default()) + ); } else { panic!("Expected Sub expression in stock update"); } @@ -413,7 +467,7 @@ fn test_build_stock_update_expr_no_flows() { metadata.insert( Ident::new("stock"), VariableMetadata { - offset: 0, + offset: Some(0), size: 1, var: &dummy_var, }, @@ -423,12 +477,14 @@ fn test_build_stock_update_expr_no_flows() { let test_ident = Ident::new("test"); metadata2.insert(main_ident.clone(), metadata); let dims_ctx = DimensionsContext::default(); + let var_sizes = whole_variable_extents(&metadata2, &main_ident); let ctx = Context::new( ContextCore { dimensions: &[], dimensions_ctx: &dims_ctx, model_name: &main_ident, metadata: &metadata2, + var_sizes: &var_sizes, module_models: &module_models, inputs, }, @@ -436,7 +492,9 @@ fn test_build_stock_update_expr_no_flows() { false, ); - let result = ctx.build_stock_update_expr(0, &stock_var).unwrap(); + let result = ctx + .build_stock_update_expr(&VarRef::base(Ident::new("stock")), &stock_var) + .unwrap(); // stock + (0.0 - 0.0) * dt if let Expr::Op2(crate::ast::BinaryOp::Add, _, dt_update_box, _) = &result { @@ -501,7 +559,7 @@ fn test_build_stock_update_expr_multiple_flows() { metadata.insert( Ident::new(name), VariableMetadata { - offset: off, + offset: Some(off), size: 1, var: &dummy_var, }, @@ -512,12 +570,14 @@ fn test_build_stock_update_expr_multiple_flows() { let test_ident = Ident::new("test"); metadata2.insert(main_ident.clone(), metadata); let dims_ctx = DimensionsContext::default(); + let var_sizes = whole_variable_extents(&metadata2, &main_ident); let ctx = Context::new( ContextCore { dimensions: &[], dimensions_ctx: &dims_ctx, model_name: &main_ident, metadata: &metadata2, + var_sizes: &var_sizes, module_models: &module_models, inputs, }, @@ -525,11 +585,16 @@ fn test_build_stock_update_expr_multiple_flows() { false, ); - let result = ctx.build_stock_update_expr(0, &stock_var).unwrap(); + let result = ctx + .build_stock_update_expr(&VarRef::base(Ident::new("stock")), &stock_var) + .unwrap(); // stock + ((in1 + in2) - (out1 + out2)) * dt if let Expr::Op2(crate::ast::BinaryOp::Add, stock_box, dt_update_box, _) = &result { - assert!(matches!(stock_box.as_ref(), Expr::Var(0, _))); + assert_eq!( + stock_box.as_ref(), + &Expr::Var(VarRef::base(Ident::new("stock")), Loc::default()) + ); if let Expr::Op2(crate::ast::BinaryOp::Mul, sub_box, dt_box, _) = dt_update_box.as_ref() { assert!(matches!(dt_box.as_ref(), Expr::Dt(_))); if let Expr::Op2(crate::ast::BinaryOp::Sub, in_sum, out_sum, _) = sub_box.as_ref() { @@ -621,7 +686,7 @@ fn test_arrayed_default_equation_applies_to_missing_elements() { model_metadata.insert( Ident::new("x"), VariableMetadata { - offset: 0, + offset: Some(0), size: 3, var: &var, }, @@ -635,12 +700,14 @@ fn test_arrayed_default_equation_applies_to_missing_elements() { HashMap::new(); let dims_ctx = DimensionsContext::from(std::slice::from_ref(&datamodel_dim)); let ident = Ident::new("test"); + let var_sizes = whole_variable_extents(&metadata, &model_name); let ctx = Context::new( ContextCore { dimensions: &dims, dimensions_ctx: &dims_ctx, model_name: &model_name, metadata: &metadata, + var_sizes: &var_sizes, module_models: &module_models, inputs: &inputs, }, @@ -652,9 +719,9 @@ fn test_arrayed_default_equation_applies_to_missing_elements() { let mut assigned: HashMap = HashMap::new(); for expr in lowered.ast { - if let Expr::AssignCurr(off, rhs) = expr { + if let Expr::AssignCurr(dst, rhs) = expr { if let Expr::Const(value, _) = *rhs { - assigned.insert(off, value); + assigned.insert(dst.element_offset, value); } else { panic!("expected AssignCurr to use scalar constants in this test"); } @@ -670,18 +737,81 @@ fn test_arrayed_default_equation_applies_to_missing_elements() { ); } +/// The stock-update shape guard, in both directions. +/// +/// Codegen emits every next-value assignment as the fused `BinOpAssignNext`, +/// so a stock update whose lowered form does not end in an `Op2` must be +/// REJECTED at lowering (a structured, per-variable error) rather than dropped +/// at emission (a stock that silently never integrates). This pins that +/// contract directly, since the shape it guards is unreachable from today's +/// `build_stock_update_expr` and so cannot be provoked through a model. +#[test] +fn stock_update_shape_guard_rejects_only_non_binary_updates() { + let level = VarRef::base(Ident::new("level")); + let var = Expr::Var(level.clone(), Loc::default()); + let c = || Expr::Const(1.0, Loc::default()); + let op2 = |op| Expr::Op2(op, Box::new(var.clone()), Box::new(c()), Loc::default()); + + // The real shape: `curr + net * dt`. + check_stock_updates_are_emittable( + &[Expr::AssignNext( + level.clone(), + Box::new(op2(BinaryOp::Add)), + )], + "level", + ) + .expect("the `Op2(Add, ..)` `build_stock_update_expr` produces must be accepted"); + + // A non-`Op2` update -- the shape a `MAX(0, ..)` `non_negative` clamp + // (GH #545) would produce. + let err = check_stock_updates_are_emittable( + &[Expr::AssignNext( + level.clone(), + Box::new(Expr::App( + crate::builtins::BuiltinFn::Max(Box::new(c()), Some(Box::new(var.clone()))), + Loc::default(), + )), + )], + "level", + ) + .expect_err("a builtin-wrapped stock update must be rejected, not silently dropped"); + assert_eq!(err.code, ErrorCode::NotSimulatable); + assert!( + err.details.as_deref().unwrap_or("").contains("level"), + "the error must name the stock so the diagnostic is attributable, got {err:?}" + ); + + // `Neq` is the one binary operator codegen does not leave as a trailing + // `Op2` (it emits `Op2 Eq` then `Not`), so it is rejected too. + assert!( + check_stock_updates_are_emittable( + &[Expr::AssignNext( + level.clone(), + Box::new(op2(BinaryOp::Neq)) + )], + "level" + ) + .is_err(), + "a `Neq` update does not end in an Op2 opcode and must be rejected" + ); + + // Nothing else in an expression list is inspected. + check_stock_updates_are_emittable(&[Expr::AssignCurr(level, Box::new(c()))], "aux") + .expect("a non-stock assignment is not a stock update"); +} + impl Var { pub(crate) fn new(ctx: &Context, var: &Variable) -> Result { // if this variable is overriden by a module input, our expression is easy - let ast: Vec = if let Some((off, _ident)) = ctx + let ast: Vec = if let Some((input_idx, _ident)) = ctx .inputs .iter() .enumerate() .find(|(_i, n)| n.as_str() == var.ident()) { vec![Expr::AssignCurr( - ctx.get_offset(&Ident::new(var.ident()))?, - Box::new(Expr::ModuleInput(off, Loc::default())), + ctx.get_ref(&Ident::new(var.ident()))?, + Box::new(Expr::ModuleInput(input_idx, Loc::default())), )] } else { match var { @@ -698,7 +828,7 @@ impl Var { inputs.iter().map(|mi| mi.dst.clone()).collect(); let inputs: Vec = inputs .into_iter() - .map(|mi| Ok(Expr::Var(ctx.get_offset(&mi.src)?, Loc::default()))) + .map(|mi| Ok(Expr::Var(ctx.get_ref(&mi.src)?, Loc::default()))) .collect::>>()?; vec![Expr::EvalModule( ident.clone(), @@ -708,7 +838,7 @@ impl Var { )] } Variable::Stock { init_ast: ast, .. } => { - let off = ctx.get_base_offset(&Ident::new(var.ident()))?; + let base = ctx.get_base_ref(&Ident::new(var.ident()))?; if ctx.is_initial { if ast.is_none() { return sim_err!(EmptyEquation, var.ident().to_string()); @@ -719,11 +849,11 @@ impl Var { let main_expr = exprs.pop().unwrap(); let main_expr = hoist_nested_array_builtins_in_scalar(main_expr, &mut exprs); - exprs.push(Expr::AssignCurr(off, Box::new(main_expr))); + exprs.push(Expr::AssignCurr(base, Box::new(main_expr))); exprs } Ast::ApplyToAll(dims, ast) => { - expand_a2a_with_hoisting(ctx, dims, ast, off)? + expand_a2a_with_hoisting(ctx, dims, ast, &base)? } Ast::Arrayed( dims, @@ -736,7 +866,7 @@ impl Var { elements, default_ast.as_ref(), *apply_default_for_missing, - off, + &base, )?, } } else { @@ -745,8 +875,8 @@ impl Var { }; match ast { Ast::Scalar(_) => vec![Expr::AssignNext( - off, - Box::new(ctx.build_stock_update_expr(off, var)?), + base.clone(), + Box::new(ctx.build_stock_update_expr(&base, var)?), )], Ast::ApplyToAll(dims, _) | Ast::Arrayed(dims, _, _, _) => { let active_dims = Arc::<[Dimension]>::from(dims.clone()); @@ -758,10 +888,13 @@ impl Var { &subscripts, ); let update_expr = ctx.build_stock_update_expr( - ctx.get_offset(&Ident::new(var.ident()))?, + &ctx.get_ref(&Ident::new(var.ident()))?, var, )?; - Ok(Expr::AssignNext(off + i, Box::new(update_expr))) + Ok(Expr::AssignNext( + base.offset_by(i), + Box::new(update_expr), + )) }) .collect(); exprs? @@ -787,7 +920,7 @@ impl Var { ast: vec![], }); } - let off = ctx.get_base_offset(&Ident::new(var.ident()))?; + let base = ctx.get_base_ref(&Ident::new(var.ident()))?; let ast = if ctx.is_initial { var.init_ast() } else { @@ -802,11 +935,11 @@ impl Var { let main_expr = exprs.pop().unwrap(); let main_expr = hoist_nested_array_builtins_in_scalar(main_expr, &mut exprs); - exprs.push(Expr::AssignCurr(off, Box::new(main_expr))); + exprs.push(Expr::AssignCurr(base.clone(), Box::new(main_expr))); exprs } Ast::ApplyToAll(dims, ast) => { - expand_a2a_with_hoisting(ctx, dims, ast, off)? + expand_a2a_with_hoisting(ctx, dims, ast, &base)? } Ast::Arrayed(dims, elements, default_ast, apply_default_for_missing) => { expand_arrayed_with_hoisting( @@ -815,7 +948,7 @@ impl Var { elements, default_ast.as_ref(), *apply_default_for_missing, - off, + &base, )? } }; @@ -825,7 +958,7 @@ impl Var { // shapes (GH #909). (A standalone lookup-only table has no // input and is handled above; ordinary auxes have no // tables.) - apply_implicit_with_lookup(exprs, off, tables) + apply_implicit_with_lookup(exprs, &base, tables) } } }; @@ -835,7 +968,8 @@ impl Var { // single chokepoint every lowering path (monolithic Module::compile // and the salsa per-variable fragment path) funnels through -- so both // backends (VM and wasmgen) see the folded form. - let ast = ast.into_iter().map(fold::fold_constants).collect(); + let ast: Vec = ast.into_iter().map(fold::fold_constants).collect(); + check_stock_updates_are_emittable(&ast, var.ident())?; Ok(Var { ident: Ident::new(var.ident()), ast, @@ -843,6 +977,60 @@ impl Var { } } +/// Reject a stock update codegen could not emit, with a structured +/// per-variable error rather than an unattributed batch failure. +/// +/// `Expr::AssignNext` is the only thing that ever writes `next[]`, and it is +/// synthesized here from `Context::build_stock_update_expr`, never by a user +/// equation. Codegen has no un-fused next-assign opcode: it emits +/// `BinOpAssignNext`, which requires the update expression's *last* emitted +/// opcode to be an `Op2` +/// (`symbolic::SymbolicByteCodeBuilder::fuse_trailing_op2_into_assign_next`). +/// Today that always holds -- `build_stock_update_expr` returns +/// `Op2(Add, curr, net * dt)` and constant folding cannot collapse it, since +/// its left operand is an `Expr::Var` -- so this check never fires. +/// +/// It exists because the property is a *shape* invariant that a future change +/// could quietly break: implementing `non_negative` (GH #545) by wrapping the +/// update in `MAX(0, ..)` would end the walk in an `Apply`, which codegen +/// cannot emit as a next-value assignment. +/// +/// What the guard buys is ATTRIBUTION, not the difference between failing and +/// not failing. Without it the build still fails, but late and vaguely: +/// codegen's error is swallowed by `compile_phase_to_per_var_bytecodes`'s +/// loud-safe `None`, the stock's phase goes missing, and `assemble_module`'s +/// `missing_vars` check reports the whole batch as +/// `failed to compile fragments for variables: ` with no reason and no +/// per-variable diagnostic. Checked here, at the chokepoint every lowering +/// path funnels through, the same failure instead surfaces as a typed +/// `NotSimulatable` `Diagnostic` attributed to the stock (through +/// `compile_var_fragment`'s `accumulate_var_compile_error`), which is what +/// invariant 7 asks for. Note that the `details` string below does not reach +/// the user today -- see the comment on `accumulate_var_compile_error` for why +/// and what the real fix is. +/// +/// `Neq` is excluded along with the non-`Op2` shapes: it is the one binary +/// operator codegen does not emit as a trailing `Op2` (it emits `Op2 Eq` then +/// `Not`). +fn check_stock_updates_are_emittable(ast: &[Expr], ident: &str) -> Result<()> { + for expr in ast { + let Expr::AssignNext(_, rhs) = expr else { + continue; + }; + let emittable = matches!(rhs.as_ref(), Expr::Op2(op, _, _, _) if *op != BinaryOp::Neq); + if !emittable { + return sim_err!( + NotSimulatable, + format!( + "the dt update for stock '{ident}' is not a binary operation, \ + so it cannot be emitted as a next-value assignment" + ) + ); + } + } + Ok(()) +} + /// Implicit WITH LOOKUP application (GH #909): rewrite each per-element /// assignment of a tables-bearing, value-bearing variable so the element's /// input equation is fed through the variable's graphical function, matching @@ -874,15 +1062,15 @@ impl Var { /// /// Only the per-element `AssignCurr` nodes the expansion paths emit are /// rewritten; hoisted pre-computations (`AssignTemp`) feed those assignments -/// and stay untouched. The table reference is `Expr::Var(base + table_idx)`: -/// codegen's `extract_table_info` resolves it by offset-range containment to -/// `(table_ident, element_offset)` -- exactly the resolution an explicit -/// `LOOKUP(var[elem], x)` call site gets -- and emits the same scalar `Lookup` -/// opcode (`graphical_functions[base_gf + element_offset]`), so both the VM -/// and the wasm backend evaluate it with no new lowering. +/// and stay untouched. The table reference is +/// `Expr::Var(base.offset_by(table_idx))`: codegen's `extract_table_info` reads +/// the owning variable's name straight off the reference -- exactly the +/// resolution an explicit `LOOKUP(var[elem], x)` call site gets -- and emits the +/// same scalar `Lookup` opcode (`graphical_functions[base_gf + element_offset]`), +/// so both the VM and the wasm backend evaluate it with no new lowering. fn apply_implicit_with_lookup( exprs: Vec, - base_off: usize, + base: &VarRef, tables: &[crate::variable::Table], ) -> Vec { if tables.is_empty() { @@ -892,12 +1080,15 @@ fn apply_implicit_with_lookup( .into_iter() .map(|expr| match expr { // The guard filters nothing in practice: every `AssignCurr` a - // Var fragment emits targets the variable's own offset range + // Var fragment emits targets this variable at or after its base // (pre-computations are `AssignTemp`). It exists purely as - // defense so a hypothetical foreign-offset assignment could - // never wrap against a nonsense element index. - Expr::AssignCurr(off, value) if off >= base_off => { - let elem = off - base_off; + // defense so a hypothetical assignment to a *different* variable, + // or one before the base, could never wrap against a nonsense + // element index. + Expr::AssignCurr(dst, value) + if dst.name == base.name && dst.element_offset >= base.element_offset => + { + let elem = dst.element_offset - base.element_offset; let table_idx = if tables.len() == 1 { 0 } else { elem }; // `false` for an out-of-range index (defensive: `tables` is // either 1 or the element count) and for an empty placeholder. @@ -908,10 +1099,10 @@ fn apply_implicit_with_lookup( if has_table { let loc = value.get_loc(); Expr::AssignCurr( - off, + dst, Box::new(Expr::App( BuiltinFn::Lookup( - Box::new(Expr::Var(base_off + table_idx, loc)), + Box::new(Expr::Var(base.offset_by(table_idx), loc)), value, loc, ), @@ -919,7 +1110,7 @@ fn apply_implicit_with_lookup( )), ) } else { - Expr::AssignCurr(off, value) + Expr::AssignCurr(dst, value) } } other => other, @@ -1205,6 +1396,10 @@ pub(crate) fn exprs_contain_array_producing_builtin(exprs: &[Expr]) -> bool { mod exprs_contain_array_producing_builtin_tests { use super::*; + fn vr(name: &str) -> VarRef { + VarRef::base(Ident::new(name)) + } + fn vem() -> Expr { // A minimal array-producing builtin call (args are irrelevant to // the predicate; only the `BuiltinFn` discriminant matters). @@ -1222,7 +1417,7 @@ mod exprs_contain_array_producing_builtin_tests { // The scalar-lowering shape `AssignCurr(off, VECTOR ELM MAP(...))` // -- the top-level case the scalar path does NOT hoist: // `contains_ ⊇ is_` catches the top-level `App`. - let exprs = vec![Expr::AssignCurr(0, Box::new(vem()))]; + let exprs = vec![Expr::AssignCurr(vr("dst"), Box::new(vem()))]; assert!(exprs_contain_array_producing_builtin(&exprs)); } @@ -1234,7 +1429,7 @@ mod exprs_contain_array_producing_builtin_tests { // `AssignTemp` recursion must still flag it. let exprs = vec![ Expr::AssignCurr( - 0, + vr("dst"), Box::new(Expr::TempArray( 0, ArrayView::contiguous(vec![1]), @@ -1249,8 +1444,8 @@ mod exprs_contain_array_producing_builtin_tests { #[test] fn does_not_flag_plain_exprs() { let exprs = vec![ - Expr::AssignCurr(0, Box::new(Expr::Const(1.0, Loc::default()))), - Expr::AssignCurr(1, Box::new(Expr::Var(0, Loc::default()))), + Expr::AssignCurr(vr("a"), Box::new(Expr::Const(1.0, Loc::default()))), + Expr::AssignCurr(vr("b"), Box::new(Expr::Var(vr("a"), Loc::default()))), ]; assert!(!exprs_contain_array_producing_builtin(&exprs)); } @@ -1713,7 +1908,7 @@ fn expand_arrayed_with_hoisting( elements: &HashMap, default_ast: Option<&crate::ast::Expr2>, apply_default_for_missing: bool, - off: usize, + base: &VarRef, ) -> Result> { let active_dims = Arc::<[Dimension]>::from(dims.to_vec()); @@ -1748,7 +1943,7 @@ fn expand_arrayed_with_hoisting( elements, default_ast, apply_default_for_missing, - off, + base, &active_dims, hoisting_ast, ) @@ -1766,12 +1961,13 @@ fn expand_arrayed_with_hoisting( let ctx = ctx.with_active_subscripts(active_dims.clone(), &subscripts); return ctx.lower(default_ast).map(|mut exprs| { let main_expr = exprs.pop().unwrap(); - exprs.push(Expr::AssignCurr(off + i, Box::new(main_expr))); + exprs + .push(Expr::AssignCurr(base.offset_by(i), Box::new(main_expr))); exprs }); } return Ok(vec![Expr::AssignCurr( - off + i, + base.offset_by(i), Box::new(Expr::Const(0.0, Loc::default())), )]); } @@ -1779,7 +1975,7 @@ fn expand_arrayed_with_hoisting( let ctx = ctx.with_active_subscripts(active_dims.clone(), &subscripts); ctx.lower(ast).map(|mut exprs| { let main_expr = exprs.pop().unwrap(); - exprs.push(Expr::AssignCurr(off + i, Box::new(main_expr))); + exprs.push(Expr::AssignCurr(base.offset_by(i), Box::new(main_expr))); exprs }) }) @@ -1798,7 +1994,7 @@ fn expand_a2a_with_hoisting( ctx: &Context, dims: &[Dimension], ast: &crate::ast::Expr2, - off: usize, + base: &VarRef, ) -> Result> { let active_dims = Arc::<[Dimension]>::from(dims.to_vec()); @@ -1819,12 +2015,12 @@ fn expand_a2a_with_hoisting( // array arguments to scalars. let mut first_exprs = first_ctx.lower_preserving_dimensions(ast)?; let main_expr = first_exprs.pop().unwrap(); - return expand_a2a_hoisted(ctx, dims, ast, off, &active_dims, first_exprs, main_expr); + return expand_a2a_hoisted(ctx, dims, ast, base, &active_dims, first_exprs, main_expr); } // Not an array-producing builtin: fall back to the standard per-element loop. // We already lowered element 0, so start from there. - first_exprs.push(Expr::AssignCurr(off, Box::new(main_expr))); + first_exprs.push(Expr::AssignCurr(base.clone(), Box::new(main_expr))); let rest: Result>> = SubscriptIterator::new(dims) .enumerate() .skip(1) @@ -1832,7 +2028,7 @@ fn expand_a2a_with_hoisting( let ctx = ctx.with_active_subscripts(active_dims.clone(), &subscripts); ctx.lower(ast).map(|mut exprs| { let main_expr = exprs.pop().unwrap(); - exprs.push(Expr::AssignCurr(off + i, Box::new(main_expr))); + exprs.push(Expr::AssignCurr(base.offset_by(i), Box::new(main_expr))); exprs }) }) @@ -1874,7 +2070,7 @@ fn expand_a2a_hoisted( ctx: &Context, dims: &[Dimension], ast: &crate::ast::Expr2, - off: usize, + base: &VarRef, active_dims: &Arc<[Dimension]>, first_exprs: Vec, main_expr: Expr, @@ -1888,7 +2084,7 @@ fn expand_a2a_hoisted( ctx, dims, ast, - off, + base, active_dims, first_exprs, main_expr, @@ -1910,7 +2106,7 @@ fn expand_a2a_hoisted( for i in 0..total_elements { let temp_idx = project_var_index_to_temp(i, &var_view, &builtin_view); result.push(Expr::AssignCurr( - off + i, + base.offset_by(i), Box::new(Expr::TempArrayElement( temp_id, builtin_view.clone(), @@ -1975,7 +2171,10 @@ fn expand_a2a_hoisted( NestedBuiltinArgMode::ScalarElement, ); result.extend(hoisted); - result.push(Expr::AssignCurr(off + i, Box::new(elem_rewritten))); + result.push(Expr::AssignCurr( + base.offset_by(i), + Box::new(elem_rewritten), + )); } } else { // Scalar args are constant: hoist once from element 0, then @@ -1992,7 +2191,7 @@ fn expand_a2a_hoisted( NestedBuiltinArgMode::ScalarElement, ); result.extend(hoisted); - result.push(Expr::AssignCurr(off, Box::new(rewritten))); + result.push(Expr::AssignCurr(base.clone(), Box::new(rewritten))); for (i, subscripts) in SubscriptIterator::new(dims).enumerate().skip(1) { let elem_ctx = ctx.with_active_subscripts(active_dims.clone(), &subscripts); @@ -2010,7 +2209,10 @@ fn expand_a2a_hoisted( false, NestedBuiltinArgMode::ScalarElement, ); - result.push(Expr::AssignCurr(off + i, Box::new(elem_rewritten))); + result.push(Expr::AssignCurr( + base.offset_by(i), + Box::new(elem_rewritten), + )); } } Ok(result) @@ -2043,8 +2245,8 @@ fn try_fold_scalar_source_elm_map(ctx: &Context, main_expr: &Expr) -> Option (*off, view.offset), + let (base, elem_flat) = match source.as_ref() { + Expr::StaticSubscript(base, view, _) if view.dims.is_empty() => (base, view.offset), _ => return None, }; // The per-element offset must be a compile-time constant (it is not a view, @@ -2052,13 +2254,13 @@ fn try_fold_scalar_source_elm_map(ctx: &Context, main_expr: &Expr) -> Option= full_len as i64 { Some(Expr::Const(f64::NAN, *loc)) } else { - Some(Expr::Var(base_off + flat as usize, *loc)) + Some(Expr::Var(base.offset_by(flat as usize), *loc)) } } @@ -2102,7 +2304,7 @@ fn expand_a2a_per_element_hoisted( ctx: &Context, dims: &[Dimension], ast: &crate::ast::Expr2, - off: usize, + base: &VarRef, active_dims: &Arc<[Dimension]>, first_exprs: Vec, first_main_expr: Expr, @@ -2153,7 +2355,7 @@ fn expand_a2a_per_element_hoisted( let main_expr = fold_scalar_source_elm_maps(ctx, main_expr); if !is_array_producing_builtin(&main_expr) && !contains_array_producing_builtin(&main_expr) { - result.push(Expr::AssignCurr(off + i, Box::new(main_expr))); + result.push(Expr::AssignCurr(base.offset_by(i), Box::new(main_expr))); continue; } @@ -2169,7 +2371,7 @@ fn expand_a2a_per_element_hoisted( builtin_view.clone(), )); result.push(Expr::AssignCurr( - off + i, + base.offset_by(i), Box::new(Expr::TempArrayElement(temp_id, builtin_view, temp_idx, loc)), )); } @@ -2188,7 +2390,7 @@ fn expand_arrayed_hoisted( elements: &HashMap, default_ast: Option<&crate::ast::Expr2>, apply_default_for_missing: bool, - off: usize, + base: &VarRef, active_dims: &Arc<[Dimension]>, hoisting_ast: &crate::ast::Expr2, ) -> Result> { @@ -2308,7 +2510,10 @@ fn expand_arrayed_hoisted( NestedBuiltinArgMode::ScalarElement, ); result.extend(hoisted); - result.push(Expr::AssignCurr(off + i, Box::new(elem_rewritten))); + result.push(Expr::AssignCurr( + base.offset_by(i), + Box::new(elem_rewritten), + )); continue; } @@ -2370,7 +2575,10 @@ fn expand_arrayed_hoisted( false, NestedBuiltinArgMode::ScalarElement, ); - result.push(Expr::AssignCurr(off + i, Box::new(elem_rewritten))); + result.push(Expr::AssignCurr( + base.offset_by(i), + Box::new(elem_rewritten), + )); } else if let Some(ast) = elem_ast { let elem_ctx = ctx.with_active_subscripts(active_dims.clone(), &subscripts); let mut elem_exprs = elem_ctx.lower(ast)?; @@ -2393,10 +2601,10 @@ fn expand_arrayed_hoisted( temp_id = temp_id.max(max + 1); } result.extend(elem_exprs); - result.push(Expr::AssignCurr(off + i, Box::new(elem_main))); + result.push(Expr::AssignCurr(base.offset_by(i), Box::new(elem_main))); } else { result.push(Expr::AssignCurr( - off + i, + base.offset_by(i), Box::new(Expr::Const(0.0, Loc::default())), )); } @@ -2572,35 +2780,41 @@ fn extract_temp_sizes_from_builtin(builtin: &BuiltinFn, temp_sizes_map: &mut Has } /// Per-variable initial expressions, kept alongside the flat runlist. -#[cfg_attr(feature = "debug-derive", derive(Debug))] -#[derive(Clone, PartialEq)] -pub(crate) struct VarInitial { - pub(crate) ident: Ident, - /// Sorted, deduplicated offsets extracted from AssignCurr nodes. - pub(crate) offsets: Vec, - pub(crate) ast: Vec, -} - +/// +/// A whole model, compiled as one unit. +/// +/// **`#[cfg(test)]`, and that gate is a load-bearing assertion, not tidiness.** +/// Production compilation is per-variable and incremental: every fragment +/// emitter builds a [`ModuleCtx`] and calls codegen directly (GH #964). Until +/// this change three production sites built a stand-in one-variable `Module` +/// by struct literal and deep-cloned five fields into it per phase. Gating the +/// type makes "no production `compiler::Module` literal remains" a compile +/// error rather than a claim someone has to re-audit. +/// +/// Every field here is one codegen reads; see [`ModuleCtx`], which is exactly +/// the set of things `Compiler` looks at and which [`Module::compile`] hands +/// it by reference. `runlist_initials` is the exception -- codegen compiles +/// initials per variable out of `runlist_initials_by_var`, and the flat list +/// exists only for the `get_initial_exprs` accessor. +#[cfg(test)] #[cfg_attr(feature = "debug-derive", derive(Debug))] pub struct Module { pub(crate) ident: Ident, - pub(crate) inputs: HashSet>, - pub(crate) n_slots: usize, - pub(crate) n_temps: usize, + pub(crate) inputs: BTreeSet>, pub(crate) temp_sizes: Vec, #[allow(dead_code)] pub(crate) runlist_initials: Vec, - pub(crate) runlist_initials_by_var: Vec, + pub(crate) runlist_initials_by_var: Vec, pub(crate) runlist_flows: Vec, pub(crate) runlist_stocks: Vec, - pub(crate) offsets: VariableOffsetMap, - #[allow(dead_code)] - pub(crate) runlist_order: Vec>, + /// The whole-model layout the emitted symbolic module is resolved against. + /// This is the monolithic path's analogue of the salsa `compute_layout` + /// query, and it is consulted exactly once, at the end of `compile()`. + pub(crate) layout: crate::compiler::symbolic::VariableLayout, + pub(crate) var_sizes: VarSizes, pub(crate) tables: HashMap, Vec
>, pub(crate) dimensions: Vec, pub(crate) dimensions_ctx: DimensionsContext, - #[allow(dead_code)] - pub(crate) module_refs: HashMap, ModuleKey>, } #[cfg(test)] @@ -2715,7 +2929,7 @@ pub(crate) fn build_metadata<'p>( offsets.insert( Ident::new("time"), VariableMetadata { - offset: 0, + offset: Some(0), size: 1, var: &IMPLICIT_TIME, }, @@ -2723,7 +2937,7 @@ pub(crate) fn build_metadata<'p>( offsets.insert( Ident::new("dt"), VariableMetadata { - offset: 1, + offset: Some(1), size: 1, var: &IMPLICIT_DT, }, @@ -2731,7 +2945,7 @@ pub(crate) fn build_metadata<'p>( offsets.insert( Ident::new("initial_time"), VariableMetadata { - offset: 2, + offset: Some(2), size: 1, var: &IMPLICIT_INITIAL_TIME, }, @@ -2739,7 +2953,7 @@ pub(crate) fn build_metadata<'p>( offsets.insert( Ident::new("final_time"), VariableMetadata { - offset: 3, + offset: Some(3), size: 1, var: &IMPLICIT_FINAL_TIME, }, @@ -2764,7 +2978,7 @@ pub(crate) fn build_metadata<'p>( offsets.insert( canonical_ident.clone(), VariableMetadata { - offset: i, + offset: Some(i), size, var: &model.variables[canonical_ident], }, @@ -2785,9 +2999,35 @@ fn calc_n_slots( metadata.values().map(|v| v.size).sum() } +#[cfg(test)] impl Module { + /// Borrow this whole-model unit as the emission context codegen consumes. + /// Every field is a reference into `self`; nothing is copied. + pub(crate) fn as_ctx(&self) -> ModuleCtx<'_> { + ModuleCtx { + ident: &self.ident, + inputs: &self.inputs, + temp_sizes: &self.temp_sizes, + runlist_initials_by_var: &self.runlist_initials_by_var, + runlist_flows: &self.runlist_flows, + runlist_stocks: &self.runlist_stocks, + var_sizes: &self.var_sizes, + tables: &self.tables, + dimensions: &self.dimensions, + dimensions_ctx: &self.dimensions_ctx, + } + } + + /// Emit this whole model and resolve it against its own layout. + /// + /// Codegen is address-neutral, so the monolithic path reaches concrete + /// bytecode the same way production assembly does: through + /// `resolve_module`. That is the whole reason one emitter can serve both -- + /// there is no second, offset-emitting codegen to keep in step. pub fn compile(&self) -> Result { - Compiler::new(self).compile() + let sym = self.as_ctx().compile()?; + crate::compiler::symbolic::resolve_module(&sym, &self.layout) + .map_err(|msg| Error::new(ErrorKind::Simulation, ErrorCode::NotSimulatable, Some(msg))) } } @@ -2826,26 +3066,6 @@ impl Module { }; let module_models = calc_module_model_map(project, model_name); - // Build module_refs: map from module variable ident to (model_name, input_set) - let module_refs: HashMap, ModuleKey> = model - .variables - .iter() - .filter_map(|(ident, var)| { - if let Variable::Module { - model_name: module_model_name, - inputs, - .. - } = var - { - let input_set: BTreeSet> = - inputs.iter().map(|mi| mi.dst.clone()).collect(); - Some((ident.clone(), (module_model_name.clone(), input_set))) - } else { - None - } - }) - .collect(); - let converted_dims: Vec = project .datamodel .dimensions @@ -2853,6 +3073,11 @@ impl Module { .map(Dimension::from) .collect(); + // Built once and shared by lowering (below) and emission (`compile`, + // through `as_ctx`), so the two agree about where a VECTOR ELM MAP + // source's storage ends. + let var_sizes: VarSizes = whole_variable_extents(&metadata, model_name); + let build_var = |ident: &Ident, is_initial| { Var::new( &Context::new( @@ -2861,6 +3086,7 @@ impl Module { dimensions_ctx: &project.dimensions_ctx, model_name, metadata: &metadata, + var_sizes: &var_sizes, module_models: &module_models, inputs, }, @@ -2887,34 +3113,12 @@ impl Module { .map(|ident| build_var(ident, false)) .collect::>>()?; - let mut runlist_order = Vec::with_capacity(flow_vars.len() + stock_vars.len()); - runlist_order.extend(flow_vars.iter().map(|v| v.ident.clone())); - runlist_order.extend(stock_vars.iter().map(|v| v.ident.clone())); - - // Build per-variable initials before flattening - let runlist_initials_by_var: Vec = initial_vars - .iter() - .map(|v| { - let mut offsets: Vec = v - .ast - .iter() - .filter_map(|expr| { - if let &Expr::AssignCurr(off, _) = expr { - Some(off) - } else { - None - } - }) - .collect(); - offsets.sort_unstable(); - offsets.dedup(); - VarInitial { - ident: v.ident.clone(), - offsets, - ast: v.ast.clone(), - } - }) - .collect(); + // Per-variable initials, kept alongside the flattened runlist. The + // `CompiledInitial::offsets` these used to carry are re-derived from + // the resolved bytecode's `AssignCurr` operands by `resolve_module`, + // which is the same set (every `Expr::AssignCurr` emits exactly one + // current-value write opcode for its target). + let runlist_initials_by_var: Vec = initial_vars.to_vec(); // Flatten out the variables so that we're just dealing with lists of expressions let runlist_initials: Vec = initial_vars.into_iter().flat_map(|v| v.ast).collect(); @@ -2932,8 +3136,7 @@ impl Module { } // Build temp_sizes vector, ordered by temp ID - let n_temps = temp_sizes_map.len(); - let mut temp_sizes: Vec = vec![0; n_temps]; + let mut temp_sizes: Vec = vec![0; temp_sizes_map.len()]; for (id, size) in temp_sizes_map { temp_sizes[id as usize] = size; } @@ -2957,34 +3160,28 @@ impl Module { .collect(); let tables = tables?; - let offsets = metadata - .into_iter() - .map(|(k, v)| { - ( - k, - v.iter() - .map(|(k, v)| (k.clone(), (v.offset, v.size))) - .collect(), - ) - }) - .collect(); + let model_metadata = &metadata[model_name]; + let layout = crate::compiler::symbolic::VariableLayout::from_offset_map( + &model_metadata + .iter() + .filter_map(|(k, v)| v.offset.map(|off| (k.clone(), (off, v.size)))) + .collect(), + n_slots, + ); Ok(Module { ident: model_name.clone(), - inputs: inputs.iter().cloned().collect(), - n_slots, - n_temps, + inputs: inputs.clone(), temp_sizes, runlist_initials, runlist_initials_by_var, runlist_flows, runlist_stocks, - offsets, - runlist_order, + layout, + var_sizes, tables, dimensions: converted_dims, dimensions_ctx: project.dimensions_ctx.clone(), - module_refs, }) } } @@ -2995,26 +3192,9 @@ impl Module { /// Returns all AssignCurr expressions that target offsets within this variable's range. pub fn get_flow_exprs(&self, var_name: &str) -> Vec<&Expr> { let canonical_name = Ident::new(var_name); - - // Look up the variable's offset range - let Some(model_offsets) = self.offsets.get(&self.ident) else { - return vec![]; - }; - let Some(&(base_offset, size)) = model_offsets.get(&canonical_name) else { - return vec![]; - }; - let offset_range = base_offset..base_offset + size; - - // Find all AssignCurr expressions that target offsets in this range self.runlist_flows .iter() - .filter(|expr| { - if let Expr::AssignCurr(off, _) = expr { - offset_range.contains(off) - } else { - false - } - }) + .filter(|expr| matches!(expr, Expr::AssignCurr(dst, _) if dst.name == canonical_name)) .collect() } @@ -3022,26 +3202,9 @@ impl Module { /// Returns all AssignCurr expressions in the initials runlist for this variable. pub fn get_initial_exprs(&self, var_name: &str) -> Vec<&Expr> { let canonical_name = Ident::new(var_name); - - // Look up the variable's offset range - let Some(model_offsets) = self.offsets.get(&self.ident) else { - return vec![]; - }; - let Some(&(base_offset, size)) = model_offsets.get(&canonical_name) else { - return vec![]; - }; - let offset_range = base_offset..base_offset + size; - - // Find all AssignCurr expressions that target offsets in this range self.runlist_initials .iter() - .filter(|expr| { - if let Expr::AssignCurr(off, _) = expr { - offset_range.contains(off) - } else { - false - } - }) + .filter(|expr| matches!(expr, Expr::AssignCurr(dst, _) if dst.name == canonical_name)) .collect() } } diff --git a/src/simlin-engine/src/compiler/pretty.rs b/src/simlin-engine/src/compiler/pretty.rs index 59873609a..4f7cd024c 100644 --- a/src/simlin-engine/src/compiler/pretty.rs +++ b/src/simlin-engine/src/compiler/pretty.rs @@ -71,12 +71,12 @@ fn pretty_subscript_index(idx: &SubscriptIndex) -> String { pub fn pretty(expr: &Expr) -> String { match expr { Expr::Const(n, _) => format!("{n}"), - Expr::Var(off, _) => format!("curr[{off}]"), - Expr::StaticSubscript(off, view, _) => { + Expr::Var(var, _) => format!("curr[{var}]"), + Expr::StaticSubscript(base, view, _) => { let dims: Vec<_> = view.dims.iter().map(|d| format!("{d}")).collect(); let strides: Vec<_> = view.strides.iter().map(|s| format!("{s}")).collect(); format!( - "curr[{off} + view(dims: [{}], strides: [{}], offset: {})]", + "curr[{base} + view(dims: [{}], strides: [{}], offset: {})]", dims.join(", "), strides.join(", "), view.offset @@ -96,12 +96,12 @@ pub fn pretty(expr: &Expr) -> String { let dims: Vec<_> = view.dims.iter().map(|d| format!("{d}")).collect(); format!("temp[{id}][{idx}] (dims: [{}])", dims.join(", ")) } - Expr::Subscript(off, args, bounds, _) => { + Expr::Subscript(base, args, bounds, _) => { let args: Vec<_> = args.iter().map(pretty_subscript_index).collect(); let string_args = args.join(", "); let bounds: Vec<_> = bounds.iter().map(|bounds| format!("{bounds}")).collect(); let string_bounds = bounds.join(", "); - format!("curr[{off} + (({string_args}) - 1); bounds: {string_bounds}]") + format!("curr[{base} + (({string_args}) - 1); bounds: {string_bounds}]") } Expr::Dt(_) => "dt".to_string(), Expr::App(builtin, _) => match builtin { @@ -266,8 +266,8 @@ pub fn pretty(expr: &Expr) -> String { Expr::If(cond, l, r, _) => { format!("if {} then {} else {}", pretty(cond), pretty(l), pretty(r)) } - Expr::AssignCurr(off, rhs) => format!("curr[{}] := {}", off, pretty(rhs)), - Expr::AssignNext(off, rhs) => format!("next[{}] := {}", off, pretty(rhs)), + Expr::AssignCurr(dst, rhs) => format!("curr[{}] := {}", dst, pretty(rhs)), + Expr::AssignNext(dst, rhs) => format!("next[{}] := {}", dst, pretty(rhs)), Expr::AssignTemp(id, expr, view) => { let dims: Vec<_> = view.dims.iter().map(|d| format!("{d}")).collect(); format!("temp[{id}][{}] <- {}", dims.join(", "), pretty(expr)) diff --git a/src/simlin-engine/src/compiler/symbolic.rs b/src/simlin-engine/src/compiler/symbolic.rs index 4d09457cc..34c4b6998 100644 --- a/src/simlin-engine/src/compiler/symbolic.rs +++ b/src/simlin-engine/src/compiler/symbolic.rs @@ -2,15 +2,26 @@ // Use of this source code is governed by the Apache License, // Version 2.0, that can be found in the LICENSE file. -//! Symbolic bytecode layer for layout-independent compilation. +//! Symbolic bytecode: the layout-independent form the compiler emits, and the +//! single late pass that turns it concrete. //! -//! The existing compiler produces bytecodes with integer offsets that depend on -//! the model's variable layout. This module introduces a symbolic representation -//! where opcodes reference variables by name instead of offset. This decouples -//! per-variable compilation from the global layout, enabling salsa to cache -//! compiled fragments that remain valid even when variables are added or removed. +//! Codegen emits `SymbolicOpcode`s whose variable operands are +//! [`crate::compiler::VarRef`]s -- a canonical variable name plus an element +//! offset -- so a compiled fragment says nothing about where its variables live. +//! That is what lets salsa cache one `PerVarBytecodes` per variable and reuse it +//! across variable additions, removals and renames, and what lets one cache +//! entry serve both the diagnostic pass and assembly. //! -//! The pipeline is: concrete bytecodes -> symbolize -> SymbolicByteCode -> resolve -> concrete bytecodes. +//! The pipeline is: lowered `Expr` (names) -> codegen -> `SymbolicByteCode` -> +//! `resolve` -> concrete bytecode. Addresses travel in exactly one direction and +//! are assigned exactly once, at assembly, against the model's final +//! `VariableLayout`. +//! +//! The peephole optimizer and the literal pool live on this side too +//! ([`SymbolicByteCodeBuilder`]): they are address-independent, and running them +//! before resolution keeps `resolve_bytecode` a strict 1:1 mapping, which is +//! what several downstream invariants rely on (the run-invariant prefix +//! boundary, the SCC segment boundaries). // These types and functions are used by the incremental compilation pipeline. @@ -22,30 +33,28 @@ use smallvec::SmallVec; use crate::bytecode::{ BuiltinId, ByteCode, ByteCodeContext, CompiledInitial, CompiledModule, DimId, DimListId, GraphicalFunctionId, LiteralId, LookupMode, ModuleDeclaration, ModuleId, ModuleInputOffset, - Op2, Opcode, PcOffset, RuntimeSparseMapping, StaticArrayView, TempId, VariableOffset, ViewId, + Op2, Opcode, PcOffset, RuntimeSparseMapping, STACK_CAPACITY, StaticArrayView, TempId, + VariableOffset, ViewId, }; use crate::common::{Canonical, Ident}; +pub(crate) use super::expr::VarRef as SymVarRef; + // ============================================================================ // Types // ============================================================================ -/// Symbolic reference to a variable location within a model. -/// Replaces raw integer offsets with a variable name and element offset, -/// making the reference independent of the model's variable layout. -#[derive(Clone, Debug, PartialEq, Eq, Hash)] -pub(crate) struct SymVarRef { - /// Canonical variable name - pub name: String, - /// Offset within the variable (0 for scalars, 0..size for array elements) - pub element_offset: usize, -} - /// Symbolic version of `Opcode`. Identical structure except opcodes that /// reference model variable offsets use `SymVarRef` instead of `VariableOffset`. /// /// Opcodes that reference global implicit variables (time, dt, etc.) keep their /// fixed offsets since those never change. +/// +/// The 3-address fused opcodes (`BinVarVar`, `AssignAddVarVarCurr`, ...) have no +/// counterpart here, and that absence is structural rather than an oversight: +/// `ByteCode::fuse_three_address` runs at `Vm::new`, on the VM's private copy of +/// already-resolved bytecode, so a fused opcode can never exist in the symbolic +/// domain. #[derive(Clone, Debug, PartialEq)] pub(crate) enum SymbolicOpcode { // === ARITHMETIC & LOGIC (unchanged) === @@ -99,9 +108,6 @@ pub(crate) enum SymbolicOpcode { AssignCurr { var: SymVarRef, }, - AssignNext { - var: SymVarRef, - }, // === BUILTINS & LOOKUPS (unchanged) === Apply { @@ -128,10 +134,33 @@ pub(crate) enum SymbolicOpcode { }, // === ARRAY VIEW STACK === - PushVarView { - var: SymVarRef, - dim_list_id: DimListId, - }, + // ── Opcodes codegen does not emit ──────────────────────────────────── + // + // The sixteen variants carrying `#[allow(dead_code)]` in this enum are ones + // `Compiler` never constructs, which -- now that codegen emits symbolic + // opcodes directly -- the dead-code lint proves rather than merely + // suggests: `Compiler` is the ONLY producer of a `SymbolicOpcode`, and + // `resolve_opcode` is the only producer of an `Opcode`, so a symbolic + // variant nothing constructs is a program the compiler cannot express. + // They are the superseded halves of two schemes: incremental view-stack + // construction (`ViewSubscriptConst`/`ViewRange`/`ViewStarRange`/ + // `ViewWildcard`/`ViewTranspose`/`DupView`/`PushTempView`/ + // `LoadTempDynamic`/`LoadIter{Element,TempElement,ViewTop}`), which + // precomputed `PushStaticView` views replaced, and broadcast iteration + // (`*BroadcastIter*`/`*BroadcastElement`/`NextBroadcastOrJump`), which + // `BeginIter` replaced. + // + // They are NOT deleted here, deliberately. Each also has an `Opcode` + // twin with a VM execution arm and a wasm-backend lowering arm (~149 + // sites plus their tests), so removing them is a change to the VM + // instruction set and the wasm parity harness -- a different subsystem, + // with a different risk profile, that would swamp this change's review + // surface. The broadcast family in particular reads as a half-landed + // feature, so retiring it is a product call rather than a cleanup. + // Sequenced as its own change; the evidence it needs is exactly this + // lint, so it does not need the empirical probe GH #964's earlier + // opcode deletions did. + #[allow(dead_code)] PushTempView { temp_id: TempId, dim_list_id: DimListId, @@ -143,6 +172,7 @@ pub(crate) enum SymbolicOpcode { var: SymVarRef, dim_list_id: DimListId, }, + #[allow(dead_code)] ViewSubscriptConst { dim_idx: u8, index: u16, @@ -150,6 +180,7 @@ pub(crate) enum SymbolicOpcode { ViewSubscriptDynamic { dim_idx: u8, }, + #[allow(dead_code)] ViewRange { dim_idx: u8, start: u16, @@ -158,15 +189,19 @@ pub(crate) enum SymbolicOpcode { ViewRangeDynamic { dim_idx: u8, }, + #[allow(dead_code)] ViewStarRange { dim_idx: u8, subdim_relation_id: u16, }, + #[allow(dead_code)] ViewWildcard { dim_idx: u8, }, + #[allow(dead_code)] ViewTranspose {}, PopView {}, + #[allow(dead_code)] DupView {}, // === TEMP ARRAY ACCESS (unchanged) === @@ -174,6 +209,7 @@ pub(crate) enum SymbolicOpcode { temp_id: TempId, index: u16, }, + #[allow(dead_code)] LoadTempDynamic { temp_id: TempId, }, @@ -183,10 +219,13 @@ pub(crate) enum SymbolicOpcode { write_temp_id: TempId, has_write_temp: bool, }, + #[allow(dead_code)] LoadIterElement {}, + #[allow(dead_code)] LoadIterTempElement { temp_id: TempId, }, + #[allow(dead_code)] LoadIterViewTop {}, LoadIterViewAt { offset: u8, @@ -234,23 +273,28 @@ pub(crate) enum SymbolicOpcode { }, // === BROADCASTING ITERATION (unchanged) === + #[allow(dead_code)] BeginBroadcastIter { n_sources: u8, dest_temp_id: TempId, }, + #[allow(dead_code)] LoadBroadcastElement { source_idx: u8, }, + #[allow(dead_code)] StoreBroadcastElement {}, + #[allow(dead_code)] NextBroadcastOrJump { jump_back: PcOffset, }, + #[allow(dead_code)] EndBroadcastIter {}, } /// Symbolic version of `ByteCode`. Contains the literal pool (unchanged) /// and symbolic opcodes. -#[derive(Clone, Debug, PartialEq)] +#[derive(Clone, Debug, Default, PartialEq)] pub(crate) struct SymbolicByteCode { pub literals: Vec, pub code: Vec, @@ -258,7 +302,7 @@ pub(crate) struct SymbolicByteCode { /// Symbolic version of `StaticArrayView`. When the view refers to a model /// variable (not a temp), `base_off` is replaced with a `SymVarRef`. -#[derive(Clone, Debug, PartialEq)] +#[derive(Clone, Debug, PartialEq, Eq, Hash)] pub(crate) struct SymbolicStaticView { pub base: SymStaticViewBase, pub dims: SmallVec<[u16; 4]>, @@ -268,7 +312,7 @@ pub(crate) struct SymbolicStaticView { pub dim_ids: SmallVec<[DimId; 4]>, } -#[derive(Clone, Debug, PartialEq)] +#[derive(Clone, Debug, PartialEq, Eq, Hash)] pub(crate) enum SymStaticViewBase { /// Model variable reference (replaces base_off when is_temp=false) Var(SymVarRef), @@ -293,10 +337,13 @@ pub(crate) struct SymbolicCompiledInitial { } /// Full symbolic representation of a `CompiledModule`. +/// +/// There is deliberately no `n_slots`: a slot count is a property of a layout, +/// and this value has none. `resolve_module` takes the slot count from the +/// layout it resolves against, which is the only place the number is meaningful. #[derive(Clone, Debug, PartialEq)] pub(crate) struct SymbolicCompiledModule { pub ident: Ident, - pub n_slots: usize, pub compiled_initials: Vec, pub compiled_flows: SymbolicByteCode, pub compiled_stocks: SymbolicByteCode, @@ -376,8 +423,13 @@ impl VariableLayout { VariableLayout { entries, n_slots } } - /// Build from a Module's offset map for a specific model. - #[allow(dead_code)] + /// Build from a model's `name -> (offset, size)` map. + /// + /// `#[cfg(test)]` because its only caller is: the test-only monolithic + /// `compiler::Module` builds its whole-model layout this way and resolves + /// its emitted symbolic module against it. Production gets the same shape + /// from the salsa `compute_layout` query instead. + #[cfg(test)] pub fn from_offset_map( offsets: &HashMap, (usize, usize)>, n_slots: usize, @@ -470,443 +522,220 @@ impl VariableLayout { } // ============================================================================ -// Reverse Offset Map (for symbolization) +// Emission: the symbolic bytecode builder // ============================================================================ -/// Maps absolute variable offsets back to (variable_name, element_within_variable). -/// Used during symbolization to convert integer offsets to symbolic references. -pub(crate) struct ReverseOffsetMap { - /// Indexed by offset. `entries[off] = Some((name, element_offset))`. - entries: Vec>, -} - -impl ReverseOffsetMap { - /// Build from a VariableLayout. - pub(crate) fn from_layout(layout: &VariableLayout) -> Self { - let mut entries: Vec> = vec![None; layout.n_slots]; - for (name, entry) in &layout.entries { - for elem in 0..entry.size { - let off = entry.offset + elem; - if off < entries.len() { - entries[off] = Some((name.clone(), elem)); - } - } +impl SymbolicOpcode { + /// The jump offset, if this opcode is a backward jump. + /// + /// Centralized here because two passes must agree on which opcodes carry a + /// PC, and a new jump opcode that failed to report itself would be + /// silently mishandled by both: the peephole optimizer would mis-relocate + /// it, and `db::assemble::segment_member_by_element` -- which REORDERS the + /// segments a jump lives between -- would not notice it escaping its + /// segment. + pub(crate) fn jump_offset(&self) -> Option { + match self { + SymbolicOpcode::NextIterOrJump { jump_back } + | SymbolicOpcode::NextBroadcastOrJump { jump_back } => Some(*jump_back), + _ => None, } - ReverseOffsetMap { entries } } - /// Look up a variable offset. - fn lookup(&self, off: u32) -> Result { - let idx = off as usize; - if idx >= self.entries.len() { - return Err(format!( - "offset {} out of range (max {})", - off, - self.entries.len() - )); - } - match &self.entries[idx] { - Some((name, elem)) => Ok(SymVarRef { - name: name.clone(), - element_offset: *elem, - }), - None => Err(format!("no variable mapped at offset {}", off)), + /// Mutably borrow the jump offset, if this opcode is a backward jump. + fn jump_offset_mut(&mut self) -> Option<&mut PcOffset> { + match self { + SymbolicOpcode::NextIterOrJump { jump_back } + | SymbolicOpcode::NextBroadcastOrJump { jump_back } => Some(jump_back), + _ => None, } } } -// ============================================================================ -// Layout Construction -// ============================================================================ - -/// Build a `VariableLayout` from the metadata produced by `build_metadata()`. +/// Accumulates one emission unit's symbolic opcodes and its literal pool. /// -/// The metadata map is `model_name -> (variable_name -> VariableMetadata)`. -/// This extracts the layout for a single model. -pub(crate) fn layout_from_metadata( - metadata: &HashMap, HashMap, super::VariableMetadata<'_>>>, - model_name: &Ident, -) -> Result { - let model_metadata = metadata.get(model_name).ok_or_else(|| { - format!( - "model '{}' not found in metadata during layout construction", - model_name.as_str() - ) - })?; - let mut entries = HashMap::with_capacity(model_metadata.len()); - let mut n_slots = 0; - - for (name, meta) in model_metadata { - entries.insert( - name.to_string(), - LayoutEntry { - offset: meta.offset, - size: meta.size, - }, - ); - n_slots = n_slots.max(meta.offset + meta.size); - } - - Ok(VariableLayout::new(entries, n_slots)) +/// This is the compiler's only bytecode builder. It runs entirely in the +/// layout-independent domain: the peephole fusions it performs +/// (`LoadConstant; AssignCurr` -> `AssignConstCurr`, `Op2; AssignCurr` -> +/// `BinOpAssignCurr`, and the emit-time `Op2` -> `BinOpAssignNext`) key on +/// opcode shape, never on an address, so fusing before resolution rather than +/// after is not an approximation -- it is the same decision made one step +/// earlier. Keeping it here is what makes `resolve_bytecode` a strict 1:1 +/// mapping, which the run-invariant flow prefix +/// (`SymbolicCompiledModule::flows_invariant_opcode_len`) and the SCC +/// per-element segmentation both depend on. +#[cfg_attr(feature = "debug-derive", derive(Debug))] +#[derive(Clone, Default)] +pub(crate) struct SymbolicByteCodeBuilder { + bytecode: SymbolicByteCode, + // keyed on the literal's bit pattern: interning only needs Eq + Hash, and + // bit-exact deduplication is the right semantic for codegen (it never + // conflates distinct values; at worst -0.0 and 0.0 get separate slots) + interned_literals: HashMap, } -// ============================================================================ -// Symbolize: Concrete -> Symbolic -// ============================================================================ - -pub(crate) fn symbolize_opcode( - op: &Opcode, - rmap: &ReverseOffsetMap, -) -> Result { - match op { - // Opcodes with variable offsets that need symbolization - Opcode::LoadVar { off } => Ok(SymbolicOpcode::LoadVar { - var: rmap.lookup(u32::from(*off))?, - }), - Opcode::LoadPrev { off } => Ok(SymbolicOpcode::SymLoadPrev { - var: rmap.lookup(u32::from(*off))?, - }), - Opcode::LoadInitial { off } => Ok(SymbolicOpcode::SymLoadInitial { - var: rmap.lookup(u32::from(*off))?, - }), - Opcode::LoadSubscript { off } => Ok(SymbolicOpcode::LoadSubscript { - var: rmap.lookup(u32::from(*off))?, - }), - Opcode::AssignCurr { off } => Ok(SymbolicOpcode::AssignCurr { - var: rmap.lookup(u32::from(*off))?, - }), - Opcode::AssignNext { off } => Ok(SymbolicOpcode::AssignNext { - var: rmap.lookup(u32::from(*off))?, - }), - Opcode::AssignConstCurr { off, literal_id } => Ok(SymbolicOpcode::AssignConstCurr { - var: rmap.lookup(u32::from(*off))?, - literal_id: *literal_id, - }), - Opcode::BinOpAssignCurr { op, off } => Ok(SymbolicOpcode::BinOpAssignCurr { - op: *op, - var: rmap.lookup(u32::from(*off))?, - }), - Opcode::BinOpAssignNext { op, off } => Ok(SymbolicOpcode::BinOpAssignNext { - op: *op, - var: rmap.lookup(u32::from(*off))?, - }), - // The 3-address fused binops AND the fused leaf assignments are created - // by `ByteCode::fuse_three_address`, which runs only on FINAL concrete - // bytecode (after `resolve`), strictly after symbolization, and only on - // the Vm's private execution copy (never the salsa-cached - // CompiledSimulation). They therefore never reach this function; seeing - // one means the fusion ran before symbolize, which is a compiler bug. - // The exhaustive match here is the guarantee no fused opcode can silently - // leak into the symbolic/incremental layer. - Opcode::BinVarVar { .. } - | Opcode::BinVarConst { .. } - | Opcode::BinConstVar { .. } - | Opcode::BinStackVar { .. } - | Opcode::BinStackConst { .. } - | Opcode::BinGlobalVar { .. } - | Opcode::BinVarGlobal { .. } - | Opcode::BinGlobalConst { .. } - | Opcode::BinConstGlobal { .. } - | Opcode::BinGlobalGlobal { .. } - | Opcode::BinStackGlobal { .. } - | Opcode::BinConstConst { .. } - | Opcode::AssignAddVarVarCurr { .. } - | Opcode::AssignSubVarVarCurr { .. } - | Opcode::AssignMulVarVarCurr { .. } - | Opcode::AssignDivVarVarCurr { .. } - | Opcode::AssignAddVarVarNext { .. } - | Opcode::AssignSubVarVarNext { .. } - | Opcode::AssignMulVarVarNext { .. } - | Opcode::AssignDivVarVarNext { .. } - | Opcode::AssignAddVarConstCurr { .. } - | Opcode::AssignSubVarConstCurr { .. } - | Opcode::AssignMulVarConstCurr { .. } - | Opcode::AssignDivVarConstCurr { .. } - | Opcode::AssignAddVarConstNext { .. } - | Opcode::AssignSubVarConstNext { .. } - | Opcode::AssignMulVarConstNext { .. } - | Opcode::AssignDivVarConstNext { .. } - | Opcode::AssignAddConstVarCurr { .. } - | Opcode::AssignSubConstVarCurr { .. } - | Opcode::AssignMulConstVarCurr { .. } - | Opcode::AssignDivConstVarCurr { .. } - | Opcode::AssignAddConstVarNext { .. } - | Opcode::AssignSubConstVarNext { .. } - | Opcode::AssignMulConstVarNext { .. } - | Opcode::AssignDivConstVarNext { .. } - | Opcode::AssignStackVarCurr { .. } - | Opcode::AssignStackVarNext { .. } - | Opcode::AssignStackConstCurr { .. } - | Opcode::AssignStackConstNext { .. } => { - unreachable!("3-address fused opcode reached symbolize_opcode") - } - Opcode::PushVarView { - base_off, - dim_list_id, - } => Ok(SymbolicOpcode::PushVarView { - var: rmap.lookup(u32::from(*base_off))?, - dim_list_id: *dim_list_id, - }), - Opcode::PushVarViewDirect { - base_off, - dim_list_id, - } => Ok(SymbolicOpcode::PushVarViewDirect { - var: rmap.lookup(u32::from(*base_off))?, - dim_list_id: *dim_list_id, - }), - - // Opcodes that are identical in symbolic form - Opcode::Op2 { op } => Ok(SymbolicOpcode::Op2 { op: *op }), - Opcode::Not {} => Ok(SymbolicOpcode::Not {}), - Opcode::LoadConstant { id } => Ok(SymbolicOpcode::LoadConstant { id: *id }), - Opcode::LoadGlobalVar { off } => Ok(SymbolicOpcode::LoadGlobalVar { off: *off }), - Opcode::PushSubscriptIndex { bounds } => { - Ok(SymbolicOpcode::PushSubscriptIndex { bounds: *bounds }) - } - Opcode::SetCond {} => Ok(SymbolicOpcode::SetCond {}), - Opcode::If {} => Ok(SymbolicOpcode::If {}), - Opcode::Ret => Ok(SymbolicOpcode::Ret), - Opcode::LoadModuleInput { input } => Ok(SymbolicOpcode::LoadModuleInput { input: *input }), - Opcode::EvalModule { id, n_inputs } => Ok(SymbolicOpcode::EvalModule { - id: *id, - n_inputs: *n_inputs, - }), - Opcode::Apply { func } => Ok(SymbolicOpcode::Apply { func: *func }), - Opcode::Lookup { - base_gf, - table_count, - mode, - } => Ok(SymbolicOpcode::Lookup { - base_gf: *base_gf, - table_count: *table_count, - mode: *mode, - }), - Opcode::PushTempView { - temp_id, - dim_list_id, - } => Ok(SymbolicOpcode::PushTempView { - temp_id: *temp_id, - dim_list_id: *dim_list_id, - }), - Opcode::PushStaticView { view_id } => { - Ok(SymbolicOpcode::PushStaticView { view_id: *view_id }) - } - Opcode::ViewSubscriptConst { dim_idx, index } => Ok(SymbolicOpcode::ViewSubscriptConst { - dim_idx: *dim_idx, - index: *index, - }), - Opcode::ViewSubscriptDynamic { dim_idx } => { - Ok(SymbolicOpcode::ViewSubscriptDynamic { dim_idx: *dim_idx }) - } - Opcode::ViewRange { - dim_idx, - start, - end, - } => Ok(SymbolicOpcode::ViewRange { - dim_idx: *dim_idx, - start: *start, - end: *end, - }), - Opcode::ViewRangeDynamic { dim_idx } => { - Ok(SymbolicOpcode::ViewRangeDynamic { dim_idx: *dim_idx }) +impl SymbolicByteCodeBuilder { + pub(crate) fn intern_literal(&mut self, lit: f64) -> LiteralId { + let key = lit.to_bits(); + if self.interned_literals.contains_key(&key) { + return self.interned_literals[&key]; } - Opcode::ViewStarRange { - dim_idx, - subdim_relation_id, - } => Ok(SymbolicOpcode::ViewStarRange { - dim_idx: *dim_idx, - subdim_relation_id: *subdim_relation_id, - }), - Opcode::ViewWildcard { dim_idx } => Ok(SymbolicOpcode::ViewWildcard { dim_idx: *dim_idx }), - Opcode::ViewTranspose {} => Ok(SymbolicOpcode::ViewTranspose {}), - Opcode::PopView {} => Ok(SymbolicOpcode::PopView {}), - Opcode::DupView {} => Ok(SymbolicOpcode::DupView {}), - Opcode::LoadTempConst { temp_id, index } => Ok(SymbolicOpcode::LoadTempConst { - temp_id: *temp_id, - index: *index, - }), - Opcode::LoadTempDynamic { temp_id } => { - Ok(SymbolicOpcode::LoadTempDynamic { temp_id: *temp_id }) - } - Opcode::BeginIter { - write_temp_id, - has_write_temp, - } => Ok(SymbolicOpcode::BeginIter { - write_temp_id: *write_temp_id, - has_write_temp: *has_write_temp, - }), - Opcode::LoadIterElement {} => Ok(SymbolicOpcode::LoadIterElement {}), - Opcode::LoadIterTempElement { temp_id } => { - Ok(SymbolicOpcode::LoadIterTempElement { temp_id: *temp_id }) - } - Opcode::LoadIterViewTop {} => Ok(SymbolicOpcode::LoadIterViewTop {}), - Opcode::LoadIterViewAt { offset } => Ok(SymbolicOpcode::LoadIterViewAt { offset: *offset }), - Opcode::StoreIterElement {} => Ok(SymbolicOpcode::StoreIterElement {}), - Opcode::NextIterOrJump { jump_back } => Ok(SymbolicOpcode::NextIterOrJump { - jump_back: *jump_back, - }), - Opcode::EndIter {} => Ok(SymbolicOpcode::EndIter {}), - Opcode::ArraySum {} => Ok(SymbolicOpcode::ArraySum {}), - Opcode::ArrayMax {} => Ok(SymbolicOpcode::ArrayMax {}), - Opcode::ArrayMin {} => Ok(SymbolicOpcode::ArrayMin {}), - Opcode::ArrayMean {} => Ok(SymbolicOpcode::ArrayMean {}), - Opcode::ArrayStddev {} => Ok(SymbolicOpcode::ArrayStddev {}), - Opcode::ArraySize {} => Ok(SymbolicOpcode::ArraySize {}), - Opcode::VectorSelect {} => Ok(SymbolicOpcode::VectorSelect {}), - Opcode::VectorElmMap { - write_temp_id, - full_source_len, - } => Ok(SymbolicOpcode::VectorElmMap { - write_temp_id: *write_temp_id, - full_source_len: *full_source_len, - }), - Opcode::VectorSortOrder { write_temp_id } => Ok(SymbolicOpcode::VectorSortOrder { - write_temp_id: *write_temp_id, - }), - Opcode::Rank { write_temp_id } => Ok(SymbolicOpcode::Rank { - write_temp_id: *write_temp_id, - }), - Opcode::LookupArray { - base_gf, - table_count, - mode, - write_temp_id, - } => Ok(SymbolicOpcode::LookupArray { - base_gf: *base_gf, - table_count: *table_count, - mode: *mode, - write_temp_id: *write_temp_id, - }), - Opcode::AllocateAvailable { write_temp_id } => Ok(SymbolicOpcode::AllocateAvailable { - write_temp_id: *write_temp_id, - }), - Opcode::AllocateByPriority { write_temp_id } => Ok(SymbolicOpcode::AllocateByPriority { - write_temp_id: *write_temp_id, - }), - Opcode::BeginBroadcastIter { - n_sources, - dest_temp_id, - } => Ok(SymbolicOpcode::BeginBroadcastIter { - n_sources: *n_sources, - dest_temp_id: *dest_temp_id, - }), - Opcode::LoadBroadcastElement { source_idx } => Ok(SymbolicOpcode::LoadBroadcastElement { - source_idx: *source_idx, - }), - Opcode::StoreBroadcastElement {} => Ok(SymbolicOpcode::StoreBroadcastElement {}), - Opcode::NextBroadcastOrJump { jump_back } => Ok(SymbolicOpcode::NextBroadcastOrJump { - jump_back: *jump_back, - }), - Opcode::EndBroadcastIter {} => Ok(SymbolicOpcode::EndBroadcastIter {}), + self.bytecode.literals.push(lit); + let literal_id = (self.bytecode.literals.len() - 1) as u16; + self.interned_literals.insert(key, literal_id); + literal_id } -} -pub(crate) fn symbolize_static_view( - view: &StaticArrayView, - rmap: &ReverseOffsetMap, -) -> Result { - let base = if view.is_temp { - SymStaticViewBase::Temp(view.base_off) - } else { - SymStaticViewBase::Var(rmap.lookup(view.base_off)?) - }; + /// Allocate a new literal slot without deduplication. + /// Used for named constants so each variable gets its own slot, + /// preventing shared-literal corruption when overriding via set_value. + pub(crate) fn push_named_literal(&mut self, lit: f64) -> LiteralId { + self.bytecode.literals.push(lit); + (self.bytecode.literals.len() - 1) as u16 + } - Ok(SymbolicStaticView { - base, - dims: view.dims.clone(), - strides: view.strides.clone(), - offset: view.offset, - sparse: view.sparse.clone(), - dim_ids: view.dim_ids.clone(), - }) -} + pub(crate) fn push_opcode(&mut self, op: SymbolicOpcode) { + self.bytecode.code.push(op) + } -pub(crate) fn symbolize_module_decl( - decl: &ModuleDeclaration, - rmap: &ReverseOffsetMap, -) -> Result { - let off = u32::try_from(decl.off) - .map_err(|_| format!("module declaration offset {} does not fit in u32", decl.off))?; - Ok(SymbolicModuleDecl { - model_name: decl.model_name.clone(), - input_set: decl.input_set.clone(), - var: rmap.lookup(off)?, - }) -} + /// Replace a just-emitted trailing `Op2` with the fused + /// `BinOpAssignNext { op, var }` that both computes it and stores the + /// result into `next[var]`. Returns `false` -- emitting nothing -- when the + /// stream does not end in an `Op2`. + /// + /// There is no un-fused `SymbolicOpcode::AssignNext`: a stock update is the + /// only thing that ever writes `next[]`, and `Context::build_stock_update_expr` + /// always returns `Op2(Add, curr_value, net * dt)`, so the operand walk + /// always ends in an `Op2` and the fused form is always emittable. Codegen + /// therefore fuses here, at emit time, and reports a typed error if a stock + /// update ever arrives in a shape that would need the un-fused opcode -- + /// most plausibly an eventual `non_negative` implementation (GH #545) + /// wrapping the update in a `MAX`. A comment could not fail; this can. + /// + /// Fusing at emit time is strictly inside `peephole_optimize`'s + /// jump-target safety envelope rather than an exception to it: the `Op2` + /// being replaced is the LAST opcode in the stream, so no jump emitted so + /// far can target it (every jump is a backward `NextIterOrJump` emitted + /// after its target), and the pair it replaces has no successor whose PC + /// could shift. + pub(crate) fn fuse_trailing_op2_into_assign_next(&mut self, var: &SymVarRef) -> bool { + match self.bytecode.code.last() { + Some(SymbolicOpcode::Op2 { op }) => { + let op = *op; + let last = self.bytecode.code.len() - 1; + self.bytecode.code[last] = SymbolicOpcode::BinOpAssignNext { + op, + var: var.clone(), + }; + true + } + _ => false, + } + } -pub(crate) fn symbolize_bytecode( - bc: &ByteCode, - rmap: &ReverseOffsetMap, -) -> Result { - let code = bc - .code - .iter() - .map(|op| symbolize_opcode(op, rmap)) - .collect::, _>>()?; + /// Returns the current number of opcodes in the bytecode + pub(crate) fn len(&self) -> usize { + self.bytecode.code.len() + } - Ok(SymbolicByteCode { - literals: bc.literals.clone(), - code, - }) + pub(crate) fn finish(self) -> SymbolicByteCode { + let mut bc = self.bytecode; + bc.peephole_optimize(); + bc + } } -/// Convert a `CompiledModule` to its symbolic representation. -/// All variable offsets are replaced with symbolic references using the layout. -#[allow(dead_code)] -pub(crate) fn symbolize_module( - module: &CompiledModule, - layout: &VariableLayout, -) -> Result { - let rmap = ReverseOffsetMap::from_layout(layout); - - let compiled_initials = module - .compiled_initials - .iter() - .map(|ci| { - Ok(SymbolicCompiledInitial { - ident: ci.ident.clone(), - bytecode: symbolize_bytecode(&ci.bytecode, &rmap)?, - }) - }) - .collect::, String>>()?; +impl SymbolicByteCode { + /// Peephole optimization pass: fuse common opcode sequences into + /// superinstructions to reduce dispatch overhead. + /// + /// Only fuses adjacent instructions when neither is a jump target. + /// Jump offsets are recalculated after fusion using an old->new PC map. + fn peephole_optimize(&mut self) { + if self.code.is_empty() { + return; + } - let compiled_flows = symbolize_bytecode(&module.compiled_flows, &rmap)?; - let compiled_stocks = symbolize_bytecode(&module.compiled_stocks, &rmap)?; + // 1. Build set of PCs that are jump targets + let mut jump_targets = vec![false; self.code.len()]; + for (pc, op) in self.code.iter().enumerate() { + if let Some(offset) = op.jump_offset() { + let target = (pc as isize + offset as isize) as usize; + assert!( + target < jump_targets.len(), + "jump at pc {pc} targets {target}, which is out of bounds (code length: {})", + self.code.len() + ); + jump_targets[target] = true; + } + } - let ctx = &*module.context; + // 2. Build old_pc -> new_pc mapping and fused output. + // pc_map has one entry per original instruction so that jump fixup + // can index by the original PC directly. + let mut optimized: Vec = Vec::with_capacity(self.code.len()); + let mut pc_map: Vec = Vec::with_capacity(self.code.len() + 1); + let mut i = 0; + while i < self.code.len() { + let new_pc = optimized.len(); + pc_map.push(new_pc); + + // Only try fusion if the next instruction is not a jump target. + // We intentionally don't check whether instruction i itself is a + // jump target: the fused instruction replaces both i and i+1 at the + // same PC, so jumps to i still land on the correct (fused) opcode. + let can_fuse = i + 1 < self.code.len() && !jump_targets[i + 1]; + + if can_fuse { + let fused = match (&self.code[i], &self.code[i + 1]) { + // Pattern: LoadConstant + AssignCurr -> AssignConstCurr + (SymbolicOpcode::LoadConstant { id }, SymbolicOpcode::AssignCurr { var }) => { + Some(SymbolicOpcode::AssignConstCurr { + var: var.clone(), + literal_id: *id, + }) + } + // Pattern: Op2 + AssignCurr -> BinOpAssignCurr + (SymbolicOpcode::Op2 { op }, SymbolicOpcode::AssignCurr { var }) => { + Some(SymbolicOpcode::BinOpAssignCurr { + op: *op, + var: var.clone(), + }) + } + _ => None, + }; + + if let Some(op) = fused { + optimized.push(op); + // Both old PCs map to the same new PC + pc_map.push(new_pc); + i += 2; + continue; + } + } - let static_views = ctx - .static_views - .iter() - .map(|sv| symbolize_static_view(sv, &rmap)) - .collect::, _>>()?; + // No pattern matched - copy opcode as-is + optimized.push(self.code[i].clone()); + i += 1; + } + // Sentinel for instructions past the end + pc_map.push(optimized.len()); - let module_decls = ctx - .modules - .iter() - .map(|md| symbolize_module_decl(md, &rmap)) - .collect::, _>>()?; + // 3. Fix up jump offsets. Iterate original code to find jumps, + // then use pc_map (indexed by old_pc) for O(1) translation. + for (old_pc, op) in self.code.iter().enumerate() { + let Some(jump_back) = op.jump_offset() else { + continue; + }; + let new_pc = pc_map[old_pc]; + let old_target = (old_pc as isize + jump_back as isize) as usize; + let new_target = pc_map[old_target]; + let new_jump_back = (new_target as isize - new_pc as isize) as PcOffset; + *optimized[new_pc].jump_offset_mut().unwrap() = new_jump_back; + } - Ok(SymbolicCompiledModule { - ident: module.ident.clone(), - n_slots: module.n_slots, - compiled_initials, - compiled_flows, - compiled_stocks, - graphical_functions: ctx.graphical_functions.clone(), - module_decls, - static_views, - arrays: ctx.arrays.clone(), - dimensions: ctx.dimensions.clone(), - subdim_relations: ctx.subdim_relations.clone(), - names: ctx.names.clone(), - temp_offsets: ctx.temp_offsets.clone(), - temp_total_size: ctx.temp_total_size, - dim_lists: ctx.dim_lists.clone(), - // The symbolize<->resolve roundtrip is opcode-count-preserving, so the - // invariant-prefix boundary index is identical in both forms (GH #712). - flows_invariant_opcode_len: module.flows_invariant_opcode_len, - }) + self.code = optimized; + } } // ============================================================================ @@ -921,11 +750,9 @@ fn sym_var_refs_in_bytecode(sbc: &SymbolicByteCode) -> impl Iterator Some(var.name.as_str()), _ => None, }) @@ -962,7 +789,10 @@ pub(crate) fn fragment_vars_in_layout( ]; for maybe_decls in &phase_decls { let Some(decls) = maybe_decls else { continue }; - if decls.iter().any(|d| layout.get(&d.var.name).is_none()) { + if decls + .iter() + .any(|d| layout.get(d.var.name.as_str()).is_none()) + { return false; } } @@ -1001,7 +831,7 @@ pub(crate) fn resolve_var_ref( var: &SymVarRef, layout: &VariableLayout, ) -> Result { - let entry = layout.get(&var.name).ok_or_else(|| { + let entry = layout.get(var.name.as_str()).ok_or_else(|| { format!( "variable '{}' not found in layout during resolution", var.name @@ -1050,9 +880,6 @@ pub(crate) fn resolve_opcode( SymbolicOpcode::AssignCurr { var } => Ok(Opcode::AssignCurr { off: resolve_var_ref(var, layout)?, }), - SymbolicOpcode::AssignNext { var } => Ok(Opcode::AssignNext { - off: resolve_var_ref(var, layout)?, - }), SymbolicOpcode::AssignConstCurr { var, literal_id } => Ok(Opcode::AssignConstCurr { off: resolve_var_ref(var, layout)?, literal_id: *literal_id, @@ -1065,10 +892,6 @@ pub(crate) fn resolve_opcode( op: *op, off: resolve_var_ref(var, layout)?, }), - SymbolicOpcode::PushVarView { var, dim_list_id } => Ok(Opcode::PushVarView { - base_off: resolve_var_ref(var, layout)?, - dim_list_id: *dim_list_id, - }), SymbolicOpcode::PushVarViewDirect { var, dim_list_id } => Ok(Opcode::PushVarViewDirect { base_off: resolve_var_ref(var, layout)?, dim_list_id: *dim_list_id, @@ -1202,7 +1025,7 @@ pub(crate) fn resolve_opcode( // constant in both sites). The structural symbolic round-trip -- // `test_renumber_vector_builtin_temp_ids` (isolated `renumber_opcode`) // and `test_vector_elm_map_full_source_len_survives_fragment_roundtrip` - // (the full `symbolize` -> `concatenate_fragments` -> `resolve_bytecode` + // (the full `concatenate_fragments` -> `resolve_bytecode` // merge path) -- complements them by pinning that this field is // invariant under renumbering (it is NOT a renumber-able resource id // like temp/lit/gf/view/dim_list/module). @@ -1249,6 +1072,26 @@ pub(crate) fn resolve_opcode( } } +/// Resolve a symbolic bytecode stream against `layout`, producing the concrete +/// bytecode the VM executes. +/// +/// **This is the only place concrete bytecode is born**, and therefore the place +/// the VM's stack-safety proof is discharged: the emitted program must not be +/// able to overflow the VM's fixed-size arithmetic stack, which is what makes +/// `vm::Stack`'s unchecked accesses sound. +/// +/// The check used to live in the per-fragment `ByteCodeBuilder::finish()`. Moving +/// it here made it strictly STRONGER, not merely equivalent: a fragment starts +/// and ends at depth 0 today, so per-fragment maxima and the concatenation's +/// maximum agree -- but `concatenate_fragments` strips each fragment's trailing +/// `Ret` and appends, so a fragment that did NOT balance would accumulate depth +/// across fragment boundaries in a way the per-fragment check could not see. +/// It also runs once per assembled phase instead of once per fragment. +/// +/// Both failure modes are reported, not asserted: an over-deep program and an +/// `Opcode::stack_effect` underflow (a compiler-metadata bug) both come back as +/// `Err`, so neither can abort a `libsimlin` host process from inside a +/// `Result`-returning function. pub(crate) fn resolve_bytecode( sbc: &SymbolicByteCode, layout: &VariableLayout, @@ -1259,10 +1102,20 @@ pub(crate) fn resolve_bytecode( .map(|op| resolve_opcode(op, layout)) .collect::, _>>()?; - Ok(ByteCode { + let bc = ByteCode { literals: sbc.literals.clone(), code, - }) + }; + + let depth = bc.max_stack_depth()?; + if depth >= STACK_CAPACITY { + return Err(format!( + "compiled bytecode requires stack depth {depth}, exceeding VM capacity \ + {STACK_CAPACITY}" + )); + } + + Ok(bc) } pub(crate) fn resolve_static_view( @@ -1271,7 +1124,7 @@ pub(crate) fn resolve_static_view( ) -> Result { let (base_off, is_temp) = match &sv.base { SymStaticViewBase::Var(var_ref) => { - let entry = layout.get(&var_ref.name).ok_or_else(|| { + let entry = layout.get(var_ref.name.as_str()).ok_or_else(|| { format!( "variable '{}' not found in layout during static view resolution", var_ref.name @@ -1297,7 +1150,7 @@ pub(crate) fn resolve_module_decl( sd: &SymbolicModuleDecl, layout: &VariableLayout, ) -> Result { - let entry = layout.get(&sd.var.name).ok_or_else(|| { + let entry = layout.get(sd.var.name.as_str()).ok_or_else(|| { format!( "module variable '{}' not found in layout during resolution", sd.var.name @@ -1332,10 +1185,10 @@ pub(crate) fn resolve_module( }) .collect::, String>>()?; - // `resolve_module` is a pure symbolic<->concrete primitive (the roundtrip - // tests symbolize its output again), so the 3-address fusion (R2) is NOT - // applied here -- the production assembler `assemble_module` applies it to - // this function's output instead, where the result is never re-symbolized. + // `resolve_module` is the single symbolic -> concrete primitive, so the + // 3-address fusion (R2) is NOT applied here -- `Vm::new` applies it to its + // own private copy of the bytecode, after the salsa-cached artifact has + // been produced. let compiled_flows = resolve_bytecode(&sym.compiled_flows, layout)?; let compiled_stocks = resolve_bytecode(&sym.compiled_stocks, layout)?; @@ -1414,12 +1267,20 @@ pub(crate) struct ConcatenatedBytecodes { /// pool. Graphical functions are excluded because they are content-de- /// duplicated (#582) rather than flat-counted -- their per-fragment base /// comes from a shared `GfDedup` remap, not a running sum. +/// +/// The three `u16`-addressed counts are held as `usize` on purpose. They are +/// COUNTS, not ids: a count of exactly `U16_ID_CAPACITY` describes a full table +/// whose every id (`0..=u16::MAX`) is representable, and a phase that follows it +/// with none of that resource assigns no id at all. Narrowing them here would +/// reject that program even though M3 -- *every assigned id is representable* -- +/// holds for it; the one place that can tell is [`resource_base`], which sees +/// the following fragment's length and is where the bound therefore lives. #[derive(Clone, Debug, Default)] pub(crate) struct ContextResourceCounts { - pub modules: u16, - pub views: u16, + pub modules: usize, + pub views: usize, pub temps: u32, - pub dim_lists: u16, + pub dim_lists: usize, } impl ContextResourceCounts { @@ -1434,11 +1295,19 @@ impl ContextResourceCounts { /// rather than this per-phase sum. The field is summed here for the /// benefit of any caller that genuinely wants the disjoint per-phase temp /// count (e.g. the `Sum` strategy / `combine_scc_fragment` accounting). + /// + /// Infallible: summing counts cannot violate M3 by itself, because a base + /// only has to be representable once a fragment uses it to name something. + /// The check lives in [`resource_base`], which is handed both the base and + /// the length of the fragment about to consume it. pub fn from_fragments(fragments: &[&PerVarBytecodes]) -> Self { - let mut counts = ContextResourceCounts::default(); + let mut modules: usize = 0; + let mut views: usize = 0; + let mut temps: u32 = 0; + let mut dim_lists: usize = 0; for frag in fragments { - counts.modules += frag.module_decls.len() as u16; - counts.views += frag.static_views.len() as u16; + modules += frag.module_decls.len(); + views += frag.static_views.len(); // Each fragment's temps start at 0, so the disjoint-layout total // is the sum of each fragment's (max_id + 1), not the global max. let frag_temp_count = frag @@ -1447,37 +1316,182 @@ impl ContextResourceCounts { .map(|(id, _)| *id + 1) .max() .unwrap_or(0); - counts.temps += frag_temp_count; - counts.dim_lists += frag.dim_lists.len() as u16; + temps += frag_temp_count; + dim_lists += frag.dim_lists.len(); + } + ContextResourceCounts { + modules, + views, + temps, + dim_lists, } - counts } } -/// The five flat resource-ID base offsets a single fragment's non-GF -/// opcodes are renumbered by (the result of absorbing that fragment into a -/// `FragmentMerger`). Graphical-function IDs are NOT a flat offset -- they -/// are content-de-duplicated (#582), so they are remapped per local slot -/// via the companion `GfRemap` rather than a single base. Pass both to -/// `renumber_opcode` / `renumber_fragment_code`. -#[derive(Clone, Copy, Debug)] -pub(crate) struct FragmentResourceOffsets { - pub lit_offset: u16, - pub mod_offset: u16, - pub view_offset: u16, - pub temp_offset: u32, - pub dl_offset: u16, +/// How many entries a `u16`-addressed merged resource table can hold: ids run +/// `0..=u16::MAX`. +const U16_ID_CAPACITY: usize = u16::MAX as usize + 1; + +#[cfg(test)] +thread_local! { + static ID_CAPACITY_OVERRIDE: std::cell::Cell> = + const { std::cell::Cell::new(None) }; } -/// Per-fragment local-GF-slot -> global-(deduped)-GF-slot map. Index `i` -/// holds the `merged_gf` index that this fragment's local -/// `graphical_functions[i]` was de-duplicated to. Total over `[0, gf_len)`, -/// so a `Lookup`/`LookupArray` `base_gf` is remapped by a single -/// `gf_remap[base_gf]` lookup; the whole-list shift in -/// `FragmentMerger::absorb_gf` guarantees `gf_remap[base + k] == -/// gf_remap[base] + k`, so the array-lookup `[base .. base + table_count]` -/// contract survives the remap. -pub(crate) type GfRemap = SmallVec<[GraphicalFunctionId; 8]>; +/// The capacity [`resource_base`] bounds against: `U16_ID_CAPACITY`, except +/// under an [`IdCapacityGuard`] in a test. +/// +/// The override exists so a test can reach the bound with a tiny fixture +/// instead of one large enough to genuinely fill a 16-bit id space -- a +/// 65,537-entry model would blow the per-test time budget many times over +/// (`docs/dev/rust.md#test-time-budgets`). It does not add a second statement +/// of the bound: the comparison still happens once, in `resource_base`; only +/// the constant it compares against moves. +#[inline] +fn id_capacity() -> usize { + #[cfg(test)] + if let Some(cap) = ID_CAPACITY_OVERRIDE.with(|c| c.get()) { + return cap; + } + U16_ID_CAPACITY +} + +/// Shrink the resource-id capacity for the current thread, restoring it on +/// drop. Thread-local because tests run in parallel. +#[cfg(test)] +pub(crate) struct IdCapacityGuard; + +#[cfg(test)] +impl IdCapacityGuard { + pub(crate) fn new(capacity: usize) -> Self { + ID_CAPACITY_OVERRIDE.with(|c| c.set(Some(capacity))); + IdCapacityGuard + } +} + +#[cfg(test)] +impl Drop for IdCapacityGuard { + fn drop(&mut self) { + ID_CAPACITY_OVERRIDE.with(|c| c.set(None)); + } +} + +/// The base id for a fragment's `frag_len` entries appended to a merged table +/// that already holds `merged_len`, itself offset by `ctx_base` (M2 + M3). +/// +/// **This is the only place the `u16` capacity bound is stated.** Every base a +/// `u16`-addressed resource is renumbered against comes from here -- the +/// merger's four, and the running offsets `db::assemble::renumber_initials_phase` +/// tracks by hand -- so the boundary cannot be one value in one place and a +/// different one in another. Everything upstream counts in `usize`. +/// +/// The ids this fragment will use are `base .. base + frag_len`, so the check +/// is that the LAST of them is still representable: `base + frag_len` must not +/// exceed the table capacity. Computed in `usize` and narrowed once, so neither +/// the base nor the end can wrap on the way to the comparison. +/// +/// **`end > U16_ID_CAPACITY` is the load-bearing line, and the `>` is exact.** +/// `end` is one past the last id, so `end == U16_ID_CAPACITY` means the last id +/// is `u16::MAX` -- addressable. Tightening it to `>=` rejects a table that +/// fills the id space exactly, which is a legal program; a reader "simplifying" +/// it that way is the live risk, and `literal_pool_past_u16_capacity_fails_loud` +/// / `a_full_ctx_base_bounds_only_the_phases_that_use_it` / +/// `the_initials_phase_shares_the_mergers_capacity_bound` all red if they do. +/// +/// A fragment carrying NONE of this resource is exempt: it names no id, so +/// there is nothing to represent, and the base it is handed is never used. Note +/// what that exemption does and does not buy. It does NOT move the boundary -- +/// deleting it leaves every test green, because the check above already admits +/// a full table. What it buys is that `base as u16` cannot WRAP when `base` is +/// exactly `U16_ID_CAPACITY`: the value is dead (no caller reads a base for a +/// fragment with no entries), but returning a saturated 65,535 rather than a +/// wrapped 0 keeps the dead value from ever being mistaken for a live one. +pub(crate) fn resource_base( + ctx_base: usize, + merged_len: usize, + frag_len: usize, + label: &str, +) -> Result { + let base = ctx_base + merged_len; + let end = base + frag_len; + if frag_len == 0 { + // No id to assign; hand back a base the caller will not use, saturated + // so the narrowing itself cannot wrap. + return Ok(base.min(u16::MAX as usize) as u16); + } + let capacity = id_capacity(); + if end > capacity { + return Err(format!( + "merged {label} count {end} exceeds the bytecode id capacity of \ + {capacity} (ids are 16-bit)" + )); + } + Ok(base as u16) +} + +/// Add a fragment-local resource id to its fragment's base, reporting M3 +/// rather than wrapping. The `u16` twin of `checked_add_u8`. +pub(crate) fn checked_add_u16(base: u16, off: u16, label: &str) -> Result { + base.checked_add(off).ok_or_else(|| { + format!( + "{label} overflow: {base} + {off} exceeds u16::MAX ({})", + u16::MAX + ) + }) +} + +/// The five flat resource-ID base offsets a single fragment's non-GF +/// opcodes are renumbered by (the result of absorbing that fragment into a +/// `FragmentMerger`). Graphical-function IDs are NOT a flat offset -- they +/// are content-de-duplicated (#582), so they are remapped per local slot +/// via the companion `GfRemap` rather than a single base. Pass both to +/// `renumber_opcode` / `renumber_fragment_code`. +#[derive(Clone, Copy, Debug)] +pub(crate) struct FragmentResourceOffsets { + pub lit_offset: u16, + pub mod_offset: u16, + pub view_offset: u16, + pub temp_offset: u32, + pub dl_offset: u16, +} + +/// The four bases for the SHARED CONTEXT resources alone -- what +/// [`FragmentMerger::absorb_context`] can honestly report. +/// +/// Separate from [`FragmentResourceOffsets`] so that "this absorb assigned no +/// literal id" is a property of the type rather than a placeholder value. The +/// all-phases aggregation keeps no literal pool, and a `lit_offset: 0` sitting +/// in its result would be indistinguishable from a real base of zero. +#[derive(Clone, Copy, Debug)] +pub(crate) struct ContextResourceOffsets { + pub mod_offset: u16, + pub view_offset: u16, + pub temp_offset: u32, + pub dl_offset: u16, +} + +impl ContextResourceOffsets { + /// Pair these context bases with the literal base of the same absorb. + fn with_literals(self, lit_offset: u16) -> FragmentResourceOffsets { + FragmentResourceOffsets { + lit_offset, + mod_offset: self.mod_offset, + view_offset: self.view_offset, + temp_offset: self.temp_offset, + dl_offset: self.dl_offset, + } + } +} + +/// Per-fragment local-GF-slot -> global-(deduped)-GF-slot map. Index `i` +/// holds the `merged_gf` index that this fragment's local +/// `graphical_functions[i]` was de-duplicated to. Total over `[0, gf_len)`, +/// so a `Lookup`/`LookupArray` `base_gf` is remapped by a single +/// `gf_remap[base_gf]` lookup; the whole-list shift in +/// `FragmentMerger::absorb_gf` guarantees `gf_remap[base + k] == +/// gf_remap[base] + k`, so the array-lookup `[base .. base + table_count]` +/// contract survives the remap. +pub(crate) type GfRemap = SmallVec<[GraphicalFunctionId; 8]>; /// How a `FragmentMerger` lays out fragment *temps* (#583). /// @@ -1485,71 +1499,196 @@ pub(crate) type GfRemap = SmallVec<[GraphicalFunctionId; 8]>; /// producing builtins like `VectorSortOrder`/`VectorElmMap`): a fragment is /// one variable's bytecode, its temps are 0-based, and they are written and /// read entirely within that variable's expression evaluation -- dead once -/// the variable's runlist segment completes. The two consumers differ in -/// whether their fragments' temp live ranges can overlap: +/// the variable's runlist segment completes. +/// +/// Temps are therefore the ONE merged resource that may legitimately be +/// shared between fragments, and obligation M5 is what bounds the sharing: +/// **in the merged opcode stream, the uses of one merged temp slot must not +/// interleave between two fragments.** Two fragments may share a slot only if +/// the emitter lays their opcodes out as disjoint contiguous runs; if it +/// interleaves them, a shared slot means one fragment reads storage the other +/// has already overwritten. The strategy is how the caller declares which +/// emission shape it is about to produce: /// -/// - `Recycle` (plain-phase `concatenate_fragments`): fragments are emitted -/// as sequential, non-overlapping runlist segments, so two fragments' -/// temps are never simultaneously live. They are max-merged into ONE -/// shared identity pool keyed by temp id -- variable A's temp 0 and -/// variable B's temp 0 collapse to global slot 0, the slot's size the max -/// of the two. This exactly matches the monolithic `Module::compile` keyed -/// max-merge (`compiler/mod.rs`), so the incremental temp count equals the -/// monolithic `n_temps` instead of summing to a count that overflows the -/// `TempId` (= `u8`) namespace. +/// - `Recycle` -- the caller emits each fragment's opcodes as one contiguous +/// run, in fragment order (`concatenate_fragments_with_gf`; the fragments +/// are sequential, non-overlapping runlist segments). Temps collapse by +/// IDENTITY into one shared pool: fragment A's temp 0 and fragment B's +/// temp 0 both become slot `base + 0`, and that slot's size is the MAX over +/// its users, so it is large enough for each of them in turn. Sharing is +/// safe because no two users are live at the same time, and it is necessary +/// because summing 0-based per-fragment temp counts across a whole model +/// overflows the `TempId` (= `u8`) namespace -- the GH #583 failure on +/// C-LEARN, whose flows phase legitimately needs 347 summed slots but far +/// fewer recycled ones. The `u8` width is why recycling is the fix rather +/// than an optimization; widening the id type was the rejected alternative +/// (see #583). /// -/// - `Sum` (`combine_scc_fragment`): the combined SCC fragment INTERLEAVES -/// its members' per-element segments per `element_order`, so members' temp -/// live ranges OVERLAP. Each member's temps must get a DISJOINT id range -/// (advancing per member) -- recycling them onto a shared slot would make -/// two simultaneously-live temps alias and silently miscompile the SCC. +/// - `Sum` -- the caller INTERLEAVES fragments' opcodes (`combine_scc_fragment` +/// emits a resolved recurrence SCC's members' per-element segments in +/// `element_order`, so `ce[0], ecc[0], ce[1], ecc[1]`). Members' temp live +/// ranges overlap, so identity recycling would alias two simultaneously-live +/// temps and silently miscompile the SCC. Each fragment gets a DISJOINT id +/// range instead, which trivially satisfies M5 because no slot has two users +/// at all. An SCC is a handful of members, so the summed count stays far +/// inside `u8`. +/// +/// Both arms are pinned as the live-range property itself rather than as a +/// layout comparison: see `merged_temp_slot_uses_never_interleave` and +/// `recycle_shares_by_identity_and_sizes_by_max` in +/// `symbolic_merge_proptest.rs`. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub(crate) enum TempStrategy { - /// Max-merge fragment temps into one identity-keyed pool (plain-phase - /// concat; matches monolithic recycling). + /// Collapse fragment temps by identity into one shared pool, each slot + /// sized to the max of its users. For an emitter that lays fragments out + /// as disjoint contiguous runs. Recycle, - /// Advance a disjoint temp id range per fragment (combined SCC fragment; - /// interleaved segments need non-overlapping live ranges). + /// Give each fragment a disjoint temp id range. For an emitter that + /// interleaves fragments' opcodes. Sum, } /// Running merge state for combining `PerVarBytecodes` into a single /// resource namespace. /// -/// This is the shared core of `concatenate_fragments` and the +/// This is the shared core of `concatenate_fragments_with_gf` and the /// per-element-granular `combine_scc_fragment` (a multi-member recurrence -/// SCC's combined fragment). The accounting that must hold across both -- -/// every fragment's literals/GFs/modules/views/dim-lists land in a -/// disjoint, non-colliding ID range -- is implemented exactly once here so -/// the two consumers cannot drift. Temps are the one resource whose layout -/// differs: see `TempStrategy`. `concatenate_fragments` absorbs each -/// fragment and immediately renumbers its whole (Ret-stripped) code; -/// `combine_scc_fragment` absorbs each *member* once and renumbers that -/// member's per-element segments with the member's offsets, emitting the -/// segments in the SCC's interleaved `element_order`. +/// SCC's combined fragment). Both consumers absorb fragments through this one +/// implementation so they cannot drift: `concatenate_fragments_with_gf` +/// absorbs each fragment and immediately renumbers its whole (Ret-stripped) +/// code, while `combine_scc_fragment` absorbs each *member* once and +/// renumbers that member's per-element segments with the member's offsets, +/// emitting the segments in the SCC's interleaved `element_order`. +/// +/// # What the merged fragment owes +/// +/// A fragment is one variable's one phase, compiled against nothing but its +/// own resources: its opcodes carry ids into its OWN `literals`, +/// `graphical_functions`, `module_decls`, `static_views`, temp slots and +/// `dim_lists`. Merging relocates those private id spaces into one shared +/// space. These are the obligations that makes sound, stated as properties of +/// the merged fragment -- not as a comparison with any other compiler. Each +/// names the test that pins it. Unqualified test names are in the sibling +/// `symbolic_merge_proptest.rs`; the interleaving consumer's half lives in +/// `db/combined_fragment_proptest.rs`. +/// +/// **M1 -- referential integrity.** For every absorbed fragment `F` and every +/// opcode of `F`, the renumbered opcode names the same resource VALUE in the +/// merged tables that the original named in `F`'s own tables: +/// `merged.literals[id + lit_off] == F.literals[id]`, and likewise for module +/// decls, static views (up to M5's temp-base shift), dim lists (up to the +/// 4-element truncation), and every table of a GF run. This is the whole +/// point of the merge; M2-M4 are the means. Pinned by +/// `merged_ids_dereference_to_their_own_fragments_resources`, with +/// `forced_rich_fragments_exercise_every_resource_kind` as its non-vacuity +/// guard. +/// +/// **M2 -- flat resources are disjoint.** Literals, module decls, static views +/// and dim lists are APPENDED: fragment `i`'s entries occupy +/// `[base_i, base_i + |R_i|)`, those ranges are pairwise disjoint, and +/// together they tile the merged table in fragment order. No fragment can +/// reach another's entry, so M1 is never satisfied by accidental sharing. +/// Pinned by `flat_resource_ranges_are_disjoint_and_tile_the_merged_table`. /// -/// `ctx_base` provides context resource ID offsets inherited from -/// preceding phases. Literal IDs are always phase-local (each phase's -/// bytecode has its own literal pool) so they are not affected by -/// `ctx_base`. Temps recycle into ONE global identity pool, so their -/// `ctx_base.temps` is 0 for every phase (the pool is not partitioned by -/// phase). When assembling a single phase in isolation, pass +/// **M3 -- ids fit their bytecode id type.** Every ASSIGNED id is +/// representable in the type the opcode field carries (`LiteralId`/`ModuleId`/ +/// `ViewId`/`DimListId` are `u16`, `TempId` and `GraphicalFunctionId` are +/// `u8`). A merged table that outgrows its id type is a loud `Err`, never a +/// wrapped id silently naming a different resource. Note what the obligation +/// does NOT say: a table of exactly `U16_ID_CAPACITY` entries satisfies it, and +/// so does any number of later phases that add nothing to one -- so the counts +/// that become a later phase's base are carried in `usize` and the bound is +/// discharged in exactly one function, [`resource_base`], which is the only +/// point that sees both a base and the length of the fragment about to consume +/// it. "Assigned" also means assigned to something the module KEEPS: a merge +/// whose pool is discarded names no id, so bounding it rejects programs that +/// are entirely valid. That is not hypothetical either -- it is why the +/// all-phases aggregation is [`merge_context_side_channels`] and not a full +/// merge. Pinned by `literal_pool_past_u16_capacity_fails_loud` (within a +/// phase), `a_full_ctx_base_bounds_only_the_phases_that_use_it` (across +/// phases), `the_initials_phase_shares_the_mergers_capacity_bound` (the +/// initials phase's own running offsets), +/// `the_all_phases_aggregation_does_not_bound_the_literal_pool` and +/// `assembly_bounds_the_retained_pools_and_not_the_aggregate` (retained vs +/// discarded), `renumber_opcode_u16_addition_overflow_is_loud`, +/// `test_renumber_opcode_u8_addition_overflow` and +/// `test_concatenate_genuinely_distinct_gf_over_capacity_fails_loud`. +/// +/// **M4 -- graphical functions share by content, and only by content.** GFs +/// are the one resource that deliberately does NOT get a disjoint range: N +/// consumers of one dependency's arrayed GF each re-extract the same tables, +/// and appending all N is how the `u8` `GraphicalFunctionId` overflowed +/// (#582). Two local slots may map to one merged id only when their content +/// is bit-identical; a `Lookup`/`LookupArray` run `[b, b + table_count)` stays +/// contiguous and in order; and the remap is TOTAL over `[0, gf_len)`, so even +/// a slot no opcode reads keeps its own content. Pinned by +/// `gf_dedup_preserves_runs_and_never_merges_distinct_content`. +/// +/// **M5 -- temps never alias between simultaneously-live fragments.** The one +/// shareable resource; see [`TempStrategy`] for the live-range argument and +/// which strategy a given emission shape needs. `temp_offsets` is the prefix +/// sum of the merged sizes, so distinct slots' storage never overlaps either. +/// Pinned by `merged_temp_slot_uses_never_interleave` and +/// `recycle_shares_by_identity_and_sizes_by_max` for the sequential emitter, +/// and by `db::combined_fragment_proptest`'s +/// `interleaved_members_never_share_a_temp_slot` for the interleaving one. +/// +/// **M6 -- absorption is 1:1 on opcodes.** The merged code is each fragment's +/// Ret-stripped opcodes, contiguous and in fragment order, plus one terminal +/// `Ret`. Two downstream boundaries are COUNTS into this stream and would move +/// silently if it were not 1:1: `flows_invariant_opcode_len` (the +/// run-invariant flow prefix, GH #712, computed in `db::assemble` as a sum of +/// Ret-stripped fragment lengths) and the SCC per-element segmentation. +/// Pinned by `merge_is_one_to_one_on_opcodes_and_prefix_lengths_are_boundaries` +/// -- which asserts the stronger form the boundary actually needs, that a +/// fragment's renumbering never depends on what FOLLOWS it -- and, for the +/// interleaving emitter, by `db::combined_fragment_proptest`'s +/// `interleave_conserves_opcodes_and_follows_element_order`. +/// +/// **M7 -- variable references are not the merger's business.** Renumbering +/// touches resource ids only: every `SymVarRef` comes through byte-identical +/// and every opcode keeps its variant. Addresses are assigned exactly once, +/// later, by `resolve_module`. Pinned by the `skeleton` half of +/// `merged_ids_dereference_to_their_own_fragments_resources`: the comparison +/// blanks only the resource ids, so every `SymVarRef` and every jump offset +/// must survive byte-identical. +/// +/// **M8 -- the phase split agrees with the whole-model merge.** `assemble_module` +/// renumbers initials, flows and stocks SEPARATELY, but takes the module's +/// `module_decls` / `static_views` / `temp_offsets` / `dim_lists` tables from a +/// single all-phases merge. So merging one phase with `ctx_base` set to the +/// summed counts of the preceding phases must assign each fragment exactly the +/// ids the all-phases merge assigns it -- otherwise a flows opcode indexes the +/// module's table with an id computed against a different table. Literals are +/// the deliberate exception: each phase's bytecode carries its own pool and +/// the all-phases pool is discarded, so `ctx_base` does not apply to them. +/// Pinned by `phase_split_assigns_the_same_ids_as_the_all_phases_merge`, with +/// `forced_rich_phase_split_uses_non_zero_bases` as its non-vacuity guard -- +/// with an all-zero `ctx_base` the split IS the all-phases merge, so the +/// property would pass without testing anything. +/// +/// `ctx_base` provides the context resource id offsets inherited from +/// preceding phases (M8). Temps recycle into ONE pool shared by every phase, +/// so `ctx_base.temps` is 0 for every phase -- the pool is not partitioned by +/// phase. When merging a single phase in isolation, pass /// `ContextResourceCounts::default()`. pub(crate) struct FragmentMerger { ctx_base: ContextResourceCounts, temp_strategy: TempStrategy, merged_literals: Vec, merged_gf: Vec>, - /// Cross-fragment graphical-function de-duplication index (#582). Maps a - /// GF *block* -- a maximal contiguous run of one or more lookup tables, - /// the granularity the monolithic `Compiler::new` lays out one-per- - /// variable -- keyed by its bit-exact content, to the global `merged_gf` - /// offset its first occurrence was appended at. A dependency arrayed GF - /// referenced by N consumer fragments produces N fragments each carrying - /// the *same* block (every consumer re-extracts the dependency's - /// `Vec
` -- see `db/var_fragment.rs`); de-duplicating the block - /// appends it once and remaps every consumer's `base_gf` by the single - /// shared offset, matching the monolithic layout. + /// Cross-fragment graphical-function de-duplication index (#582, M4). + /// Maps a GF *block* -- a maximal contiguous run of one or more lookup + /// tables, the granularity `Compiler::new` lays a fragment's tables out in + /// (one run per table-bearing variable, `codegen.rs`) -- keyed by its + /// bit-exact content, to the global `merged_gf` offset its first + /// occurrence was appended at. A dependency arrayed GF referenced by N + /// consumer fragments produces N fragments each carrying the *same* block + /// (every consumer re-extracts the dependency's `Vec
` -- see + /// `db/var_fragment.rs`); de-duplicating the block appends it once and + /// remaps every consumer's `base_gf` by the single shared offset, which is + /// what keeps the merged count proportional to the DISTINCT tables a model + /// has rather than to how many fragments mention them. /// /// A fragment's blocks are the *maximal* contiguous intervals its /// `Lookup`/`LookupArray` opcodes reference (overlapping/nested ranges @@ -1603,24 +1742,28 @@ fn gf_block_key(tables: &[Vec<(f64, f64)>]) -> GfBlockKey { /// `(start, len)` blocks covering `[0, gf_len)` exactly, sorted by `start` /// (#582). /// -/// Each `Lookup`/`LookupArray` `base_gf` addresses a run of `table_count` -/// tables starting at `base_gf`, and `base_gf` is always an originating -/// variable's block start (the monolithic `Compiler::new` only ever emits a -/// `base_gf` from its one-per-variable `table_base_ids` map). Within a -/// fragment one such run can be *nested* inside another: a per-element -/// arrayed GF `g` is read both as the whole array (`LookupArray { base, |D| -/// }`) and at one element (`Lookup { base + e, 1 }` -- fully inside the -/// array's range). The whole-array run is the real block; the nested -/// element run is a sub-reference, NOT a separate block (splitting the +/// This is a property of the FRAGMENT, read off its own opcodes, and it needs +/// exactly two things from the emitter that produced it (`Compiler::new` -- +/// the one codegen both the fragment path and the test-only whole-model path +/// share): each `base_gf` an opcode carries is the start of some table-bearing +/// variable's run, and distinct variables' runs are disjoint. Both hold +/// because `Compiler::new` assigns every `base_gf` out of one +/// `table_base_ids` map built by laying each variable's tables down +/// contiguously from a monotonically advancing cursor. +/// +/// Given that, a `Lookup`/`LookupArray` run `[base_gf, base_gf + table_count)` +/// can be *nested* inside another run but never partially overlap it: a +/// per-element arrayed GF `g` is read both as the whole array +/// (`LookupArray { base, |D| }`) and at one element (`Lookup { base + e, 1 }`, +/// fully inside the array's range). The whole-array run is the real block; the +/// nested element run is a sub-reference, NOT a separate block (splitting the /// block at the element boundary could scatter the array across the deduped -/// table and miscompile the `[base .. base + table_count]` array lookup). -/// Distinct variables' blocks are laid out *disjointly* by `Compiler::new`, -/// so opcode runs are only ever disjoint or nested -- never partially -/// overlapping. The blocks are therefore the *maximal-by-inclusion* opcode -/// runs (nested runs dropped), plus one block per maximal *un-referenced* -/// gap (over-collected dependency tables `db/var_fragment.rs` gathered but -/// no opcode reads -- never read, so an imperfect gap boundary cannot -/// miscompile, only mildly affect the deduped count). +/// table and miscompile the `[base .. base + table_count]` array lookup). The +/// blocks are therefore the *maximal-by-inclusion* opcode runs (nested runs +/// dropped), plus one block per maximal *un-referenced* gap (over-collected +/// dependency tables `db/var_fragment.rs` gathered but no opcode reads -- +/// never read, so an imperfect gap boundary cannot miscompile, only mildly +/// affect the deduped count). /// /// `Err` only if a run extends past `gf_len` or two runs partially overlap /// (a corrupt fragment the engine never produces); loud-safe. @@ -1708,9 +1851,10 @@ impl FragmentMerger { Self::new_with_temp_strategy(ctx_base, TempStrategy::Sum) } - /// New merger with an explicit temp strategy. `concatenate_fragments` - /// (plain-phase, sequential segments) uses `TempStrategy::Recycle` to - /// match the monolithic keyed max-merge; the SCC path uses `Sum`. + /// New merger with an explicit temp strategy (M5). A caller that emits + /// each fragment's opcodes as one contiguous run passes + /// `TempStrategy::Recycle`; one that interleaves them passes `Sum`. See + /// [`TempStrategy`]. pub(crate) fn new_with_temp_strategy( ctx_base: &ContextResourceCounts, temp_strategy: TempStrategy, @@ -1739,58 +1883,112 @@ impl FragmentMerger { &mut self, frag: &PerVarBytecodes, ) -> Result<(FragmentResourceOffsets, GfRemap), String> { - let off = self.absorb_non_gf(frag); + let off = self.absorb_non_gf(frag)?; let gf_remap = self.absorb_gf(frag)?; Ok((off, gf_remap)) } /// Absorb one fragment's flat (non-GF) side-channels -- literals, /// modules, views, temp sizes, dim lists -- into the running merge - /// state and return the five flat resource base offsets. The literal / - /// module / view / dim-list offsets are computed from the *pre-merge* - /// lengths (each is a distinct resource, laid out disjointly), then those - /// side-channels are appended (`Temp`-based static views shifted by this - /// fragment's `temp_offset`, dim-lists truncated to 4). The *temp* offset - /// instead follows `temp_strategy` (#583): `Sum` advances per fragment - /// (disjoint ranges, for `combine_scc_fragment`'s interleaved segments); - /// `Recycle` uses the fixed `ctx_base.temps` so every fragment's temps - /// max-merge into one identity pool (plain-phase concat, matching the - /// monolithic keyed max-merge). Graphical functions are handled - /// separately by `absorb_gf` (content-de-duplicated, #582). - pub(crate) fn absorb_non_gf(&mut self, frag: &PerVarBytecodes) -> FragmentResourceOffsets { - // Literals are phase-local; no ctx_base offset needed. Modules, - // views, and dim-lists are appended unshifted, so their offset is - // `ctx_base + cumulative_appended` (no double-count: the appended - // entries do NOT carry the ctx_base, so `merged_X.len()` excludes - // it). Temps are different: see below. - let lit_offset = self.merged_literals.len() as u16; - let mod_offset = self.merged_modules.len() as u16 + self.ctx_base.modules; - let view_offset = self.merged_views.len() as u16 + self.ctx_base.views; + /// state and return the five flat resource base offsets. + /// + /// M2: the literal / module / view / dim-list offsets are the *pre-merge* + /// lengths, and the fragment's entries are then appended, so this + /// fragment's entries occupy exactly `[base, base + len)` and no earlier + /// fragment's range can overlap it. Two things are rewritten on the way + /// in, and both are M1 (a merged entry must still denote what the + /// fragment's entry denoted): a `Temp`-based static view is shifted by + /// this fragment's `temp_offset` so it still points at the same temp, and + /// a dim-list is truncated to the 4 elements the `(u8, [u16; 4])` merged + /// representation holds. + /// + /// The *temp* offset instead follows `temp_strategy` (#583, M5): `Sum` + /// advances per fragment so each gets a disjoint range; `Recycle` uses + /// the fixed `ctx_base.temps` so every fragment's temp `t` lands on the + /// same slot `t + base`, max-merged below. See [`TempStrategy`] for which + /// emission shape needs which. Graphical functions are handled separately + /// by `absorb_gf` (content-de-duplicated, #582, M4). + /// + /// `Err` on M3: a merged table that outgrows its `u16` id type. The base + /// offsets are checked here rather than only at `renumber_opcode` because + /// a fragment may carry entries no opcode reads (over-collected dependency + /// resources), and those still consume ids for the fragments after it. + pub(crate) fn absorb_non_gf( + &mut self, + frag: &PerVarBytecodes, + ) -> Result { + // Literals are phase-local; no ctx_base offset needed (M8). They are + // also the ONE resource an aggregation may legitimately not want, which + // is why they are the half that lives here rather than in + // `absorb_context` -- see that method's rustdoc. + let lit_offset = resource_base( + 0, + self.merged_literals.len(), + frag.symbolic.literals.len(), + "literal", + )?; + let ctx = self.absorb_context(frag)?; + self.merged_literals + .extend_from_slice(&frag.symbolic.literals); + Ok(ctx.with_literals(lit_offset)) + } + + /// Absorb one fragment's SHARED CONTEXT side-channels -- module decls, + /// static views, temp sizes, dim lists -- and return their base offsets. + /// The literal pool is deliberately not touched. + /// + /// This is the whole of `absorb_non_gf` except the literal pool, and it is + /// separate because the two have different lifetimes in the assembled + /// module. The context tables are shared by every phase and reported ONCE, + /// by the all-phases aggregation ([`merge_context_side_channels`]); the + /// literal pools are per-phase and each is retained by the phase that built + /// it. An aggregation that wants the former must not be bounded by the + /// latter, because it assigns no merged literal id to anything. + fn absorb_context(&mut self, frag: &PerVarBytecodes) -> Result { + // Modules, views, and dim-lists are appended unshifted, so their offset + // is `ctx_base + cumulative_appended` (no double-count: the appended + // entries do NOT carry the ctx_base, so `merged_X.len()` excludes it). + // Temps are different: see below. + let mod_offset = resource_base( + self.ctx_base.modules, + self.merged_modules.len(), + frag.module_decls.len(), + "module declaration", + )?; + let view_offset = resource_base( + self.ctx_base.views, + self.merged_views.len(), + frag.static_views.len(), + "static view", + )?; // #583: temps recycle (plain-phase) or sum (interleaved SCC). // // `Recycle`: a FIXED base (`ctx_base.temps`, which is 0 for every // plain phase since temps share ONE global identity pool). The // per-fragment max-merge below places fragment temp id `t` at slot - // `t + base`, so every fragment's id 0 collapses to the same slot - // -- the monolithic keyed max-merge. + // `t + base`, so every fragment's id 0 collapses to the same slot. // `Sum`: advance by the running pool length so each fragment gets a - // disjoint range (interleaved SCC segments need non-overlapping - // live ranges). NOTE the previous unconditional + // disjoint range (interleaved segments need non-overlapping live + // ranges). NOTE the previous unconditional // `merged_temp_sizes.len() + ctx_base.temps` double-counted // `ctx_base.temps`: temps are stored at `id + temp_offset` (which // already includes the base), so `merged_temp_sizes.len()` absorbs // the base -- adding it again diverged `flows_concat` from the - // all-phases `merged` table. The recycle path's fixed base removes - // that divergence; the Sum path runs only with `ctx_base.temps == 0` - // (`combine_scc_fragment` passes a default ctx_base). + // all-phases `merged` table (an M8 violation). The recycle path's + // fixed base removes that divergence; the Sum path runs only with + // `ctx_base.temps == 0` (`combine_scc_fragment` passes a default + // ctx_base). let temp_offset = match self.temp_strategy { TempStrategy::Recycle => self.ctx_base.temps, TempStrategy::Sum => self.merged_temp_sizes.len() as u32 + self.ctx_base.temps, }; - let dl_offset = self.merged_dim_lists.len() as u16 + self.ctx_base.dim_lists; + let dl_offset = resource_base( + self.ctx_base.dim_lists, + self.merged_dim_lists.len(), + frag.dim_lists.len(), + "dimension list", + )?; - self.merged_literals - .extend_from_slice(&frag.symbolic.literals); self.merged_modules.extend_from_slice(&frag.module_decls); self.merged_views.extend(frag.static_views.iter().map(|sv| { let base = match &sv.base { @@ -1818,13 +2016,12 @@ impl FragmentMerger { self.merged_temp_sizes[new_id as usize].max(*size); } - FragmentResourceOffsets { - lit_offset, + Ok(ContextResourceOffsets { mod_offset, view_offset, temp_offset, dl_offset, - } + }) } /// Content-de-duplicate one fragment's graphical-function *blocks* into @@ -1890,11 +2087,28 @@ impl FragmentMerger { /// `code` is the already-renumbered, Ret-stripped opcode stream; /// a single trailing `Ret` is appended iff `code` is non-empty /// (preserving the original `concatenate_fragments` behavior). - fn into_concatenated(self, mut code: Vec) -> ConcatenatedBytecodes { + fn into_concatenated(mut self, mut code: Vec) -> ConcatenatedBytecodes { if !code.is_empty() { code.push(SymbolicOpcode::Ret); } + let literals = std::mem::take(&mut self.merged_literals); + let side = self.into_side_channels(); + ConcatenatedBytecodes { + bytecode: SymbolicByteCode { literals, code }, + graphical_functions: side.graphical_functions, + module_decls: side.module_decls, + static_views: side.static_views, + temp_offsets: side.temp_offsets, + temp_total_size: side.temp_total_size, + dim_lists: side.dim_lists, + } + } + /// Consume the merger and finalize just the SHARED CONTEXT side-channels, + /// discarding any literal pool. `temp_offsets` is the prefix sum of the + /// merged temp sizes, computed here and nowhere else so the two finishers + /// cannot disagree about the temp layout. + fn into_side_channels(self) -> ContextSideChannels { let mut temp_offsets = Vec::with_capacity(self.merged_temp_sizes.len()); let mut offset = 0usize; for &size in &self.merged_temp_sizes { @@ -1902,11 +2116,7 @@ impl FragmentMerger { offset += size; } - ConcatenatedBytecodes { - bytecode: SymbolicByteCode { - literals: self.merged_literals, - code, - }, + ContextSideChannels { graphical_functions: self.merged_gf, module_decls: self.merged_modules, static_views: self.merged_views, @@ -1989,6 +2199,59 @@ pub(crate) fn renumber_fragment_code( Ok(()) } +/// The context resources every phase of an assembled module SHARES, reported +/// once. These are the only things the all-phases aggregation exists to +/// produce; the bytecode and literal pool a full merge would also build are +/// per-phase and are retained by the phases that build them. +pub(crate) struct ContextSideChannels { + pub graphical_functions: Vec>, + pub module_decls: Vec, + pub static_views: Vec, + pub temp_offsets: Vec, + pub temp_total_size: usize, + pub dim_lists: Vec<(u8, [u16; 4])>, +} + +/// Aggregate the SHARED CONTEXT resources of every fragment in a module, in the +/// all-phases order (initials, flows, stocks), so the module reports one +/// module-decl / static-view / temp / dim-list table that each phase's +/// separately-renumbered ids index. +/// +/// This is deliberately NOT a merge. The phases that keep bytecode +/// (`concatenate_fragments_with_gf` for flows and stocks, +/// `db::assemble::renumber_initials_phase` for initials) each keep their own +/// literal pool, so an aggregate pool over all three corresponds to no id the +/// module retains. Building one was not merely wasted work: it put the whole +/// model's literal count under `resource_base`'s `u16` bound, so a model whose +/// every RETAINED pool was comfortably addressable failed to assemble once the +/// three phases' pools summed past 65,536 -- roughly 33k scalar stocks, which +/// is inside the size class this compiler targets. Assembly used to survive it +/// by wrapping the offsets of a stream nobody read; the capacity check turned +/// that into a hard error. +/// +/// M8 is unaffected: it obliges the per-phase renumber and the all-phases +/// aggregation to assign the same *context* ids, and literals were never part +/// of it (`absorb_non_gf` bases them at 0 in every phase precisely because they +/// are phase-local). Graphical functions come from the shared `dedup`, exactly +/// as they do for a phase merge. +pub(crate) fn merge_context_side_channels( + fragments: &[&PerVarBytecodes], + ctx_base: &ContextResourceCounts, + dedup: &GfDedup, +) -> Result { + // `Recycle` to match the phase merges: temp slots collapse by identity into + // the one pool every phase shares, and this aggregation is what sizes it. + let mut merger = FragmentMerger::new_with_temp_strategy(ctx_base, TempStrategy::Recycle); + for frag in fragments { + merger.absorb_context(frag)?; + } + let mut side = merger.into_side_channels(); + // The merger never touched GF (`absorb_context`), so install the shared + // deduped table; every phase reports the same `graphical_functions`. + side.graphical_functions = dedup.tables.clone(); + Ok(side) +} + /// Merge a single phase's `PerVarBytecodes` into one stream, renumbering /// `LiteralId`, `GraphicalFunctionId`, `ModuleId`, `ViewId`, `TempId`, and /// `DimListId` to avoid collisions across fragments, with the graphical @@ -2024,10 +2287,10 @@ pub(crate) struct GfDedup { } impl GfDedup { - /// De-duplicate the GF table lists of `fragments` (in order) by - /// bit-exact content, matching the monolithic `Compiler::new`'s - /// one-list-per-variable layout. Value-exact: genuinely-different lists - /// never share an offset. `Err` if the *distinct* GF count exceeds + /// De-duplicate the GF blocks of `fragments` (in order) by bit-exact + /// content (M4). Value-exact: two blocks share an offset only when their + /// content is identical, so a `Lookup` can never be redirected to a + /// different table. `Err` if the *distinct* GF count exceeds /// `GraphicalFunctionId` capacity (the genuine-capacity case the dedup /// cannot help -- escalate, do not widen the ID width here). pub fn build(fragments: &[&PerVarBytecodes]) -> Result { @@ -2058,28 +2321,29 @@ impl GfDedup { /// over (0 when `dedup` covers exactly `fragments`; the running phase /// offset when one `GfDedup` spans initials + flows + stocks). /// -/// The non-GF resource accounting is byte-for-byte the original -/// `concatenate_fragments` loop -- only the GF base now comes from the -/// shared deduped remap instead of a flat `gf_off`, so the output is -/// identical to before for any model whose GF table lists were already -/// distinct. +/// This is the sequential-emission consumer of `FragmentMerger`: each +/// fragment's Ret-stripped opcodes are appended as ONE contiguous run, in +/// fragment order (M6), which is what makes `TempStrategy::Recycle` sound +/// here (M5) and what makes a prefix of the fragment list correspond to a +/// prefix of the merged opcode stream -- the boundary `assemble_module` +/// counts for the run-invariant flow prefix. pub(crate) fn concatenate_fragments_with_gf( fragments: &[&PerVarBytecodes], ctx_base: &ContextResourceCounts, dedup: &GfDedup, gf_index_base: usize, ) -> Result { - // Plain-phase concat: temps RECYCLE into one identity pool (matching the - // monolithic keyed max-merge), since fragments are sequential, non- - // overlapping runlist segments. `combine_scc_fragment` (interleaved - // segments) uses the disjoint `Sum` path instead. + // Fragments are appended whole and in order below, so no two fragments' + // temp uses interleave and they may share slots by identity (M5). + // `combine_scc_fragment`, which interleaves per-element segments, uses the + // disjoint `Sum` path instead. let mut merger = FragmentMerger::new_with_temp_strategy(ctx_base, TempStrategy::Recycle); let mut merged_code: Vec = Vec::new(); for (i, frag) in fragments.iter().enumerate() { // Only the flat resources are merged here; GF numbering comes from // the shared `dedup` so it is coherent across phases. - let off = merger.absorb_non_gf(frag); + let off = merger.absorb_non_gf(frag)?; renumber_fragment_code( &frag.symbolic.code, &off, @@ -2134,8 +2398,10 @@ fn remap_gf( /// is translated through `gf_remap` (the fragment's per-slot local->global /// map from `FragmentMerger::absorb_gf`) rather than a flat add. /// -/// Returns `Err` if a per-opcode temp id would overflow `TempId` (= `u8`) -/// after offsetting (the `checked_add_u8` below) or if a `base_gf` is out of +/// Every offsetting add is checked (M3): a wrapped id is a well-formed +/// program that names a different resource, so the failure mode of getting +/// this wrong is wrong numbers with no diagnostic. `Err` on a temp id past +/// `TempId` (= `u8`), on any `u16` id past its type, or on a `base_gf` out of /// range for `gf_remap` (a corrupt fragment). /// /// There is no separate `temp_off > u8::MAX` precheck (#583): the plain- @@ -2146,6 +2412,12 @@ fn remap_gf( /// an SCC summing past 255 -- is still caught loud by `checked_add_u8`, /// which adds the actual `temp_id` to the offset (the precheck only saw the /// offset, so it could not have been the real bound anyway). +/// +/// The `u16` adds are belt-and-braces alongside `absorb_non_gf`'s capacity +/// check: that one bounds the merged TABLE, this one bounds the id an +/// individual opcode carries, and a caller can reach this function with a base +/// the merger did not compute (`assemble_module`'s per-initial renumber loop +/// tracks its own running offsets). pub(crate) fn renumber_opcode( op: &SymbolicOpcode, lit_off: u16, @@ -2167,10 +2439,12 @@ pub(crate) fn renumber_opcode( ) })?; Ok(match op { - SymbolicOpcode::LoadConstant { id } => SymbolicOpcode::LoadConstant { id: *id + lit_off }, + SymbolicOpcode::LoadConstant { id } => SymbolicOpcode::LoadConstant { + id: checked_add_u16(*id, lit_off, "LiteralId")?, + }, SymbolicOpcode::AssignConstCurr { var, literal_id } => SymbolicOpcode::AssignConstCurr { var: var.clone(), - literal_id: *literal_id + lit_off, + literal_id: checked_add_u16(*literal_id, lit_off, "LiteralId")?, }, SymbolicOpcode::Lookup { base_gf, @@ -2182,27 +2456,23 @@ pub(crate) fn renumber_opcode( mode: *mode, }, SymbolicOpcode::EvalModule { id, n_inputs } => SymbolicOpcode::EvalModule { - id: *id + mod_off, + id: checked_add_u16(*id, mod_off, "ModuleId")?, n_inputs: *n_inputs, }, SymbolicOpcode::PushStaticView { view_id } => SymbolicOpcode::PushStaticView { - view_id: *view_id + view_off, + view_id: checked_add_u16(*view_id, view_off, "ViewId")?, }, SymbolicOpcode::PushTempView { temp_id, dim_list_id, } => SymbolicOpcode::PushTempView { temp_id: checked_add_u8(*temp_id, temp_off_u8, "TempId")?, - dim_list_id: *dim_list_id + dl_off, - }, - SymbolicOpcode::PushVarView { var, dim_list_id } => SymbolicOpcode::PushVarView { - var: var.clone(), - dim_list_id: *dim_list_id + dl_off, + dim_list_id: checked_add_u16(*dim_list_id, dl_off, "DimListId")?, }, SymbolicOpcode::PushVarViewDirect { var, dim_list_id } => { SymbolicOpcode::PushVarViewDirect { var: var.clone(), - dim_list_id: *dim_list_id + dl_off, + dim_list_id: checked_add_u16(*dim_list_id, dl_off, "DimListId")?, } } SymbolicOpcode::LoadTempConst { temp_id, index } => SymbolicOpcode::LoadTempConst { @@ -2272,6 +2542,14 @@ pub(crate) fn renumber_opcode( }) } +#[cfg(test)] +#[path = "symbolic_builder_tests.rs"] +mod builder_tests; + +#[cfg(test)] +#[path = "symbolic_merge_proptest.rs"] +mod merge_proptest; + // ============================================================================ // Tests // ============================================================================ @@ -2281,6 +2559,11 @@ mod tests { use super::*; use crate::bytecode::Op2; + /// A symbolic reference to `name`'s element `elem`. + fn sref(name: &str, elem: usize) -> SymVarRef { + SymVarRef::new(Ident::new(name), elem) + } + fn simple_layout() -> VariableLayout { let mut entries = HashMap::new(); // Root model: implicit vars at 0-3, then user vars alphabetically @@ -2296,109 +2579,46 @@ mod tests { VariableLayout::new(entries, 6) } + /// The two reference-bearing opcode families every model uses. #[test] - fn test_reverse_offset_map_basic() { - let layout = simple_layout(); - let rmap = ReverseOffsetMap::from_layout(&layout); - - let var = rmap.lookup(4).unwrap(); - assert_eq!(var.name, "births"); - assert_eq!(var.element_offset, 0); - - let var = rmap.lookup(5).unwrap(); - assert_eq!(var.name, "population"); - assert_eq!(var.element_offset, 0); - } - - #[test] - fn test_reverse_offset_map_array() { - let mut entries = HashMap::new(); - entries.insert("arr".to_string(), LayoutEntry { offset: 4, size: 3 }); - let layout = VariableLayout::new(entries, 7); - let rmap = ReverseOffsetMap::from_layout(&layout); - - assert_eq!(rmap.lookup(4).unwrap().element_offset, 0); - assert_eq!(rmap.lookup(5).unwrap().element_offset, 1); - assert_eq!(rmap.lookup(6).unwrap().element_offset, 2); - assert_eq!(rmap.lookup(4).unwrap().name, "arr"); - } - - #[test] - fn test_reverse_offset_map_out_of_range() { - let layout = simple_layout(); - let rmap = ReverseOffsetMap::from_layout(&layout); - assert!(rmap.lookup(99).is_err()); - } - - #[test] - fn test_symbolize_load_var() { - let layout = simple_layout(); - let rmap = ReverseOffsetMap::from_layout(&layout); - - let op = Opcode::LoadVar { off: 5 }; - let sym = symbolize_opcode(&op, &rmap).unwrap(); - assert_eq!( - sym, + fn test_resolve_load_var_and_assign_curr() { + assert_resolves( SymbolicOpcode::LoadVar { - var: SymVarRef { - name: "population".to_string(), - element_offset: 0 - } - } + var: sref("population", 0), + }, + Opcode::LoadVar { off: 5 }, ); - } - - #[test] - fn test_symbolize_assign_curr() { - let layout = simple_layout(); - let rmap = ReverseOffsetMap::from_layout(&layout); - - let op = Opcode::AssignCurr { off: 4 }; - let sym = symbolize_opcode(&op, &rmap).unwrap(); - assert_eq!( - sym, + assert_resolves( SymbolicOpcode::AssignCurr { - var: SymVarRef { - name: "births".to_string(), - element_offset: 0 - } - } + var: sref("births", 0), + }, + Opcode::AssignCurr { off: 4 }, ); } + /// Opcodes that carry no variable reference pass through resolution with + /// their operands untouched. #[test] - fn test_symbolize_passthrough_opcodes() { - let layout = simple_layout(); - let rmap = ReverseOffsetMap::from_layout(&layout); - - // These opcodes should pass through unchanged - let op = Opcode::LoadGlobalVar { off: 1 }; - let sym = symbolize_opcode(&op, &rmap).unwrap(); - assert_eq!(sym, SymbolicOpcode::LoadGlobalVar { off: 1 }); - - let op = Opcode::Op2 { op: Op2::Add }; - let sym = symbolize_opcode(&op, &rmap).unwrap(); - assert_eq!(sym, SymbolicOpcode::Op2 { op: Op2::Add }); - - let op = Opcode::Ret; - let sym = symbolize_opcode(&op, &rmap).unwrap(); - assert_eq!(sym, SymbolicOpcode::Ret); + fn test_resolve_passthrough_opcodes() { + assert_resolves( + SymbolicOpcode::LoadGlobalVar { off: 1 }, + Opcode::LoadGlobalVar { off: 1 }, + ); + assert_resolves( + SymbolicOpcode::Op2 { op: Op2::Add }, + Opcode::Op2 { op: Op2::Add }, + ); + assert_resolves(SymbolicOpcode::Ret, Opcode::Ret); } #[test] fn test_resolve_var_ref() { let layout = simple_layout(); - let var = SymVarRef { - name: "population".to_string(), - element_offset: 0, - }; + let var = sref("population", 0); assert_eq!(resolve_var_ref(&var, &layout).unwrap(), 5); - let var = SymVarRef { - name: "births".to_string(), - element_offset: 0, - }; + let var = sref("births", 0); assert_eq!(resolve_var_ref(&var, &layout).unwrap(), 4); } @@ -2414,10 +2634,7 @@ mod tests { ); let layout = VariableLayout::new(entries, 13); - let var = SymVarRef { - name: "arr".to_string(), - element_offset: 2, - }; + let var = sref("arr", 2); assert_eq!(resolve_var_ref(&var, &layout).unwrap(), 12); } @@ -2428,37 +2645,25 @@ mod tests { let layout = VariableLayout::new(entries, 7); // element_offset == size (out of bounds) - let var = SymVarRef { - name: "arr".to_string(), - element_offset: 3, - }; + let var = sref("arr", 3); assert!( resolve_var_ref(&var, &layout).is_err(), "element_offset >= size should fail" ); // element_offset well beyond size - let var = SymVarRef { - name: "arr".to_string(), - element_offset: 100, - }; + let var = sref("arr", 100); assert!(resolve_var_ref(&var, &layout).is_err()); // element_offset at max valid index should succeed - let var = SymVarRef { - name: "arr".to_string(), - element_offset: 2, - }; + let var = sref("arr", 2); assert_eq!(resolve_var_ref(&var, &layout).unwrap(), 6); } #[test] fn test_resolve_missing_variable() { let layout = simple_layout(); - let var = SymVarRef { - name: "nonexistent".to_string(), - element_offset: 0, - }; + let var = sref("nonexistent", 0); assert!(resolve_var_ref(&var, &layout).is_err()); } @@ -2479,10 +2684,7 @@ mod tests { ); let layout = VariableLayout::new(entries, (VariableOffset::MAX as usize) + 2); - let var = SymVarRef { - name: "huge".to_string(), - element_offset: 0, - }; + let var = sref("huge", 0); let err = resolve_var_ref(&var, &layout).expect_err("offset beyond u16 must error"); assert!( err.contains("huge") && err.contains("65535"), @@ -2500,10 +2702,7 @@ mod tests { }, ); let layout = VariableLayout::new(entries, (VariableOffset::MAX as usize) + 4); - let var = SymVarRef { - name: "arr".to_string(), - element_offset: 3, - }; + let var = sref("arr", 3); assert!( resolve_var_ref(&var, &layout).is_err(), "element offset crossing the u16 limit must error" @@ -2522,10 +2721,7 @@ mod tests { }, ); let layout = VariableLayout::new(entries, (VariableOffset::MAX as usize) + 1); - let var = SymVarRef { - name: "edge".to_string(), - element_offset: 0, - }; + let var = sref("edge", 0); assert_eq!(resolve_var_ref(&var, &layout).unwrap(), VariableOffset::MAX); } @@ -2553,112 +2749,158 @@ mod tests { assert!(check_layout_addressable(171_597, "main").is_err()); } + /// A whole bytecode stream resolves opcode-for-opcode, literals intact. #[test] - fn test_bytecode_roundtrip() { + fn test_bytecode_resolution() { let layout = simple_layout(); - let bc = ByteCode { + let sym = SymbolicByteCode { literals: vec![1.0, 0.5], code: vec![ - Opcode::LoadVar { off: 5 }, // population - Opcode::LoadConstant { id: 1 }, // 0.5 - Opcode::Op2 { op: Op2::Mul }, - Opcode::AssignCurr { off: 4 }, // births - Opcode::Ret, + SymbolicOpcode::LoadVar { + var: sref("population", 0), + }, + SymbolicOpcode::LoadConstant { id: 1 }, + SymbolicOpcode::Op2 { op: Op2::Mul }, + SymbolicOpcode::AssignCurr { + var: sref("births", 0), + }, + SymbolicOpcode::Ret, ], }; - let sym = symbolize_bytecode(&bc, &ReverseOffsetMap::from_layout(&layout)).unwrap(); let resolved = resolve_bytecode(&sym, &layout).unwrap(); - - assert_eq!(bc.literals, resolved.literals); - assert_eq!(bc.code.len(), resolved.code.len()); - for (i, (orig, res)) in bc.code.iter().zip(resolved.code.iter()).enumerate() { - assert!( - opcode_eq(orig, res), - "opcode mismatch at index {}: {:?} vs {:?}", - i, - orig, - res - ); - } + assert_eq!(resolved.literals, vec![1.0, 0.5]); + assert_eq!( + resolved.code, + vec![ + Opcode::LoadVar { off: 5 }, + Opcode::LoadConstant { id: 1 }, + Opcode::Op2 { op: Op2::Mul }, + Opcode::AssignCurr { off: 4 }, + Opcode::Ret, + ] + ); } + /// The three superinstructions the peephole and the emit-time stock fusion + /// produce all carry a reference and all resolve. #[test] - fn test_bytecode_roundtrip_superinstructions() { + fn test_bytecode_resolution_superinstructions() { let layout = simple_layout(); - let bc = ByteCode { + // The two `BinOpAssign*` forms pop their operands, so the stream has + // to be stack-balanced: `resolve_bytecode` validates depth (that is + // where the VM's unchecked stack access is proven sound). + let sym = SymbolicByteCode { literals: vec![100.0, 0.0], code: vec![ + SymbolicOpcode::AssignConstCurr { + var: sref("population", 0), + literal_id: 0, + }, + SymbolicOpcode::LoadConstant { id: 0 }, + SymbolicOpcode::LoadConstant { id: 1 }, + SymbolicOpcode::BinOpAssignCurr { + op: Op2::Add, + var: sref("births", 0), + }, + SymbolicOpcode::LoadConstant { id: 0 }, + SymbolicOpcode::LoadConstant { id: 1 }, + SymbolicOpcode::BinOpAssignNext { + op: Op2::Mul, + var: sref("population", 0), + }, + SymbolicOpcode::Ret, + ], + }; + + let resolved = resolve_bytecode(&sym, &layout).unwrap(); + assert_eq!( + resolved.code, + vec![ Opcode::AssignConstCurr { off: 5, literal_id: 0, }, + Opcode::LoadConstant { id: 0 }, + Opcode::LoadConstant { id: 1 }, Opcode::BinOpAssignCurr { op: Op2::Add, off: 4, }, + Opcode::LoadConstant { id: 0 }, + Opcode::LoadConstant { id: 1 }, Opcode::BinOpAssignNext { op: Op2::Mul, off: 5, }, Opcode::Ret, - ], - }; - - let sym = symbolize_bytecode(&bc, &ReverseOffsetMap::from_layout(&layout)).unwrap(); - let resolved = resolve_bytecode(&sym, &layout).unwrap(); - - assert_eq!(bc.code.len(), resolved.code.len()); - for (i, (orig, res)) in bc.code.iter().zip(resolved.code.iter()).enumerate() { - assert!( - opcode_eq(orig, res), - "opcode mismatch at index {}: {:?} vs {:?}", - i, - orig, - res - ); - } + ] + ); } + /// The implicit globals keep their fixed absolute slots: they are read by + /// `LoadGlobalVar`, which never goes through a layout at all. #[test] - fn test_bytecode_roundtrip_global_vars() { + fn test_bytecode_resolution_global_vars() { let layout = simple_layout(); - let bc = ByteCode { + let sym = SymbolicByteCode { literals: vec![], code: vec![ - Opcode::LoadGlobalVar { off: 0 }, // time - Opcode::LoadGlobalVar { off: 1 }, // dt - Opcode::Op2 { op: Op2::Add }, - Opcode::AssignCurr { off: 4 }, - Opcode::Ret, + SymbolicOpcode::LoadGlobalVar { off: 0 }, + SymbolicOpcode::LoadGlobalVar { off: 1 }, + SymbolicOpcode::Op2 { op: Op2::Add }, + SymbolicOpcode::AssignCurr { + var: sref("births", 0), + }, + SymbolicOpcode::Ret, ], }; - let sym = symbolize_bytecode(&bc, &ReverseOffsetMap::from_layout(&layout)).unwrap(); let resolved = resolve_bytecode(&sym, &layout).unwrap(); + assert_eq!( + resolved.code, + vec![ + Opcode::LoadGlobalVar { off: 0 }, + Opcode::LoadGlobalVar { off: 1 }, + Opcode::Op2 { op: Op2::Add }, + Opcode::AssignCurr { off: 4 }, + Opcode::Ret, + ] + ); + } - for (i, (orig, res)) in bc.code.iter().zip(resolved.code.iter()).enumerate() { - assert!( - opcode_eq(orig, res), - "opcode mismatch at index {}: {:?} vs {:?}", - i, - orig, - res - ); - } + /// A bytecode stream deeper than the VM's fixed arithmetic stack is a + /// compile error, not an abort: `resolve_bytecode` is the single place + /// concrete bytecode is born, so it is where the VM's unchecked stack + /// access is proven sound. + #[test] + fn test_resolution_rejects_stack_overflow() { + let layout = simple_layout(); + + let mut code: Vec = + vec![SymbolicOpcode::LoadConstant { id: 0 }; STACK_CAPACITY]; + code.push(SymbolicOpcode::Ret); + let sym = SymbolicByteCode { + literals: vec![1.0], + code, + }; + + let err = resolve_bytecode(&sym, &layout).unwrap_err(); + assert!( + err.contains("exceeding VM capacity"), + "expected a stack-capacity error, got: {err}" + ); } #[test] - fn test_static_view_roundtrip_var() { + fn test_static_view_resolution_var() { let layout = simple_layout(); - let rmap = ReverseOffsetMap::from_layout(&layout); - let view = StaticArrayView { - base_off: 5, - is_temp: false, + let sym = SymbolicStaticView { + base: SymStaticViewBase::Var(sref("population", 0)), dims: SmallVec::from_slice(&[3]), strides: SmallVec::from_slice(&[1]), offset: 0, @@ -2666,24 +2908,19 @@ mod tests { dim_ids: SmallVec::from_slice(&[0]), }; - let sym = symbolize_static_view(&view, &rmap).unwrap(); - assert!(matches!(sym.base, SymStaticViewBase::Var(_))); - let resolved = resolve_static_view(&sym, &layout).unwrap(); - assert_eq!(view.base_off, resolved.base_off); - assert_eq!(view.is_temp, resolved.is_temp); - assert_eq!(view.dims, resolved.dims); - assert_eq!(view.offset, resolved.offset); + assert_eq!(resolved.base_off, 5); + assert!(!resolved.is_temp); + assert_eq!(resolved.dims, sym.dims); + assert_eq!(resolved.offset, 0); } #[test] - fn test_static_view_roundtrip_temp() { + fn test_static_view_resolution_temp() { let layout = simple_layout(); - let rmap = ReverseOffsetMap::from_layout(&layout); - let view = StaticArrayView { - base_off: 7, - is_temp: true, + let sym = SymbolicStaticView { + base: SymStaticViewBase::Temp(7), dims: SmallVec::from_slice(&[2, 3]), strides: SmallVec::from_slice(&[3, 1]), offset: 0, @@ -2691,67 +2928,53 @@ mod tests { dim_ids: SmallVec::from_slice(&[0, 1]), }; - let sym = symbolize_static_view(&view, &rmap).unwrap(); - assert!(matches!(sym.base, SymStaticViewBase::Temp(7))); - let resolved = resolve_static_view(&sym, &layout).unwrap(); - assert_eq!(view.base_off, resolved.base_off); - assert_eq!(view.is_temp, resolved.is_temp); + assert_eq!(resolved.base_off, 7); + assert!(resolved.is_temp); } #[test] - fn test_module_decl_roundtrip() { + fn test_module_decl_resolution() { let layout = simple_layout(); - let rmap = ReverseOffsetMap::from_layout(&layout); - let decl = ModuleDeclaration { + let sym = SymbolicModuleDecl { model_name: Ident::new("sub_model"), input_set: BTreeSet::new(), - off: 4, + var: sref("births", 0), }; - let sym = symbolize_module_decl(&decl, &rmap).unwrap(); - assert_eq!(sym.var.name, "births"); - let resolved = resolve_module_decl(&sym, &layout).unwrap(); - assert_eq!(decl.off, resolved.off); - assert_eq!(decl.model_name, resolved.model_name); + assert_eq!(resolved.off, 4); + assert_eq!(resolved.model_name, Ident::new("sub_model")); } + /// The property the whole symbolic layer exists for: one fragment, two + /// layouts. Adding a variable ahead of `population` moves its slot, and the + /// SAME symbolic bytecode resolves to the new slot with no recompilation. #[test] fn test_layout_independence() { - // Symbolize with one layout, resolve with a different layout. - // The symbolic bytecodes should produce correct concrete offsets - // for the new layout. - - let layout1 = simple_layout(); // births=4, population=5 - - let bc = ByteCode { + let sym = SymbolicByteCode { literals: vec![0.1], code: vec![ - Opcode::LoadVar { off: 5 }, // population in layout1 - Opcode::LoadConstant { id: 0 }, - Opcode::Op2 { op: Op2::Mul }, - Opcode::AssignCurr { off: 4 }, // births in layout1 - Opcode::Ret, + SymbolicOpcode::LoadVar { + var: sref("population", 0), + }, + SymbolicOpcode::LoadConstant { id: 0 }, + SymbolicOpcode::Op2 { op: Op2::Mul }, + SymbolicOpcode::AssignCurr { + var: sref("births", 0), + }, + SymbolicOpcode::Ret, ], }; - // Symbolize using layout1 - let sym = symbolize_bytecode(&bc, &ReverseOffsetMap::from_layout(&layout1)).unwrap(); + // layout1: births=4, population=5 + let resolved1 = resolve_bytecode(&sym, &simple_layout()).unwrap(); + assert_eq!(resolved1.code[0], Opcode::LoadVar { off: 5 }); + assert_eq!(resolved1.code[3], Opcode::AssignCurr { off: 4 }); - // Verify symbolic opcodes reference variable names, not offsets - assert_eq!( - sym.code[0], - SymbolicOpcode::LoadVar { - var: SymVarRef { - name: "population".to_string(), - element_offset: 0 - } - } - ); - - // Create layout2 with different offsets (swapped positions + new variable) + // layout2 inserts `growth_rate` alphabetically between births and + // population, so population moves to 6 and births stays at 4. let mut entries2 = HashMap::new(); entries2.insert("time".to_string(), LayoutEntry { offset: 0, size: 1 }); entries2.insert("dt".to_string(), LayoutEntry { offset: 1, size: 1 }); @@ -2760,7 +2983,6 @@ mod tests { LayoutEntry { offset: 2, size: 1 }, ); entries2.insert("final_time".to_string(), LayoutEntry { offset: 3, size: 1 }); - // New variable inserted alphabetically between births and population entries2.insert("births".to_string(), LayoutEntry { offset: 4, size: 1 }); entries2.insert( "growth_rate".to_string(), @@ -2769,13 +2991,9 @@ mod tests { entries2.insert("population".to_string(), LayoutEntry { offset: 6, size: 1 }); let layout2 = VariableLayout::new(entries2, 7); - // Resolve using layout2 - let resolved = resolve_bytecode(&sym, &layout2).unwrap(); - - // population is now at offset 6 (was 5) - assert!(opcode_eq(&resolved.code[0], &Opcode::LoadVar { off: 6 })); - // births is still at offset 4 - assert!(opcode_eq(&resolved.code[3], &Opcode::AssignCurr { off: 4 })); + let resolved2 = resolve_bytecode(&sym, &layout2).unwrap(); + assert_eq!(resolved2.code[0], Opcode::LoadVar { off: 6 }); + assert_eq!(resolved2.code[3], Opcode::AssignCurr { off: 4 }); } #[test] @@ -2799,39 +3017,20 @@ mod tests { assert_eq!(offsets, vec![5, 6, 7]); } - // Helper: compare opcodes for equality. - // Opcode doesn't derive PartialEq, so we compare via Debug representation. - #[cfg(feature = "debug-derive")] - fn opcode_eq(a: &Opcode, b: &Opcode) -> bool { - format!("{:?}", a) == format!("{:?}", b) - } - - // When debug-derive is not enabled, compare by encoding a known - // discriminant + payload check for the opcodes we use in tests. - #[cfg(not(feature = "debug-derive"))] - fn opcode_eq(a: &Opcode, b: &Opcode) -> bool { - // Use the symbolize/resolve roundtrip property: if both opcodes - // symbolize to the same SymbolicOpcode, they are equal. - // We need a layout that covers all offsets used in tests. - let mut entries = HashMap::new(); - for off in 0..20 { - entries.insert( - format!("__test_var_{}", off), - LayoutEntry { - offset: off, - size: 1, - }, - ); - } - let layout = VariableLayout::new(entries, 20); - let rmap = ReverseOffsetMap::from_layout(&layout); - - let sym_a = symbolize_opcode(a, &rmap); - let sym_b = symbolize_opcode(b, &rmap); - match (sym_a, sym_b) { - (Ok(sa), Ok(sb)) => sa == sb, - _ => false, - } + /// Assert that `sym` resolves against `simple_layout()` to `expected`. + /// + /// This is the shape every opcode-family test takes now that resolution is + /// the only direction of travel: build the symbolic opcode the compiler + /// emits, resolve it, and pin the concrete opcode the VM will execute. + #[track_caller] + fn assert_resolves(sym: SymbolicOpcode, expected: Opcode) { + let layout = simple_layout(); + let resolved = resolve_opcode(&sym, &layout) + .unwrap_or_else(|e| panic!("resolve failed for {sym:?}: {e}")); + assert!( + resolved == expected, + "resolve produced the wrong opcode for {sym:?}" + ); } // ==================================================================== @@ -2851,7 +3050,18 @@ mod tests { } } - fn compile_and_roundtrip(dm_project: &crate::datamodel::Project, model_name: &str) { + /// Compile a real model through the production incremental path and check + /// that every address in the result is one the model's layout actually + /// assigns. + /// + /// This replaced a `symbolize(compiled) -> resolve -> compare` roundtrip, + /// which tested that two functions inverted each other; one of them no + /// longer exists, because production never travels concrete-to-symbolic. + /// What is worth asserting now is the forward direction end to end: the + /// resolved module is addressed against the same layout the assembler + /// used, each `CompiledInitial` carries exactly the write targets its own + /// bytecode has, and each module declaration points at a variable's base. + fn compile_and_check_resolution(dm_project: &crate::datamodel::Project, model_name: &str) { let mut db = crate::db::SimlinDb::default(); let sync = crate::db::sync_from_datamodel_incremental(&mut db, dm_project, None); let sim = crate::db::compile_project_incremental(&db, sync.project, model_name) @@ -2861,148 +3071,73 @@ mod tests { let source_model = sync.models[model_name].source_model; // The root module is assembled against the root-shifted layout - // (implicit globals at fixed slots 0..3, body at +IMPLICIT_VAR_COUNT), - // so the symbolize/resolve roundtrip must use that same layout. + // (implicit globals at fixed slots 0..3, body at +IMPLICIT_VAR_COUNT). let layout = crate::db::compute_layout(&db, source_model, sync.project).root_shifted(); - let sym = symbolize_module(compiled, &layout) - .unwrap_or_else(|e| panic!("symbolize_module failed: {e}")); + assert_eq!(compiled.n_slots, layout.n_slots); - let resolved = - resolve_module(&sym, &layout).unwrap_or_else(|e| panic!("resolve_module failed: {e}")); - - // Verify structural equivalence - assert_eq!(compiled.ident, resolved.ident); - assert_eq!(compiled.n_slots, resolved.n_slots); - assert_eq!( - compiled.compiled_initials.len(), - resolved.compiled_initials.len() - ); - - // Compare initials - for (orig, res) in compiled - .compiled_initials - .iter() - .zip(resolved.compiled_initials.iter()) - { - assert_eq!(orig.ident, res.ident); - assert_eq!( - orig.offsets, res.offsets, - "initial offsets mismatch for {}", - orig.ident - ); - assert_eq!(orig.bytecode.literals, res.bytecode.literals); - assert_eq!( - orig.bytecode.code.len(), - res.bytecode.code.len(), - "initial code length mismatch for {}", - orig.ident - ); - for (i, (o, r)) in orig - .bytecode - .code - .iter() - .zip(res.bytecode.code.iter()) - .enumerate() - { - assert!( - opcode_eq(o, r), - "initial opcode mismatch at index {} for {}: {:?} vs {:?}", - i, - orig.ident, - o, - r - ); - } + // Every slot a layout entry owns, so an emitted offset can be checked + // for being a real address rather than a stray integer. + let mut owned: Vec = vec![false; layout.n_slots]; + for entry in layout.entries.values() { + owned[entry.offset..entry.offset + entry.size].fill(true); } + let check = |bc: &ByteCode, what: &str| { + for op in &bc.code { + if let Some(off) = referenced_offset(op) { + assert!( + (off as usize) < layout.n_slots && owned[off as usize], + "{what}: opcode references slot {off}, which no layout entry owns" + ); + } + } + }; - // Compare flows bytecode - assert_eq!( - compiled.compiled_flows.literals, - resolved.compiled_flows.literals - ); - assert_eq!( - compiled.compiled_flows.code.len(), - resolved.compiled_flows.code.len(), - "flows code length mismatch" - ); - for (i, (o, r)) in compiled - .compiled_flows - .code - .iter() - .zip(resolved.compiled_flows.code.iter()) - .enumerate() - { - assert!( - opcode_eq(o, r), - "flows opcode mismatch at index {}: {:?} vs {:?}", - i, - o, - r + for init in compiled.compiled_initials.iter() { + check(&init.bytecode, "initials"); + assert_eq!( + init.offsets, + extract_assign_curr_offsets(&init.bytecode), + "initial `{}` carries offsets its bytecode does not write", + init.ident ); } + check(&compiled.compiled_flows, "flows"); + check(&compiled.compiled_stocks, "stocks"); - // Compare stocks bytecode - assert_eq!( - compiled.compiled_stocks.literals, - resolved.compiled_stocks.literals - ); - assert_eq!( - compiled.compiled_stocks.code.len(), - resolved.compiled_stocks.code.len(), - "stocks code length mismatch" - ); - for (i, (o, r)) in compiled - .compiled_stocks - .code - .iter() - .zip(resolved.compiled_stocks.code.iter()) - .enumerate() - { + for md in compiled.context.modules.iter() { assert!( - opcode_eq(o, r), - "stocks opcode mismatch at index {}: {:?} vs {:?}", - i, - o, - r + md.off < layout.n_slots && owned[md.off], + "module declaration for '{}' points at slot {}, which no layout entry owns", + md.model_name, + md.off ); } - - // Compare context fields - assert_eq!( - compiled.context.graphical_functions, - resolved.context.graphical_functions - ); - assert_eq!( - compiled.context.modules.len(), - resolved.context.modules.len() - ); - for (orig_md, res_md) in compiled - .context - .modules - .iter() - .zip(resolved.context.modules.iter()) - { - assert_eq!(orig_md.model_name, res_md.model_name); - assert_eq!(orig_md.off, res_md.off); - assert_eq!(orig_md.input_set, res_md.input_set); + for sv in compiled.context.static_views.iter() { + if !sv.is_temp { + assert!( + (sv.base_off as usize) < layout.n_slots && owned[sv.base_off as usize], + "static view base {} is not an owned slot", + sv.base_off + ); + } } + } - assert_eq!( - compiled.context.static_views.len(), - resolved.context.static_views.len() - ); - for (orig_sv, res_sv) in compiled - .context - .static_views - .iter() - .zip(resolved.context.static_views.iter()) - { - assert_eq!(orig_sv.base_off, res_sv.base_off); - assert_eq!(orig_sv.is_temp, res_sv.is_temp); - assert_eq!(orig_sv.dims, res_sv.dims); - assert_eq!(orig_sv.strides, res_sv.strides); - assert_eq!(orig_sv.offset, res_sv.offset); + /// The model slot an opcode reads or writes, if it has one. Mirrors the + /// nine `SymVarRef`-carrying symbolic opcode families. + fn referenced_offset(op: &Opcode) -> Option { + match op { + Opcode::LoadVar { off } + | Opcode::LoadPrev { off } + | Opcode::LoadInitial { off } + | Opcode::LoadSubscript { off } + | Opcode::AssignCurr { off } + | Opcode::AssignConstCurr { off, .. } + | Opcode::BinOpAssignCurr { off, .. } + | Opcode::BinOpAssignNext { off, .. } => Some(*off), + Opcode::PushVarViewDirect { base_off, .. } => Some(*base_off), + _ => None, } } @@ -3021,7 +3156,7 @@ mod tests { ], )], ); - compile_and_roundtrip(&dm_project, "main"); + compile_and_check_resolution(&dm_project, "main"); } #[test] @@ -3037,7 +3172,7 @@ mod tests { ], )], ); - compile_and_roundtrip(&dm_project, "main"); + compile_and_check_resolution(&dm_project, "main"); } #[test] @@ -3053,32 +3188,52 @@ mod tests { ], )], ); - compile_and_roundtrip(&dm_project, "main"); + compile_and_check_resolution(&dm_project, "main"); } + /// The resolved module's slot count comes from the layout, never from the + /// symbolic value. `SymbolicCompiledModule` carries no slot count at all + /// now, so this pins the remaining half: a bigger layout produces a bigger + /// module from the same fragments. #[test] - fn test_resolve_uses_layout_n_slots_not_symbolic() { - let dm_project = x_project( - default_sim_specs(), - &[x_model( - "main", - vec![x_aux("a", "1", None), x_aux("b", "a + 1", None)], - )], - ); - let mut db = crate::db::SimlinDb::default(); - let sync = crate::db::sync_from_datamodel_incremental(&mut db, &dm_project, None); - let sim = crate::db::compile_project_incremental(&db, sync.project, "main") - .expect("incremental compile should succeed"); - - let compiled = &sim.modules[&sim.root]; + fn test_resolve_uses_layout_n_slots() { + let layout = simple_layout(); + let sym = SymbolicCompiledModule { + ident: Ident::new("main"), + compiled_initials: vec![], + compiled_flows: SymbolicByteCode { + literals: vec![], + code: vec![ + SymbolicOpcode::LoadVar { + var: sref("population", 0), + }, + SymbolicOpcode::AssignCurr { + var: sref("births", 0), + }, + SymbolicOpcode::Ret, + ], + }, + compiled_stocks: SymbolicByteCode::default(), + graphical_functions: vec![], + module_decls: vec![], + static_views: vec![], + arrays: vec![], + dimensions: vec![], + subdim_relations: vec![], + names: vec![], + temp_offsets: vec![], + temp_total_size: 0, + dim_lists: vec![], + flows_invariant_opcode_len: 0, + }; - let source_model = sync.models["main"].source_model; - // The root module was assembled against the root-shifted layout, so - // symbolize it back with that same layout. - let layout = crate::db::compute_layout(&db, source_model, sync.project).root_shifted(); - let sym = symbolize_module(compiled, &layout).unwrap(); + assert_eq!( + resolve_module(&sym, &layout).unwrap().n_slots, + layout.n_slots + ); - // Create a layout with more slots (simulating a variable addition) + // A variable added past the end grows the layout; the same fragments + // resolve into the bigger module. let mut bigger_entries = layout.entries.clone(); bigger_entries.insert( "new_var".to_string(), @@ -3088,15 +3243,9 @@ mod tests { }, ); let bigger_layout = VariableLayout::new(bigger_entries, layout.n_slots + 1); - - let resolved = resolve_module(&sym, &bigger_layout).unwrap(); assert_eq!( - resolved.n_slots, bigger_layout.n_slots, - "resolved module should use layout's n_slots, not the stale symbolic value" - ); - assert_ne!( - resolved.n_slots, sym.n_slots, - "resolved n_slots should differ from symbolic n_slots when layout changed" + resolve_module(&sym, &bigger_layout).unwrap().n_slots, + bigger_layout.n_slots ); } @@ -3104,6 +3253,9 @@ mod tests { // u16 truncation boundary tests (issue #291) // ==================================================================== + /// A model bigger than u16 can address is rejected before assembly, but the + /// resolution primitives must handle large *usize* offsets correctly up to + /// that point rather than truncating. #[test] fn test_large_offset_static_view() { let large_off: usize = 70_000; @@ -3116,11 +3268,9 @@ mod tests { }, ); let layout = VariableLayout::new(entries, large_off + 3); - let rmap = ReverseOffsetMap::from_layout(&layout); - let view = StaticArrayView { - base_off: large_off as u32, - is_temp: false, + let sym = SymbolicStaticView { + base: SymStaticViewBase::Var(sref("big_var", 0)), dims: SmallVec::from_slice(&[3]), strides: SmallVec::from_slice(&[1]), offset: 0, @@ -3128,15 +3278,6 @@ mod tests { dim_ids: SmallVec::from_slice(&[0]), }; - let sym = symbolize_static_view(&view, &rmap).unwrap(); - match &sym.base { - SymStaticViewBase::Var(var_ref) => { - assert_eq!(var_ref.name, "big_var"); - assert_eq!(var_ref.element_offset, 0); - } - SymStaticViewBase::Temp(_) => panic!("expected Var, got Temp"), - } - let resolved = resolve_static_view(&sym, &layout).unwrap(); assert_eq!(resolved.base_off, large_off as u32); assert!(!resolved.is_temp); @@ -3154,221 +3295,251 @@ mod tests { }, ); let layout = VariableLayout::new(entries, large_off + 5); - let rmap = ReverseOffsetMap::from_layout(&layout); - let decl = ModuleDeclaration { + let sym = SymbolicModuleDecl { model_name: Ident::new("sub"), input_set: BTreeSet::new(), - off: large_off, + var: sref("big_module", 0), }; - let sym = symbolize_module_decl(&decl, &rmap).unwrap(); - assert_eq!(sym.var.name, "big_module"); - let resolved = resolve_module_decl(&sym, &layout).unwrap(); assert_eq!(resolved.off, large_off); } - #[test] - #[cfg(target_pointer_width = "64")] - fn test_module_decl_offset_overflow_is_rejected() { - let mut entries = HashMap::new(); - entries.insert("wrapped".to_string(), LayoutEntry { offset: 4, size: 1 }); - let layout = VariableLayout::new(entries, 5); - let rmap = ReverseOffsetMap::from_layout(&layout); - - let overflowing_off = (u32::MAX as usize) + 5; - let decl = ModuleDeclaration { - model_name: Ident::new("sub"), - input_set: BTreeSet::new(), - off: overflowing_off, - }; - - let err = symbolize_module_decl(&decl, &rmap).unwrap_err(); - assert!( - err.contains("does not fit in u32"), - "expected explicit overflow error, got: {err}" - ); - } - - #[test] - fn test_unmapped_offset() { - let mut entries = HashMap::new(); - entries.insert("a".to_string(), LayoutEntry { offset: 0, size: 1 }); - // Offset 1 is allocated but not mapped to any variable - let layout = VariableLayout::new(entries, 3); - let rmap = ReverseOffsetMap::from_layout(&layout); - - assert!(rmap.lookup(0).is_ok()); - let err = rmap.lookup(1).unwrap_err(); - assert!(err.contains("no variable mapped at offset")); - } - // ==================================================================== - // Opcode roundtrip coverage: passthrough opcodes + // Resolution coverage: one case per opcode family // ==================================================================== + // + // `resolve_opcode` is an exhaustive match over `SymbolicOpcode`, so a new + // variant is a compile error there; these pin that each existing arm maps + // to the right concrete opcode with its operands intact. #[test] - fn test_roundtrip_control_flow_and_builtin_opcodes() { - let layout = simple_layout(); - let rmap = ReverseOffsetMap::from_layout(&layout); - - let opcodes = vec![ - Opcode::Not {}, - Opcode::SetCond {}, - Opcode::If {}, - Opcode::LoadModuleInput { input: 3 }, - Opcode::EvalModule { id: 0, n_inputs: 2 }, - Opcode::Apply { - func: BuiltinId::Abs, - }, - Opcode::Lookup { - base_gf: 0, - table_count: 4, - mode: LookupMode::Interpolate, - }, - Opcode::Lookup { - base_gf: 1, - table_count: 1, - mode: LookupMode::Forward, - }, - Opcode::Ret, - ]; - - for op in &opcodes { - let sym = symbolize_opcode(op, &rmap).unwrap(); - let resolved = resolve_opcode(&sym, &layout).unwrap(); - assert!(opcode_eq(op, &resolved), "roundtrip failed for {:?}", sym); + fn test_resolve_control_flow_and_builtin_opcodes() { + for (sym, concrete) in [ + (SymbolicOpcode::Not {}, Opcode::Not {}), + (SymbolicOpcode::SetCond {}, Opcode::SetCond {}), + (SymbolicOpcode::If {}, Opcode::If {}), + ( + SymbolicOpcode::LoadModuleInput { input: 3 }, + Opcode::LoadModuleInput { input: 3 }, + ), + ( + SymbolicOpcode::EvalModule { id: 0, n_inputs: 2 }, + Opcode::EvalModule { id: 0, n_inputs: 2 }, + ), + ( + SymbolicOpcode::Apply { + func: BuiltinId::Abs, + }, + Opcode::Apply { + func: BuiltinId::Abs, + }, + ), + ( + SymbolicOpcode::Lookup { + base_gf: 0, + table_count: 4, + mode: LookupMode::Interpolate, + }, + Opcode::Lookup { + base_gf: 0, + table_count: 4, + mode: LookupMode::Interpolate, + }, + ), + ( + SymbolicOpcode::Lookup { + base_gf: 1, + table_count: 1, + mode: LookupMode::Forward, + }, + Opcode::Lookup { + base_gf: 1, + table_count: 1, + mode: LookupMode::Forward, + }, + ), + (SymbolicOpcode::Ret, Opcode::Ret), + ] { + assert_resolves(sym, concrete); } } #[test] - fn test_roundtrip_view_stack_opcodes() { - let layout = simple_layout(); - let rmap = ReverseOffsetMap::from_layout(&layout); - - let opcodes = vec![ - Opcode::PushVarView { - base_off: 4, - dim_list_id: 0, - }, - Opcode::PushTempView { - temp_id: 1, - dim_list_id: 2, - }, - Opcode::PushStaticView { view_id: 3 }, - Opcode::PushVarViewDirect { - base_off: 5, - dim_list_id: 1, - }, - Opcode::ViewSubscriptConst { - dim_idx: 0, - index: 2, - }, - Opcode::ViewSubscriptDynamic { dim_idx: 1 }, - Opcode::ViewRange { - dim_idx: 0, - start: 1, - end: 5, - }, - Opcode::ViewRangeDynamic { dim_idx: 2 }, - Opcode::ViewStarRange { - dim_idx: 0, - subdim_relation_id: 7, - }, - Opcode::ViewWildcard { dim_idx: 1 }, - Opcode::ViewTranspose {}, - Opcode::PopView {}, - Opcode::DupView {}, - ]; - - for op in &opcodes { - let sym = symbolize_opcode(op, &rmap).unwrap(); - let resolved = resolve_opcode(&sym, &layout).unwrap(); - assert!(opcode_eq(op, &resolved), "roundtrip failed for {:?}", sym); + fn test_resolve_view_stack_opcodes() { + for (sym, concrete) in [ + ( + SymbolicOpcode::PushStaticView { view_id: 3 }, + Opcode::PushStaticView { view_id: 3 }, + ), + ( + SymbolicOpcode::PushVarViewDirect { + var: sref("population", 0), + dim_list_id: 1, + }, + Opcode::PushVarViewDirect { + base_off: 5, + dim_list_id: 1, + }, + ), + ( + SymbolicOpcode::ViewSubscriptDynamic { dim_idx: 1 }, + Opcode::ViewSubscriptDynamic { dim_idx: 1 }, + ), + ( + SymbolicOpcode::ViewRangeDynamic { dim_idx: 2 }, + Opcode::ViewRangeDynamic { dim_idx: 2 }, + ), + (SymbolicOpcode::PopView {}, Opcode::PopView {}), + ] { + assert_resolves(sym, concrete); } } #[test] - fn test_roundtrip_temp_and_subscript_opcodes() { - let layout = simple_layout(); - let rmap = ReverseOffsetMap::from_layout(&layout); - - let opcodes = vec![ - Opcode::LoadTempConst { - temp_id: 0, - index: 3, - }, - Opcode::LoadTempDynamic { temp_id: 2 }, - Opcode::PushSubscriptIndex { bounds: 4 }, - Opcode::LoadSubscript { off: 5 }, - Opcode::AssignNext { off: 4 }, - ]; - - for op in &opcodes { - let sym = symbolize_opcode(op, &rmap).unwrap(); - let resolved = resolve_opcode(&sym, &layout).unwrap(); - assert!(opcode_eq(op, &resolved), "roundtrip failed for {:?}", sym); + fn test_resolve_temp_and_subscript_opcodes() { + for (sym, concrete) in [ + ( + SymbolicOpcode::LoadTempConst { + temp_id: 0, + index: 3, + }, + Opcode::LoadTempConst { + temp_id: 0, + index: 3, + }, + ), + ( + SymbolicOpcode::PushSubscriptIndex { bounds: 4 }, + Opcode::PushSubscriptIndex { bounds: 4 }, + ), + ( + SymbolicOpcode::LoadSubscript { + var: sref("population", 0), + }, + Opcode::LoadSubscript { off: 5 }, + ), + ( + SymbolicOpcode::BinOpAssignNext { + op: Op2::Add, + var: sref("births", 0), + }, + Opcode::BinOpAssignNext { + op: Op2::Add, + off: 4, + }, + ), + ( + SymbolicOpcode::SymLoadPrev { + var: sref("population", 0), + }, + Opcode::LoadPrev { off: 5 }, + ), + ( + SymbolicOpcode::SymLoadInitial { + var: sref("births", 0), + }, + Opcode::LoadInitial { off: 4 }, + ), + ] { + assert_resolves(sym, concrete); } } #[test] - fn test_roundtrip_iteration_opcodes() { - let layout = simple_layout(); - let rmap = ReverseOffsetMap::from_layout(&layout); - - let opcodes = vec![ - Opcode::BeginIter { - write_temp_id: 0, - has_write_temp: true, - }, - Opcode::BeginIter { - write_temp_id: 0, - has_write_temp: false, - }, - Opcode::LoadIterElement {}, - Opcode::LoadIterTempElement { temp_id: 1 }, - Opcode::LoadIterViewTop {}, - Opcode::LoadIterViewAt { offset: 2 }, - Opcode::StoreIterElement {}, - Opcode::NextIterOrJump { jump_back: -5 }, - Opcode::EndIter {}, - ]; - - for op in &opcodes { - let sym = symbolize_opcode(op, &rmap).unwrap(); - let resolved = resolve_opcode(&sym, &layout).unwrap(); - assert!(opcode_eq(op, &resolved), "roundtrip failed for {:?}", sym); + fn test_resolve_iteration_opcodes() { + for (sym, concrete) in [ + ( + SymbolicOpcode::BeginIter { + write_temp_id: 0, + has_write_temp: true, + }, + Opcode::BeginIter { + write_temp_id: 0, + has_write_temp: true, + }, + ), + ( + SymbolicOpcode::BeginIter { + write_temp_id: 0, + has_write_temp: false, + }, + Opcode::BeginIter { + write_temp_id: 0, + has_write_temp: false, + }, + ), + ( + SymbolicOpcode::LoadIterViewAt { offset: 2 }, + Opcode::LoadIterViewAt { offset: 2 }, + ), + ( + SymbolicOpcode::StoreIterElement {}, + Opcode::StoreIterElement {}, + ), + ( + SymbolicOpcode::NextIterOrJump { jump_back: -5 }, + Opcode::NextIterOrJump { jump_back: -5 }, + ), + (SymbolicOpcode::EndIter {}, Opcode::EndIter {}), + ] { + assert_resolves(sym, concrete); } } #[test] - fn test_roundtrip_broadcast_and_reduction_opcodes() { - let layout = simple_layout(); - let rmap = ReverseOffsetMap::from_layout(&layout); - - let opcodes = vec![ - Opcode::ArraySum {}, - Opcode::ArrayMax {}, - Opcode::ArrayMin {}, - Opcode::ArrayMean {}, - Opcode::ArrayStddev {}, - Opcode::ArraySize {}, - Opcode::BeginBroadcastIter { - n_sources: 2, - dest_temp_id: 0, - }, - Opcode::LoadBroadcastElement { source_idx: 0 }, - Opcode::LoadBroadcastElement { source_idx: 1 }, - Opcode::StoreBroadcastElement {}, - Opcode::NextBroadcastOrJump { jump_back: -4 }, - Opcode::EndBroadcastIter {}, - ]; - - for op in &opcodes { - let sym = symbolize_opcode(op, &rmap).unwrap(); - let resolved = resolve_opcode(&sym, &layout).unwrap(); - assert!(opcode_eq(op, &resolved), "roundtrip failed for {:?}", sym); + fn test_resolve_reduction_and_vector_opcodes() { + for (sym, concrete) in [ + (SymbolicOpcode::ArraySum {}, Opcode::ArraySum {}), + (SymbolicOpcode::ArrayMax {}, Opcode::ArrayMax {}), + (SymbolicOpcode::ArrayMin {}, Opcode::ArrayMin {}), + (SymbolicOpcode::ArrayMean {}, Opcode::ArrayMean {}), + (SymbolicOpcode::ArrayStddev {}, Opcode::ArrayStddev {}), + (SymbolicOpcode::ArraySize {}, Opcode::ArraySize {}), + (SymbolicOpcode::VectorSelect {}, Opcode::VectorSelect {}), + ( + SymbolicOpcode::VectorElmMap { + write_temp_id: 1, + full_source_len: 12, + }, + Opcode::VectorElmMap { + write_temp_id: 1, + full_source_len: 12, + }, + ), + ( + SymbolicOpcode::VectorSortOrder { write_temp_id: 2 }, + Opcode::VectorSortOrder { write_temp_id: 2 }, + ), + ( + SymbolicOpcode::Rank { write_temp_id: 3 }, + Opcode::Rank { write_temp_id: 3 }, + ), + ( + SymbolicOpcode::LookupArray { + base_gf: 2, + table_count: 3, + mode: LookupMode::Backward, + write_temp_id: 4, + }, + Opcode::LookupArray { + base_gf: 2, + table_count: 3, + mode: LookupMode::Backward, + write_temp_id: 4, + }, + ), + ( + SymbolicOpcode::AllocateAvailable { write_temp_id: 5 }, + Opcode::AllocateAvailable { write_temp_id: 5 }, + ), + ( + SymbolicOpcode::AllocateByPriority { write_temp_id: 6 }, + Opcode::AllocateByPriority { write_temp_id: 6 }, + ), + ] { + assert_resolves(sym, concrete); } } @@ -3377,25 +3548,23 @@ mod tests { // ==================================================================== #[test] - fn test_symbolize_opcode_out_of_range_offset() { - let mut entries = HashMap::new(); - entries.insert("a".to_string(), LayoutEntry { offset: 0, size: 1 }); - let layout = VariableLayout::new(entries, 1); - let rmap = ReverseOffsetMap::from_layout(&layout); - - let op = Opcode::LoadVar { off: 50 }; - let err = symbolize_opcode(&op, &rmap).unwrap_err(); - assert!(err.contains("out of range")); + fn test_resolve_opcode_unknown_variable() { + let layout = simple_layout(); + let err = resolve_opcode( + &SymbolicOpcode::LoadVar { + var: sref("nonexistent", 0), + }, + &layout, + ) + .unwrap_err(); + assert!(err.contains("not found in layout"), "got: {err}"); } #[test] fn test_resolve_static_view_missing_variable() { let layout = simple_layout(); let sym_view = SymbolicStaticView { - base: SymStaticViewBase::Var(SymVarRef { - name: "nonexistent".to_string(), - element_offset: 0, - }), + base: SymStaticViewBase::Var(sref("nonexistent", 0)), dims: SmallVec::from_slice(&[3]), strides: SmallVec::from_slice(&[1]), offset: 0, @@ -3413,10 +3582,7 @@ mod tests { let sym_decl = SymbolicModuleDecl { model_name: Ident::new("sub"), input_set: BTreeSet::new(), - var: SymVarRef { - name: "nonexistent".to_string(), - element_offset: 0, - }, + var: sref("nonexistent", 0), }; let err = resolve_module_decl(&sym_decl, &layout).unwrap_err(); @@ -3445,7 +3611,7 @@ mod tests { ), ], ); - compile_and_roundtrip(&dm_project, "main"); + compile_and_check_resolution(&dm_project, "main"); } // ==================================================================== @@ -3805,30 +3971,24 @@ mod tests { // End-to-end belt-and-suspenders for the Phase 5 `full_source_len` // opcode field. `test_renumber_vector_builtin_temp_ids` covers the // isolated `renumber_opcode` call; this exercises the *real merge - // path* a compiled model takes -- `symbolize_opcode` -> + // path* a compiled model takes -- codegen's symbolic opcode -> // `concatenate_fragments` (absorb + `renumber_fragment_code`) -> // `resolve_bytecode` -- with the VECTOR ELM MAP opcode merged AFTER a // temp-contributing fragment so its `write_temp_id` gets a *non-zero* // renumber offset. The invariant under test: `full_source_len` is an // absolute element count, NOT a renumber-able resource id, so it must - // come out of symbolize -> concatenate/renumber -> resolve - // byte-identical even though `write_temp_id` is offset. If - // `full_source_len` were ever (mistakenly) treated like a temp id, it - // would be shifted by `frag_a`'s temp count here and this test would - // fail; the existing renumber unit test would not catch that - // regression because it never drives the fragment merger. + // come out of concatenate/renumber -> resolve byte-identical even + // though `write_temp_id` is offset. If `full_source_len` were ever + // (mistakenly) treated like a temp id, it would be shifted by + // `frag_a`'s temp count here and this test would fail; the existing + // renumber unit test would not catch that regression because it never + // drives the fragment merger. const GENUINE_FULL_SOURCE_LEN: u32 = 6; // e.g. d[DimA,DimB] = 3 x 2 - let original = Opcode::VectorElmMap { + let symbolic_elm_map = SymbolicOpcode::VectorElmMap { write_temp_id: 0, full_source_len: GENUINE_FULL_SOURCE_LEN, }; - // The VectorElmMap opcode carries no variable offset, so an empty - // layout (=> empty ReverseOffsetMap) is sufficient for symbolization. - let empty_layout = VariableLayout::new(HashMap::new(), 0); - let rmap = ReverseOffsetMap::from_layout(&empty_layout); - let symbolic_elm_map = symbolize_opcode(&original, &rmap).unwrap(); - // frag_a is a temp-bearing fragment; frag_b carries the VectorElmMap. // #583: the plain-phase concat RECYCLES temps into one identity pool, // so frag_b's id-0 write_temp_id recycles to slot 0 (not summed past @@ -3867,8 +4027,9 @@ mod tests { }; let merged = concatenate_fragments(&[&frag_a, &frag_b], &based).unwrap(); - // Resolve back to concrete bytecode (same empty layout: the - // VectorElmMap opcode carries no SymVarRef). + // Resolve to concrete bytecode against an empty layout: the + // VectorElmMap opcode carries no variable reference. + let empty_layout = VariableLayout::new(HashMap::new(), 0); let resolved = resolve_bytecode(&merged.bytecode, &empty_layout).unwrap(); let elm_map = resolved @@ -3893,18 +4054,342 @@ mod tests { the fragment merger renumbered this opcode" ); // The invariant: full_source_len is absolute, not renumbered. It must - // survive symbolize -> concatenate/renumber -> resolve unchanged even + // survive concatenate/renumber -> resolve unchanged even // though write_temp_id was offset. assert_eq!( elm_map.1, GENUINE_FULL_SOURCE_LEN, - "full_source_len must survive the symbolize -> fragment-merge -> \ - resolve round-trip byte-identical (it is an absolute element \ + "full_source_len must survive the fragment-merge -> resolve path \ + byte-identical (it is an absolute element \ count, not a renumber-able resource id); a corrupted value would \ feed the VM a wrong full-source extent and break genuine VECTOR \ ELM MAP results" ); } + // ==================================================================== + // M3: a resource id past its bytecode type is reported, never wrapped. + // + // `GraphicalFunctionId` (#582) and `TempId` (#583) both overflowed on + // C-LEARN and both fail loud. The four `u16` resources -- literals, module + // decls, static views, dim lists -- were the unguarded members of the same + // family: `absorb_non_gf` narrowed their bases with `as u16` and + // `renumber_opcode` added them unchecked, so a merged table past 65,536 + // entries produced a wrapped id that names a real, in-range resource. The + // program runs and returns different numbers, with no diagnostic anywhere. + // + // The bound is EXACT in both directions -- a table of 65,536 entries is + // addressable, 65,537 is not -- and the tests below say so at each of the + // three places a base is computed, because the first fix stated it once per + // place and the three disagreed: the cross-phase and initials-phase + // narrowings rejected a count of exactly 65,536 that the merger accepts. + // ==================================================================== + + /// A fragment carrying `n` literals and nothing else. + fn literal_pool_frag(n: usize) -> PerVarBytecodes { + PerVarBytecodes { + symbolic: SymbolicByteCode { + literals: vec![1.0; n], + code: vec![SymbolicOpcode::LoadConstant { id: 0 }, SymbolicOpcode::Ret], + }, + graphical_functions: vec![], + module_decls: vec![], + static_views: vec![], + temp_sizes: vec![], + dim_lists: vec![], + } + } + + /// M3 for the literal pool, the cheapest `u16` resource to overflow and the + /// one whose failure is worst: a wrapped `LiteralId` names a real, in-range + /// literal, so the program runs and returns different numbers. + #[test] + fn literal_pool_past_u16_capacity_fails_loud() { + let cap = u16::MAX as usize + 1; + let a = literal_pool_frag(cap - 1); + let b = literal_pool_frag(4); + let err = concatenate_fragments(&[&a, &b], &ContextResourceCounts::default()) + .expect_err("a literal pool past u16 capacity must be reported, not wrapped"); + assert!( + err.contains("literal") && err.contains("16-bit"), + "expected a loud literal-capacity error, got: {err}" + ); + + // ...and one entry less is still fine, so the bound is exact rather than + // conservative. + let c = literal_pool_frag(1); + concatenate_fragments(&[&a, &c], &ContextResourceCounts::default()) + .expect("a pool of exactly u16::MAX + 1 literals is addressable"); + + // A fragment carrying NONE of the resource is exempt even at a full + // table: it names no id, so there is nothing that has to be + // representable. Without the exemption the base -- which this fragment + // never uses -- would be rejected for being one past `u16::MAX`. + let empty = PerVarBytecodes { + symbolic: SymbolicByteCode { + literals: vec![], + code: vec![SymbolicOpcode::Ret], + }, + graphical_functions: vec![], + module_decls: vec![], + static_views: vec![], + temp_sizes: vec![], + dim_lists: vec![], + }; + concatenate_fragments(&[&a, &c, &empty], &ContextResourceCounts::default()) + .expect("a fragment with no literals must not be bounded by a full pool"); + } + + /// A fragment carrying `n` dim lists and nothing else. + fn dim_list_frag(n: usize) -> PerVarBytecodes { + PerVarBytecodes { + symbolic: SymbolicByteCode { + literals: vec![], + code: vec![ + SymbolicOpcode::PushVarViewDirect { + var: SymVarRef::base(Ident::new("x")), + dim_list_id: 0, + }, + SymbolicOpcode::Ret, + ], + }, + graphical_functions: vec![], + module_decls: vec![], + static_views: vec![], + temp_sizes: vec![], + dim_lists: vec![vec![1u16]; n], + } + } + + /// The same bound, one level up: across PHASES, where the base is a + /// preceding phase's count rather than a running merged length. + /// + /// The literal pool cannot reach this path -- literals are phase-local, so + /// their `ctx_base` is always 0 -- which is why the exactness the test above + /// pins did not extend to the three resources that DO carry a `ctx_base`. + /// Those went through a separate narrowing (`ContextResourceCounts` held + /// `u16` fields and `assemble_module` summed them with a checked add), and + /// that narrowing rejected a count of exactly `U16_ID_CAPACITY` outright -- + /// disagreeing with `resource_base`, which accepts a table of exactly that + /// size because every id in it is `<= u16::MAX`. A model whose initials + /// filled the table and whose flows and stocks then named no dim list at all + /// was rejected with every one of its ids valid. + #[test] + fn a_full_ctx_base_bounds_only_the_phases_that_use_it() { + let cap = U16_ID_CAPACITY; + let full = dim_list_frag(cap); + let one = dim_list_frag(1); + let none = dim_list_frag(0); + + // A phase count is just a count: filling the table exactly is legal, and + // saying so is not the same as handing out an id past the end. + let at_capacity = ContextResourceCounts::from_fragments(&[&full]); + assert_eq!(at_capacity.dim_lists, cap); + + // A following phase that names no dim list assigns no id, so it merges. + concatenate_fragments(&[&none], &at_capacity) + .expect("a phase carrying no dim lists is not bounded by a full table"); + + // One that names even a single dim list would need id 65,536. + let err = concatenate_fragments(&[&one], &at_capacity) + .expect_err("a dim list id past u16 capacity must be reported, not wrapped"); + assert!( + err.contains("dimension list") && err.contains("16-bit"), + "expected a loud dim-list-capacity error, got: {err}" + ); + + // ...and the bound is exact from below: one entry short of capacity, the + // following phase's single entry is id 65,535 and merges. + let nearly = ContextResourceCounts::from_fragments(&[&dim_list_frag(cap - 1)]); + concatenate_fragments(&[&one], &nearly) + .expect("the last addressable dim list id (u16::MAX) must merge"); + + // The same exactness within one phase's merged table, so the two halves + // of the bound are stated against the same resource. + concatenate_fragments( + &[&dim_list_frag(cap - 1), &one], + &ContextResourceCounts::default(), + ) + .expect("a merged table of exactly u16::MAX + 1 dim lists is addressable"); + concatenate_fragments(&[&full, &one], &ContextResourceCounts::default()) + .expect_err("one dim list past a full merged table must be reported"); + concatenate_fragments(&[&full, &none], &ContextResourceCounts::default()) + .expect("a fragment with no dim lists must not be bounded by a full table"); + } + + /// The initials phase tracks its own running offsets rather than going + /// through the merger (each initial keeps its own bytecode, so each is + /// renumbered at literal offset 0), and it has to agree with the merger + /// about where the table ends. + /// + /// It did not: it narrowed each initial's count and its running total to + /// `u16` eagerly, so an initials list that filled the table exactly was + /// rejected the moment the LAST initial's count was folded in -- with + /// nothing left to assign an id to. + #[test] + fn the_initials_phase_shares_the_mergers_capacity_bound() { + let cap = U16_ID_CAPACITY; + let full = dim_list_frag(cap); + let one = dim_list_frag(1); + let none = dim_list_frag(0); + + let run = |frags: &[&PerVarBytecodes]| -> Result<(), String> { + let dedup = GfDedup::build(frags)?; + let named: Vec<(String, &PerVarBytecodes)> = frags + .iter() + .enumerate() + .map(|(i, f)| (format!("init{i}"), *f)) + .collect(); + crate::db::renumber_initials_phase(&named, &dedup).map(|_| ()) + }; + + run(&[&full]).expect("initials that fill the dim-list table exactly are addressable"); + run(&[&full, &none]).expect("an initial naming no dim list is not bounded by a full table"); + let err = run(&[&full, &one]) + .expect_err("an initial needing dim list id 65,536 must be reported"); + assert!( + err.contains("dimension list") && err.contains("16-bit"), + "expected a loud dim-list-capacity error, got: {err}" + ); + } + + /// The all-phases aggregation of the shared context tables must NOT bound + /// the model's literal count, because it retains no literal pool. + /// + /// Each compiled initial keeps its own pool and the flows and stocks phases + /// keep one each; the aggregation exists only for the module-decl / + /// static-view / temp / dim-list tables those three phases' ids index. A + /// full merge over every fragment additionally built a literal pool spanning + /// the whole model and threw it away -- but not before `resource_base` + /// bounded it, so a model whose every RETAINED pool was comfortably + /// addressable stopped assembling once the three summed past capacity. + #[test] + fn the_all_phases_aggregation_does_not_bound_the_literal_pool() { + let cap = U16_ID_CAPACITY; + // Two fragments that together overrun the literal id space. + let full = literal_pool_frag(cap); + let one = literal_pool_frag(1); + let frags = [&full, &one]; + let dedup = GfDedup::build(&frags).expect("no GFs to dedup"); + + // Merging them into ONE stream assigns literal id 65,536 and is an + // error -- that pool would be retained, so the bound is right. + let err = + concatenate_fragments_with_gf(&frags, &ContextResourceCounts::default(), &dedup, 0) + .expect_err("a retained pool past capacity must be reported"); + assert!( + err.contains("literal") && err.contains("16-bit"), + "expected a loud literal-capacity error, got: {err}" + ); + + // Aggregating their context side-channels assigns no literal id at all, + // so the same fragments are fine. + let side = merge_context_side_channels(&frags, &ContextResourceCounts::default(), &dedup) + .expect("the side-channel aggregation must not be bounded by a literal pool"); + assert!( + side.module_decls.is_empty() && side.static_views.is_empty(), + "these fragments carry no context resources; only literals" + ); + } + + /// The same property end-to-end, through `assemble_module` on a real model, + /// so a future refactor that points the all-phases aggregation back at a + /// full merge fails here rather than only at the unit level. + /// + /// Reaching the bound honestly would need a model with 65,537 literals, + /// which is orders of magnitude outside the per-test time budget, so the + /// capacity is shrunk instead (`docs/dev/rust.md#test-time-budgets`, the + /// repo's stated preference over building a fixture large enough to trip a + /// production threshold). Five scalar stocks give one literal per initial + /// (five pools of one), five in the flows pool and five in the stocks pool + /// -- fifteen in aggregate. At a capacity of 8 every RETAINED pool fits and + /// only the discarded aggregate does not. + /// + /// The second half is what stops this from being a test that merely proves + /// a check was deleted: at a capacity of 4 the flows pool genuinely does not + /// fit, and assembly must still fail. + #[test] + fn assembly_bounds_the_retained_pools_and_not_the_aggregate() { + use crate::test_common::TestProject; + + let model = || { + let mut p = TestProject::new("literal_capacity"); + for i in 0..5 { + p = p + .stock( + &format!("s{i}"), + &format!("{}", i + 1), + &[&format!("f{i}")], + &[], + None, + ) + .flow(&format!("f{i}"), &format!("{}", 10 + i), None); + } + p + }; + + { + let _cap = IdCapacityGuard::new(8); + model() + .compile_incremental() + .expect("every retained literal pool fits; only the discarded aggregate does not"); + } + + { + let _cap = IdCapacityGuard::new(4); + let err = model() + .compile_incremental() + .expect_err("the flows phase's own five-literal pool does not fit in four ids"); + assert!( + format!("{err:?}").contains("literal"), + "expected the failure to name the literal pool, got: {err:?}" + ); + } + } + + /// M3 for the per-opcode add. `renumber_opcode` can be reached with a base the + /// merger did not compute -- `assemble_module`'s per-initial loop tracks its + /// own running module / view / dim-list offsets -- so the add is checked there + /// too rather than relying on the merger's table bound alone. + #[test] + fn renumber_opcode_u16_addition_overflow_is_loud() { + let cases: Vec<(SymbolicOpcode, &str, [u16; 4])> = vec![ + ( + SymbolicOpcode::LoadConstant { id: u16::MAX }, + "LiteralId", + [1, 0, 0, 0], + ), + ( + SymbolicOpcode::EvalModule { + id: u16::MAX, + n_inputs: 0, + }, + "ModuleId", + [0, 1, 0, 0], + ), + ( + SymbolicOpcode::PushStaticView { view_id: u16::MAX }, + "ViewId", + [0, 0, 1, 0], + ), + ( + SymbolicOpcode::PushVarViewDirect { + var: SymVarRef::base(Ident::new("x")), + dim_list_id: u16::MAX, + }, + "DimListId", + [0, 0, 0, 1], + ), + ]; + for (op, label, [lit, md, vw, dl]) in cases { + let err = renumber_opcode(&op, lit, &[], md, vw, 0, dl) + .expect_err("an id at u16::MAX plus a non-zero base must be reported"); + assert!( + err.contains(label) && err.contains("overflow"), + "expected a loud {label} overflow, got: {err}" + ); + } + } + #[test] fn test_renumber_opcode_u8_addition_overflow() { // temp_off=200 fits in u8, but base temp_id=100 + 200 = 300 overflows u8 @@ -3918,17 +4403,18 @@ mod tests { } // ==================================================================== - // #582: cross-fragment graphical-function de-duplication. + // #582, M4: cross-fragment graphical-function de-duplication. // // `concatenate_fragments` previously appended every fragment's // `graphical_functions` with no de-duplication (the flat running // `gf_offset = merged_gf.len()`), so a dependency arrayed GF referenced // by N consumer fragments duplicated N times and `renumber_opcode`'s // `gf_off > u8::MAX` guard tripped once the duplicated count crossed - // 255 -- even though the *distinct* count is small. The monolithic - // `Compiler::new` (codegen.rs) carries each variable's GF list exactly - // once (its `module.tables` map is keyed by ident), so the incremental - // path was incorrect-by-omission, not merely capacity-limited. + // 255 -- even though the *distinct* count is small. Codegen lays each + // table-bearing variable's tables down exactly once per fragment (its + // `tables` map is keyed by ident); the duplication was entirely ACROSS + // fragments, which is why de-duplicating on block content is a complete + // fix rather than a capacity workaround. // ==================================================================== /// Build a single-`Lookup` fragment carrying one scalar GF table whose @@ -4190,20 +4676,26 @@ mod tests { } // ==================================================================== - // #583: match the monolithic temp recycling in the plain-phase concat. + // #583, M5: the sequential concat shares temp slots by IDENTITY. // - // The monolithic `Module::compile` flattens every variable's exprs into - // one runlist and max-merges their temps via a `HashMap` - // (`compiler/mod.rs`): since each variable's temps are 0-based scratch - // that die at that variable's runlist-segment end, variable A's temp 0 - // and variable B's temp 0 collapse to ONE global slot 0. The plain-phase - // incremental concat must produce the SAME identity-recycled pool (one - // shared pool across initials/flows/stocks), not a per-fragment SUM -- - // summing both wastes slots AND, with a non-zero phase ctx_base, drives - // the renumbered `temp_id` past `u8::MAX` (the C-LEARN - // `temp offset ... exceeds TempId capacity` failure). `combine_scc_fragment` - // stays on the disjoint (sum) path because its per-element segments - // interleave (overlapping live ranges) -- see `db/combined_fragment_tests`. + // A fragment's temps are 0-based scratch that dies at the end of that + // fragment's runlist segment, and `concatenate_fragments_with_gf` emits + // each fragment as one contiguous run, so no two fragments' temp uses + // interleave and fragment A's temp 0 may occupy the same slot as fragment + // B's temp 0 -- sized to the MAX of its users, since each of them holds it + // alone for the length of its own segment. That sharing is not an + // optimization: `TempId` is `u8`, and summing 0-based per-fragment counts + // across a model overflows it (the C-LEARN + // `temp offset ... exceeds TempId capacity` failure, #583). + // `combine_scc_fragment` interleaves per-element segments, so its members' + // live ranges DO overlap and it stays on the disjoint `Sum` path -- see + // `db/combined_fragment_tests`. + // + // These fixtures pin the rule on hand-built shapes; the general statement + // (over arbitrary fragment sets, including the non-interleaving property + // that is the actual justification) is + // `symbolic_merge_proptest::recycle_shares_by_identity_and_sizes_by_max` + // and `merged_temp_slot_uses_never_interleave`. // ==================================================================== /// Build a single-variable-shaped fragment carrying a `VectorSortOrder` @@ -4229,51 +4721,30 @@ mod tests { } } - /// The monolithic `Module::compile` temp count: the keyed max-merge of - /// every fragment's `(temp_id, size)` pairs (`compiler/mod.rs` flattens - /// every variable's exprs and merges into one `HashMap`). - /// `n_temps` is the number of distinct ids (== `max_id + 1` for the - /// dense 0-based ids real per-variable fragments produce). - fn monolithic_n_temps(frags: &[&PerVarBytecodes]) -> usize { - let mut map: HashMap = HashMap::new(); - for frag in frags { - for (id, size) in &frag.temp_sizes { - let e = map.entry(*id).or_insert(0); - *e = (*e).max(*size); - } - } - map.len() - } - #[test] - fn test_concatenate_recycles_temps_to_match_monolithic() { + fn test_concatenate_recycles_three_id_zero_temps_onto_one_slot() { // Three per-variable fragments, each with its own 0-based temp 0. - // Monolithic recycles them to ONE slot (n_temps == 1). The plain- - // phase concat must do the same: a single shared identity pool, not - // three summed slots. + // They are emitted as three disjoint contiguous runs, so all three may + // hold ONE slot in turn: the pool is one slot wide, sized to the + // largest of them. let frag_a = sort_order_temp_frag(0, 4); let frag_b = sort_order_temp_frag(0, 8); let frag_c = sort_order_temp_frag(0, 2); let refs: Vec<&PerVarBytecodes> = vec![&frag_a, &frag_b, &frag_c]; - let expected = monolithic_n_temps(&refs); - assert_eq!(expected, 1, "monolithic recycles three id-0 temps to one"); - let no_base = ContextResourceCounts::default(); let merged = concatenate_fragments(&refs, &no_base).unwrap(); - // The recycled pool has exactly `expected` slots, and the surviving - // slot's size is the MAX over the fragments that used it (8), since - // they share the storage (the monolithic keyed max-merge). assert_eq!( merged.temp_offsets.len(), - expected, - "incremental plain-phase temp count must EQUAL the monolithic \ - Module::compile n_temps (recycle, not sum)" + 1, + "three fragments' id-0 temps must recycle onto one shared slot, \ + not sum to three" ); assert_eq!( merged.temp_total_size, 8, - "the recycled slot's size is the max over the fragments using it" + "a shared slot must be sized to the LARGEST of its users, or the \ + fragment needing 8 elements would write past its storage" ); // Every renumbered temp opcode must resolve in-range (index < pool). @@ -4291,9 +4762,9 @@ mod tests { #[test] fn test_concatenate_temp_recycle_distinct_ids_max_merge() { - // A fragment using temp ids {0, 1} and another using {0}. Monolithic - // merges to ids {0, 1} (2 slots, sizes max-merged per id). The - // plain-phase concat must match: 2 slots, not 3 summed. + // A fragment using temp ids {0, 1} and another using {0}: the pool is + // the UNION of the ids (2 slots), each sized to the max of the + // fragments using that id -- not 3 summed slots. let frag_a = PerVarBytecodes { symbolic: SymbolicByteCode { literals: vec![], @@ -4315,11 +4786,12 @@ mod tests { let frag_b = sort_order_temp_frag(0, 8); let refs: Vec<&PerVarBytecodes> = vec![&frag_a, &frag_b]; - let expected = monolithic_n_temps(&refs); - assert_eq!(expected, 2, "ids {{0,1}} merged with {{0}} -> 2 distinct"); - let merged = concatenate_fragments(&refs, &ContextResourceCounts::default()).unwrap(); - assert_eq!(merged.temp_offsets.len(), expected); + assert_eq!( + merged.temp_offsets.len(), + 2, + "ids {{0,1}} unioned with {{0}} -> 2 slots" + ); // Slot 0 size = max(4, 8) = 8; slot 1 size = 6. assert_eq!(merged.temp_total_size, 8 + 6); diff --git a/src/simlin-engine/src/compiler/symbolic_builder_tests.rs b/src/simlin-engine/src/compiler/symbolic_builder_tests.rs new file mode 100644 index 000000000..becf4bfac --- /dev/null +++ b/src/simlin-engine/src/compiler/symbolic_builder_tests.rs @@ -0,0 +1,590 @@ +// Copyright 2026 The Simlin Authors. All rights reserved. +// Use of this source code is governed by the Apache License, +// Version 2.0, that can be found in the LICENSE file. + +//! Tests for the symbolic bytecode builder and its peephole optimizer -- the +//! literal pool, the emit-time next-assign fusion, and the pairwise +//! superinstruction fusion with its jump-offset fixup. +//! +//! These moved here wholesale when codegen started emitting symbolic opcodes +//! (GH #964): the builder and the peephole are address-independent, so they now +//! run before resolution, and there is no concrete-opcode twin of either left +//! to test. + +use super::*; + +/// The reference for slot `n` of the old concrete tests, as a distinct +/// variable name. The peephole never inspects a reference beyond copying it, +/// so all these assertions need is that distinct operands stay distinguishable +/// and survive fusion intact. +fn v(n: u16) -> SymVarRef { + SymVarRef::base(crate::common::Ident::new(&format!("v{n}"))) +} + +#[test] +fn test_memoizing_interning() { + let mut bytecode = SymbolicByteCodeBuilder::default(); + let a1 = bytecode.intern_literal(1.0); + let b1 = bytecode.intern_literal(1.01); + let b2 = bytecode.intern_literal(1.01); + let b3 = bytecode.intern_literal(1.01); + let a2 = bytecode.intern_literal(1.0); + let b4 = bytecode.intern_literal(1.01); + + assert_eq!(a1, a2); + assert_eq!(b1, b2); + assert_eq!(b1, b3); + assert_eq!(b1, b4); + assert_ne!(a1, b1); + + let bytecode = bytecode.finish(); + assert_eq!(2, bytecode.literals.len()); +} + +#[test] +fn test_push_named_literal_no_dedup() { + let mut builder = SymbolicByteCodeBuilder::default(); + let a = builder.push_named_literal(0.1); + let b = builder.push_named_literal(0.1); + let c = builder.push_named_literal(0.1); + + assert_ne!(a, b); + assert_ne!(b, c); + assert_ne!(a, c); +} + +#[test] +#[should_panic(expected = "jump at pc 0 targets")] +fn test_peephole_panics_on_out_of_bounds_jump_target() { + // A jump that targets beyond the code length indicates a compiler bug + let mut bc = SymbolicByteCode { + literals: vec![], + code: vec![SymbolicOpcode::NextIterOrJump { jump_back: 10 }], + }; + bc.peephole_optimize(); +} + +#[test] +fn test_peephole_empty_bytecode() { + let mut bc = SymbolicByteCode { + code: vec![], + literals: vec![], + }; + bc.peephole_optimize(); + assert!(bc.code.is_empty()); +} + +#[test] +fn test_peephole_single_instruction() { + let mut bc = SymbolicByteCode { + code: vec![SymbolicOpcode::Ret], + literals: vec![], + }; + bc.peephole_optimize(); + assert_eq!(bc.code.len(), 1); + assert!(matches!(bc.code[0], SymbolicOpcode::Ret)); +} + +#[test] +fn test_peephole_no_fusible_patterns() { + let mut bc = SymbolicByteCode { + code: vec![ + SymbolicOpcode::LoadVar { var: v(0) }, + SymbolicOpcode::LoadVar { var: v(1) }, + SymbolicOpcode::Not {}, + SymbolicOpcode::Ret, + ], + literals: vec![], + }; + bc.peephole_optimize(); + assert_eq!(bc.code.len(), 4); + assert_eq!(bc.code[0], SymbolicOpcode::LoadVar { var: v(0) }); + assert_eq!(bc.code[1], SymbolicOpcode::LoadVar { var: v(1) }); + assert!(matches!(bc.code[2], SymbolicOpcode::Not {})); + assert!(matches!(bc.code[3], SymbolicOpcode::Ret)); +} + +#[test] +fn test_peephole_load_constant_assign_curr_fusion() { + let mut bc = SymbolicByteCode { + code: vec![ + SymbolicOpcode::LoadConstant { id: 0 }, + SymbolicOpcode::AssignCurr { var: v(5) }, + ], + literals: vec![42.0], + }; + bc.peephole_optimize(); + + assert_eq!(bc.code.len(), 1); + match &bc.code[0] { + SymbolicOpcode::AssignConstCurr { var, literal_id } => { + assert_eq!(*var, v(5)); + assert_eq!(*literal_id, 0); + } + _ => panic!("expected AssignConstCurr"), + } +} + +#[test] +fn test_peephole_op2_assign_curr_fusion() { + let mut bc = SymbolicByteCode { + code: vec![ + SymbolicOpcode::LoadVar { var: v(0) }, + SymbolicOpcode::LoadVar { var: v(1) }, + SymbolicOpcode::Op2 { op: Op2::Add }, + SymbolicOpcode::AssignCurr { var: v(2) }, + ], + literals: vec![], + }; + bc.peephole_optimize(); + + // LoadVar, LoadVar stay; Op2+AssignCurr fuse into BinOpAssignCurr + assert_eq!(bc.code.len(), 3); + assert_eq!(bc.code[0], SymbolicOpcode::LoadVar { var: v(0) }); + assert_eq!(bc.code[1], SymbolicOpcode::LoadVar { var: v(1) }); + match &bc.code[2] { + SymbolicOpcode::BinOpAssignCurr { op, var } => { + assert!(matches!(op, Op2::Add)); + assert_eq!(*var, v(2)); + } + _ => panic!("expected BinOpAssignCurr"), + } +} + +/// The next-value assign is fused at EMIT time (there is no un-fused +/// `AssignNext` opcode for the peephole to fuse), so this pins the +/// builder helper codegen uses. +#[test] +fn test_builder_fuses_trailing_op2_into_assign_next() { + let mut builder = SymbolicByteCodeBuilder::default(); + builder.push_opcode(SymbolicOpcode::LoadVar { var: v(0) }); + builder.push_opcode(SymbolicOpcode::LoadVar { var: v(1) }); + builder.push_opcode(SymbolicOpcode::Op2 { op: Op2::Mul }); + assert!(builder.fuse_trailing_op2_into_assign_next(&v(3))); + builder.push_opcode(SymbolicOpcode::Ret); + + let bc = builder.finish(); + assert_eq!(bc.code.len(), 4); + match &bc.code[2] { + SymbolicOpcode::BinOpAssignNext { op, var } => { + assert!(matches!(op, Op2::Mul)); + assert_eq!(*var, v(3)); + } + _ => panic!("expected BinOpAssignNext"), + } +} + +/// The other half of the contract: when the operand walk did NOT end in +/// an `Op2` -- the shape an eventual `non_negative` implementation +/// (GH #545) would produce by wrapping the stock update in a builtin -- +/// the helper refuses and emits nothing, so codegen can raise a typed +/// error instead of silently dropping the store. +#[test] +fn test_builder_refuses_assign_next_fusion_without_trailing_op2() { + let mut builder = SymbolicByteCodeBuilder::default(); + builder.push_opcode(SymbolicOpcode::LoadVar { var: v(0) }); + builder.push_opcode(SymbolicOpcode::Apply { + func: BuiltinId::Max, + }); + assert!(!builder.fuse_trailing_op2_into_assign_next(&v(3))); + assert_eq!(builder.len(), 2, "a refused fusion must emit nothing"); + + // ...and an empty stream is refused too, rather than panicking. + let mut empty = SymbolicByteCodeBuilder::default(); + assert!(!empty.fuse_trailing_op2_into_assign_next(&v(0))); + assert_eq!(empty.len(), 0); +} + +#[test] +fn test_peephole_all_op2_variants_fuse() { + // Verify every Op2 variant can be fused with AssignCurr + let ops = [ + Op2::Add, + Op2::Sub, + Op2::Mul, + Op2::Div, + Op2::Exp, + Op2::Mod, + Op2::Gt, + Op2::Gte, + Op2::Lt, + Op2::Lte, + Op2::Eq, + Op2::And, + Op2::Or, + ]; + for op in ops { + let mut bc = SymbolicByteCode { + code: vec![ + SymbolicOpcode::Op2 { op }, + SymbolicOpcode::AssignCurr { var: v(10) }, + ], + literals: vec![], + }; + bc.peephole_optimize(); + assert_eq!(bc.code.len(), 1, "failed for op variant"); + assert!(matches!(bc.code[0], SymbolicOpcode::BinOpAssignCurr { .. })); + } +} + +#[test] +fn test_peephole_multiple_fusions() { + // Two independent fusion opportunities in sequence + let mut bc = SymbolicByteCode { + code: vec![ + SymbolicOpcode::LoadConstant { id: 0 }, + SymbolicOpcode::AssignCurr { var: v(0) }, + SymbolicOpcode::LoadVar { var: v(1) }, + SymbolicOpcode::LoadVar { var: v(2) }, + SymbolicOpcode::Op2 { op: Op2::Sub }, + SymbolicOpcode::AssignCurr { var: v(3) }, + ], + literals: vec![1.0], + }; + bc.peephole_optimize(); + + // LoadConstant+AssignCurr -> AssignConstCurr + // LoadVar, LoadVar stay + // Op2+AssignCurr -> BinOpAssignCurr + assert_eq!(bc.code.len(), 4); + assert!(matches!(bc.code[0], SymbolicOpcode::AssignConstCurr { .. })); + assert_eq!(bc.code[1], SymbolicOpcode::LoadVar { var: v(1) }); + assert_eq!(bc.code[2], SymbolicOpcode::LoadVar { var: v(2) }); + assert!(matches!(bc.code[3], SymbolicOpcode::BinOpAssignCurr { .. })); +} + +#[test] +fn test_peephole_mixed_fusible_and_nonfusible() { + let mut bc = SymbolicByteCode { + code: vec![ + SymbolicOpcode::LoadVar { var: v(0) }, + SymbolicOpcode::Not {}, + SymbolicOpcode::LoadConstant { id: 0 }, + SymbolicOpcode::AssignCurr { var: v(1) }, + SymbolicOpcode::LoadVar { var: v(2) }, + SymbolicOpcode::Ret, + ], + literals: vec![0.0], + }; + bc.peephole_optimize(); + + // LoadVar, Not stay; LoadConstant+AssignCurr fuse; LoadVar, Ret stay + assert_eq!(bc.code.len(), 5); + assert_eq!(bc.code[0], SymbolicOpcode::LoadVar { var: v(0) }); + assert!(matches!(bc.code[1], SymbolicOpcode::Not {})); + assert!(matches!(bc.code[2], SymbolicOpcode::AssignConstCurr { .. })); + assert_eq!(bc.code[3], SymbolicOpcode::LoadVar { var: v(2) }); + assert!(matches!(bc.code[4], SymbolicOpcode::Ret)); +} + +#[test] +fn test_peephole_jump_target_prevents_fusion() { + // If instruction i+1 is a jump target, don't fuse i with i+1. + // Layout (before optimization): + // 0: LoadConstant { id: 0 } <- loop body start (jump target) + // 1: AssignCurr { off: 0 } + // 2: NextIterOrJump { jump_back: -2 } (target = 2 + (-2) = 0) + // 3: Ret + // + // Instruction 0 is a jump target, so even though 0 is LoadConstant + // and 1 is AssignCurr, we should NOT fuse them because instruction 0 + // is a jump target. Wait -- actually the check is whether i+1 is a + // jump target. Here instruction 0 IS a jump target. The optimizer checks + // `!jump_targets[i + 1]` to decide whether to fuse i with i+1. + // + // For i=0: jump_targets[1] is false, so fusion IS allowed. + // The jump target protection matters when the SECOND instruction of a + // potential pair is a jump target. Let's build that scenario: + // + // 0: Ret <- something before the loop + // 1: LoadVar { off: 5 } <- jump target (loop body start) + // 2: NextIterOrJump { jump_back: -1 } (target = 2 + (-1) = 1) + // 3: Ret + // + // For i=0 (Ret): can_fuse checks jump_targets[1] = true -> no fusion. + // This prevents fusing Ret with LoadVar, which is correct. + // + // A more realistic scenario: Op2 followed by AssignCurr where the + // AssignCurr is a jump target. + let mut bc = SymbolicByteCode { + code: vec![ + SymbolicOpcode::Op2 { op: Op2::Add }, // 0 + SymbolicOpcode::AssignCurr { var: v(0) }, // 1 -- jump target + SymbolicOpcode::NextIterOrJump { jump_back: -1 }, // 2 -> target = 2-1 = 1 + SymbolicOpcode::Ret, // 3 + ], + literals: vec![], + }; + bc.peephole_optimize(); + + // Fusion of 0+1 should be prevented because instruction 1 is a jump target + assert_eq!(bc.code.len(), 4); + assert!(matches!(bc.code[0], SymbolicOpcode::Op2 { op: Op2::Add })); + assert_eq!(bc.code[1], SymbolicOpcode::AssignCurr { var: v(0) }); + assert!(matches!(bc.code[2], SymbolicOpcode::NextIterOrJump { .. })); + assert!(matches!(bc.code[3], SymbolicOpcode::Ret)); +} + +#[test] +fn test_peephole_jump_target_only_blocks_specific_pair() { + // Verify that a jump target only blocks fusion of the pair where + // the second instruction is the target, not other pairs. + // + // 0: LoadConstant { id: 0 } + // 1: AssignCurr { off: 0 } <- NOT a jump target, so 0+1 CAN fuse + // 2: LoadVar { off: 5 } <- jump target + // 3: NextIterOrJump { jump_back: -1 } (target = 3-1 = 2) + // 4: Ret + let mut bc = SymbolicByteCode { + code: vec![ + SymbolicOpcode::LoadConstant { id: 0 }, + SymbolicOpcode::AssignCurr { var: v(0) }, + SymbolicOpcode::LoadVar { var: v(5) }, + SymbolicOpcode::NextIterOrJump { jump_back: -1 }, + SymbolicOpcode::Ret, + ], + literals: vec![1.0], + }; + bc.peephole_optimize(); + + // 0+1 should fuse (neither target), 2 stays (it's a jump target, but + // the previous instruction was AssignCurr which doesn't match any pattern + // anyway), 3 stays, 4 stays + assert_eq!(bc.code.len(), 4); + assert_eq!( + bc.code[0], + SymbolicOpcode::AssignConstCurr { + var: v(0), + literal_id: 0 + } + ); + assert_eq!(bc.code[1], SymbolicOpcode::LoadVar { var: v(5) }); + assert!(matches!(bc.code[2], SymbolicOpcode::NextIterOrJump { .. })); + assert!(matches!(bc.code[3], SymbolicOpcode::Ret)); +} + +#[test] +fn test_peephole_jump_offset_recalculation_next_iter() { + // When fusion shrinks the code, jump offsets must be recalculated. + // This test places a fusion BEFORE the loop (outside the jump target + // to jump instruction range) so the fixup works correctly. + // + // Before optimization: + // 0: LoadConstant { id: 0 } \ + // 1: AssignCurr { off: 0 } / -> fuse + // 2: LoadVar { off: 1 } <- jump target + // 3: AssignCurr { off: 2 } + // 4: NextIterOrJump { jump_back: -2 } target = 4+(-2) = 2 + // 5: Ret + // + // After optimization: + // 0: AssignConstCurr (fused 0+1) + // 1: LoadVar { off: 1 } (jump target) + // 2: AssignCurr { off: 2 } + // 3: NextIterOrJump { jump_back: -2 } (loop body unchanged) + // 4: Ret + let mut bc = SymbolicByteCode { + code: vec![ + SymbolicOpcode::LoadConstant { id: 0 }, // 0 + SymbolicOpcode::AssignCurr { var: v(0) }, // 1 + SymbolicOpcode::LoadVar { var: v(1) }, // 2 (jump target) + SymbolicOpcode::AssignCurr { var: v(2) }, // 3 + SymbolicOpcode::NextIterOrJump { jump_back: -2 }, // 4, target=2 + SymbolicOpcode::Ret, // 5 + ], + literals: vec![1.0], + }; + bc.peephole_optimize(); + + assert_eq!(bc.code.len(), 5); + assert!(matches!(bc.code[0], SymbolicOpcode::AssignConstCurr { .. })); + assert_eq!(bc.code[1], SymbolicOpcode::LoadVar { var: v(1) }); + assert_eq!(bc.code[2], SymbolicOpcode::AssignCurr { var: v(2) }); + match &bc.code[3] { + SymbolicOpcode::NextIterOrJump { jump_back } => { + assert_eq!(*jump_back, -2, "jump_back should remain -2"); + } + _ => panic!("expected NextIterOrJump"), + } + assert!(matches!(bc.code[4], SymbolicOpcode::Ret)); +} + +#[test] +fn test_peephole_fusion_inside_loop_body() { + let mut bc = SymbolicByteCode { + code: vec![ + SymbolicOpcode::LoadVar { var: v(0) }, // 0 (jump target) + SymbolicOpcode::Op2 { op: Op2::Add }, // 1 \ + SymbolicOpcode::AssignCurr { var: v(1) }, // 2 / fuse + SymbolicOpcode::NextIterOrJump { jump_back: -3 }, // 3, target=0 + SymbolicOpcode::Ret, // 4 + ], + literals: vec![], + }; + bc.peephole_optimize(); + + // 1+2 fuse -> BinOpAssignCurr + // Result: [LoadVar, BinOpAssignCurr, NextIterOrJump, Ret] + assert_eq!(bc.code.len(), 4); + assert_eq!(bc.code[0], SymbolicOpcode::LoadVar { var: v(0) }); + assert_eq!( + bc.code[1], + SymbolicOpcode::BinOpAssignCurr { + op: Op2::Add, + var: v(1) + } + ); + match &bc.code[2] { + SymbolicOpcode::NextIterOrJump { jump_back } => { + // new PC 2, target should be new PC 0 -> jump_back = -2 + assert_eq!(*jump_back, -2); + } + other => panic!( + "expected NextIterOrJump, got {:?}", + std::mem::discriminant(other) + ), + } + assert!(matches!(bc.code[3], SymbolicOpcode::Ret)); +} + +#[test] +fn test_peephole_jump_offset_recalculation_next_broadcast() { + // Same as above but with NextBroadcastOrJump + let mut bc = SymbolicByteCode { + code: vec![ + SymbolicOpcode::LoadConstant { id: 0 }, // 0 + SymbolicOpcode::AssignCurr { var: v(0) }, // 1 + SymbolicOpcode::LoadVar { var: v(1) }, // 2 (jump target) + SymbolicOpcode::NextBroadcastOrJump { jump_back: -1 }, // 3, target=2 + SymbolicOpcode::Ret, // 4 + ], + literals: vec![1.0], + }; + bc.peephole_optimize(); + + // 0+1 fuse -> AssignConstCurr at new PC 0 + // 2 -> new PC 1 (jump target) + // 3 -> new PC 2 + // 4 -> new PC 3 + assert_eq!(bc.code.len(), 4); + assert!(matches!(bc.code[0], SymbolicOpcode::AssignConstCurr { .. })); + assert_eq!(bc.code[1], SymbolicOpcode::LoadVar { var: v(1) }); + match &bc.code[2] { + SymbolicOpcode::NextBroadcastOrJump { jump_back } => { + // new PC 2, target should be new PC 1 + assert_eq!(*jump_back, -1, "jump_back should be -1"); + } + _ => panic!("expected NextBroadcastOrJump"), + } + assert!(matches!(bc.code[3], SymbolicOpcode::Ret)); +} + +#[test] +fn test_peephole_no_fusion_when_patterns_dont_match() { + // Op2 followed by something other than AssignCurr + let mut bc = SymbolicByteCode { + code: vec![ + SymbolicOpcode::Op2 { op: Op2::Add }, + SymbolicOpcode::Not {}, + SymbolicOpcode::Ret, + ], + literals: vec![], + }; + bc.peephole_optimize(); + + assert_eq!(bc.code.len(), 3); + assert!(matches!(bc.code[0], SymbolicOpcode::Op2 { op: Op2::Add })); + assert!(matches!(bc.code[1], SymbolicOpcode::Not {})); +} + +#[test] +fn test_peephole_load_constant_not_followed_by_assign_curr() { + // LoadConstant not followed by AssignCurr should not fuse + let mut bc = SymbolicByteCode { + code: vec![ + SymbolicOpcode::LoadConstant { id: 0 }, + SymbolicOpcode::Not {}, + SymbolicOpcode::Ret, + ], + literals: vec![1.0], + }; + bc.peephole_optimize(); + + assert_eq!(bc.code.len(), 3); + assert!(matches!(bc.code[0], SymbolicOpcode::LoadConstant { id: 0 })); +} + +#[test] +fn test_peephole_via_builder() { + // Verify that SymbolicByteCodeBuilder::finish() runs peephole_optimize + let mut builder = SymbolicByteCodeBuilder::default(); + let lit_id = builder.intern_literal(3.125); + builder.push_opcode(SymbolicOpcode::LoadConstant { id: lit_id }); + builder.push_opcode(SymbolicOpcode::AssignCurr { var: v(7) }); + builder.push_opcode(SymbolicOpcode::Ret); + + let bc = builder.finish(); + assert_eq!(bc.code.len(), 2); + match &bc.code[0] { + SymbolicOpcode::AssignConstCurr { var, literal_id } => { + assert_eq!(*var, v(7)); + assert_eq!(*literal_id, lit_id); + } + _ => panic!("expected AssignConstCurr after builder finish"), + } + assert!(matches!(bc.code[1], SymbolicOpcode::Ret)); +} + +#[test] +fn test_peephole_consecutive_fusions_chain() { + // Three consecutive fusible pairs + let mut bc = SymbolicByteCode { + code: vec![ + SymbolicOpcode::LoadConstant { id: 0 }, + SymbolicOpcode::AssignCurr { var: v(0) }, + SymbolicOpcode::LoadConstant { id: 1 }, + SymbolicOpcode::AssignCurr { var: v(1) }, + SymbolicOpcode::Op2 { op: Op2::Div }, + SymbolicOpcode::AssignCurr { var: v(2) }, + ], + literals: vec![1.0, 2.0], + }; + bc.peephole_optimize(); + + assert_eq!(bc.code.len(), 3); + assert_eq!( + bc.code[0], + SymbolicOpcode::AssignConstCurr { + var: v(0), + literal_id: 0 + } + ); + assert_eq!( + bc.code[1], + SymbolicOpcode::AssignConstCurr { + var: v(1), + literal_id: 1 + } + ); + match &bc.code[2] { + SymbolicOpcode::BinOpAssignCurr { op, var } => { + assert!(matches!(op, Op2::Div)); + assert_eq!(*var, v(2)); + } + _ => panic!("expected BinOpAssignCurr"), + } +} + +#[test] +fn test_peephole_last_instruction_not_fused_alone() { + // If the fusible first instruction is the very last one, no fusion happens + let mut bc = SymbolicByteCode { + code: vec![SymbolicOpcode::Ret, SymbolicOpcode::LoadConstant { id: 0 }], + literals: vec![1.0], + }; + bc.peephole_optimize(); + + assert_eq!(bc.code.len(), 2); + assert!(matches!(bc.code[0], SymbolicOpcode::Ret)); + assert!(matches!(bc.code[1], SymbolicOpcode::LoadConstant { id: 0 })); +} diff --git a/src/simlin-engine/src/compiler/symbolic_merge_proptest.rs b/src/simlin-engine/src/compiler/symbolic_merge_proptest.rs new file mode 100644 index 000000000..c015fb5eb --- /dev/null +++ b/src/simlin-engine/src/compiler/symbolic_merge_proptest.rs @@ -0,0 +1,1538 @@ +// Copyright 2026 The Simlin Authors. All rights reserved. +// Use of this source code is governed by the Apache License, +// Version 2.0, that can be found in the LICENSE file. + +//! Property tests for [`FragmentMerger`]'s obligations M1-M8, stated over +//! arbitrary well-formed fragment sets. +//! +//! The obligations themselves are written out on `FragmentMerger`'s rustdoc; +//! this file is where they are checked. Every property here is a statement +//! about the merged fragment ALONE -- what its opcodes name, how its resource +//! ranges are laid out, how many opcodes it has. None of them consults another +//! compiler, and none of them re-implements the rule under test as a test-local +//! oracle. The distinction matters: the temp-merging tests these replace +//! computed their expected value with a hand-written copy of the merge rule and +//! then asserted that copy equalled a hard-coded constant, so they could only +//! ever fail if the merger disagreed with a belief about a THIRD implementation +//! -- one that has since become `#[cfg(test)]` and no longer shares an address +//! representation with the code they constrained. +//! +//! The central device is [`Denotation`]: what an opcode's resource-id operands +//! NAME, resolved against the tables that opcode is addressed relative to, plus +//! the opcode with every resource id blanked out. Two opcodes have equal +//! denotations exactly when they are the same instruction over the same +//! resources, whatever ids they spell those resources with -- which is M1 and +//! M7 in one comparison. +//! +//! Case counts are deliberately modest (the whole file is well under a second +//! on a debug build): these are pure in-memory merges with no salsa, no +//! parsing, and no simulation, so a few hundred cases per property costs +//! almost nothing, but the fragment generator is small on purpose too. It is +//! not trying to be a fuzzer for codegen; it is trying to cover the SHAPES the +//! obligations quantify over -- several fragments, resources both referenced +//! and unreferenced, GF blocks that collide by content and blocks that do not, +//! nested GF references, temps, temp-based views, and a self-contained +//! iteration loop. + +use proptest::prelude::*; +use proptest::test_runner::TestCaseError; + +use super::*; + +// ============================================================================ +// Generated fragments +// ============================================================================ + +/// One graphical-function block of a generated fragment: `len` tables whose +/// content is a pure function of `content`, so two blocks generated with the +/// same `(content, len)` are bit-identical and MUST de-duplicate, while any +/// other pair must not. +#[derive(Clone, Debug)] +struct GfBlockSpec { + content: u8, + len: usize, + /// Whether any opcode reads this block. An unread block is the + /// "over-collected dependency table" case `gf_blocks_of_fragment` fills in + /// as a gap, and it still consumes ids. + referenced: bool, +} + +/// A generated fragment. +/// +/// `tag` makes every resource VALUE unique to this fragment, so an id that +/// lands on the wrong fragment's entry is always detectable rather than +/// accidentally equal. That applies to literals, module names, view dims AND +/// dim-list contents -- `build_fragment` mixes `tag` into each of them, so a +/// spec field here carries the SHAPE and the tag supplies the identity. +/// +/// Graphical-function content is the single exception: it is keyed on +/// `GfBlockSpec::content` precisely so collisions across fragments are +/// generated deliberately, which is what M4's de-duplication needs. +#[derive(Clone, Debug)] +struct FragSpec { + tag: usize, + n_literals: usize, + gf_blocks: Vec, + n_modules: usize, + n_views: usize, + /// Base the generated views on a temp rather than a variable (only honored + /// when the fragment declares at least one temp). + views_on_temps: bool, + /// Sizes of this fragment's temps, at dense 0-based ids. + temp_sizes: Vec, + dim_lists: Vec>, + /// Emit a self-contained `BeginIter .. NextIterOrJump .. EndIter` loop, so + /// the properties see a backward jump. + with_loop: bool, + /// How many per-element writes this fragment makes. Every fragment writes + /// at least one element, which is what makes it segmentable. + n_elements: usize, +} + +fn gf_block_spec() -> impl Strategy { + // Only three distinct contents, so collisions across fragments are common. + (0u8..3, 1usize..3, any::()).prop_map(|(content, len, referenced)| GfBlockSpec { + content, + len, + referenced, + }) +} + +fn frag_spec() -> impl Strategy { + ( + 0usize..3, // n_literals + prop::collection::vec(gf_block_spec(), 0..3), + 0usize..2, // n_modules + 0usize..3, // n_views + any::(), // views_on_temps + prop::collection::vec(1usize..4, 0..3), // temp_sizes + prop::collection::vec(prop::collection::vec(1u16..8, 1..5), 0..3), + any::(), // with_loop + 1usize..4, // n_elements + ) + .prop_map( + |( + n_literals, + gf_blocks, + n_modules, + n_views, + views_on_temps, + temp_sizes, + dim_lists, + with_loop, + n_elements, + )| FragSpec { + // Overwritten by `build_fragments` with the fragment's index. + tag: 0, + n_literals, + gf_blocks, + n_modules, + n_views, + views_on_temps, + temp_sizes, + dim_lists, + with_loop, + n_elements, + }, + ) +} + +/// One to four fragments, each tagged with its position so its resource values +/// are unique. +fn frag_specs() -> impl Strategy> { + prop::collection::vec(frag_spec(), 1..5).prop_map(|mut specs| { + for (i, spec) in specs.iter_mut().enumerate() { + spec.tag = i + 1; + } + specs + }) +} + +/// Every fragment carries every resource kind, plus a multi-table GF block, an +/// unreferenced GF block, temp-based views and a loop. The non-vacuity +/// companion to `frag_specs`. +fn forced_rich_specs() -> impl Strategy> { + let rich = (1usize..3, 0u8..3, 1usize..4, 1usize..4).prop_map( + |(n_literals, content, temp_a, temp_b)| FragSpec { + tag: 0, + n_literals, + gf_blocks: vec![ + GfBlockSpec { + content, + len: 2, + referenced: true, + }, + GfBlockSpec { + content: (content + 1) % 3, + len: 1, + referenced: false, + }, + ], + n_modules: 1, + n_views: 2, + views_on_temps: true, + temp_sizes: vec![temp_a, temp_b], + dim_lists: vec![vec![2, 3], vec![4]], + with_loop: true, + // Two elements, so every fragment has two segments and the loop + // sits inside one of them. + n_elements: 2, + }, + ); + // At least three, so `check_phase_split`'s thirds give every phase a + // fragment and the derived `ctx_base`s are non-zero. + prop::collection::vec(rich, 3..5).prop_map(|mut specs| { + for (i, spec) in specs.iter_mut().enumerate() { + spec.tag = i + 1; + } + specs + }) +} + +fn frag_var(tag: usize) -> Ident { + Ident::new(&format!("v{tag}")) +} + +/// The content of table `k` of a block generated with `content`. A pure +/// function of both, so equality of generated blocks is decidable from their +/// specs alone. +fn gf_table(content: u8, k: usize) -> Vec<(f64, f64)> { + vec![(k as f64, content as f64 * 100.0 + k as f64)] +} + +/// Materialize a `FragSpec` into the `PerVarBytecodes` shape +/// `compile_phase_to_per_var_bytecodes` produces: 0-based resource ids, one +/// contiguous run of opcodes per written element, and a single trailing `Ret`. +fn build_fragment(spec: &FragSpec) -> PerVarBytecodes { + let name = frag_var(spec.tag); + + let literals: Vec = (0..spec.n_literals) + .map(|i| (spec.tag * 1000 + i) as f64) + .collect(); + + let mut graphical_functions: Vec> = Vec::new(); + let mut block_starts: Vec = Vec::new(); + for block in &spec.gf_blocks { + block_starts.push(graphical_functions.len()); + for k in 0..block.len { + graphical_functions.push(gf_table(block.content, k)); + } + } + + let module_decls: Vec = (0..spec.n_modules) + .map(|i| SymbolicModuleDecl { + model_name: Ident::new(&format!("sub{}_{i}", spec.tag)), + input_set: BTreeSet::new(), + var: SymVarRef::new(name.clone(), i), + }) + .collect(); + + let on_temps = spec.views_on_temps && !spec.temp_sizes.is_empty(); + let static_views: Vec = (0..spec.n_views) + .map(|i| SymbolicStaticView { + base: if on_temps { + SymStaticViewBase::Temp((i % spec.temp_sizes.len()) as u32) + } else { + SymStaticViewBase::Var(SymVarRef::new(name.clone(), i)) + }, + dims: smallvec::smallvec![(spec.tag * 10 + i) as u16], + strides: smallvec::smallvec![1], + offset: i as u32, + sparse: SmallVec::new(), + dim_ids: SmallVec::new(), + }) + .collect(); + + let temp_sizes: Vec<(u32, usize)> = spec + .temp_sizes + .iter() + .enumerate() + .map(|(i, size)| (i as u32, *size)) + .collect(); + + // Tag-derived, like the literals and the view dims above: two fragments + // generated from the same shape must still carry DIFFERENT dim lists, or a + // mis-assigned dim-list id lands on an entry that happens to be equal and + // nothing notices. `forced_rich_specs` gives every fragment the same shape + // on purpose, so without this its dim lists were identical and the M8 guard + // could not see a dim-list base error at all. + let dim_lists: Vec> = spec + .dim_lists + .iter() + .map(|dl| dl.iter().map(|v| v + (spec.tag as u16) * 100).collect()) + .collect(); + + // Spread references to the fragment's resources across its elements, so a + // fragment with several elements exercises several segments and a fragment + // with one element carries all of them in a single segment. + let mut code: Vec = Vec::new(); + let mut next_lit = 0usize; + let mut next_block = 0usize; + let mut next_module = 0usize; + let mut next_view = 0usize; + let mut next_dl = 0usize; + let mut next_temp = 0usize; + for e in 0..spec.n_elements { + if next_lit < literals.len() { + code.push(SymbolicOpcode::LoadConstant { + id: next_lit as LiteralId, + }); + next_lit += 1; + } + if next_block < spec.gf_blocks.len() { + let block = &spec.gf_blocks[next_block]; + if block.referenced { + let start = block_starts[next_block]; + // The whole-block read, exactly the shape codegen emits for a + // subscripted arrayed GF (`base_gf` + the variable's table + // count). + code.push(SymbolicOpcode::Lookup { + base_gf: start as GraphicalFunctionId, + table_count: block.len as u16, + mode: LookupMode::Interpolate, + }); + if block.len >= 2 { + // A NESTED single-table read inside the same block: the + // `g[e](x)`-alongside-`g[D!](x)` shape whose runs must be + // relocated together. + code.push(SymbolicOpcode::Lookup { + base_gf: (start + 1) as GraphicalFunctionId, + table_count: 1, + mode: LookupMode::Interpolate, + }); + } + } + next_block += 1; + } + if next_module < module_decls.len() { + code.push(SymbolicOpcode::EvalModule { + id: next_module as ModuleId, + n_inputs: 0, + }); + next_module += 1; + } + if next_view < static_views.len() { + code.push(SymbolicOpcode::PushStaticView { + view_id: next_view as ViewId, + }); + code.push(SymbolicOpcode::PopView {}); + next_view += 1; + } + if next_dl < spec.dim_lists.len() { + code.push(SymbolicOpcode::PushVarViewDirect { + var: SymVarRef::base(name.clone()), + dim_list_id: next_dl as DimListId, + }); + code.push(SymbolicOpcode::PopView {}); + next_dl += 1; + } + if next_temp < temp_sizes.len() { + code.push(SymbolicOpcode::LoadTempConst { + temp_id: next_temp as TempId, + index: 0, + }); + code.push(SymbolicOpcode::VectorSortOrder { + write_temp_id: next_temp as TempId, + }); + next_temp += 1; + } + if spec.with_loop && !temp_sizes.is_empty() { + // Self-contained: `BeginIter` and its backward jump both live + // inside THIS element's segment, which is what lets the SCC + // interleaver reorder segments without invalidating the jump. + code.push(SymbolicOpcode::BeginIter { + write_temp_id: 0, + has_write_temp: true, + }); + code.push(SymbolicOpcode::LoadIterViewAt { offset: 0 }); + code.push(SymbolicOpcode::StoreIterElement {}); + code.push(SymbolicOpcode::NextIterOrJump { jump_back: -2 }); + code.push(SymbolicOpcode::EndIter {}); + } + code.push(SymbolicOpcode::AssignCurr { + var: SymVarRef::new(name.clone(), e), + }); + } + code.push(SymbolicOpcode::Ret); + + PerVarBytecodes { + symbolic: SymbolicByteCode { literals, code }, + graphical_functions, + module_decls, + static_views, + temp_sizes, + dim_lists, + } +} + +fn build_fragments(specs: &[FragSpec]) -> Vec { + specs.iter().map(build_fragment).collect() +} + +fn as_refs(frags: &[PerVarBytecodes]) -> Vec<&PerVarBytecodes> { + frags.iter().collect() +} + +/// A fragment's opcode count after its single trailing `Ret` is stripped -- +/// the quantity `db::assemble` sums to place the run-invariant flow prefix +/// boundary (M6). +fn ret_stripped_len(frag: &PerVarBytecodes) -> usize { + let code = &frag.symbolic.code; + if code.last() == Some(&SymbolicOpcode::Ret) { + code.len() - 1 + } else { + code.len() + } +} + +// ============================================================================ +// Denotation: what an opcode's ids NAME +// ============================================================================ + +/// The tables an opcode's resource ids are addressed relative to, plus the +/// base to subtract before indexing them. +/// +/// A merged table is PHASE-local while the ids the merger hands out are global +/// (they carry `ctx_base`), so dereferencing a merged opcode against a merged +/// table means subtracting the base back off. That asymmetry is real -- it is +/// how `assemble_module` can renumber a phase against preceding phases' counts +/// while every phase reports the same tables -- so the test models it rather +/// than hiding it. +struct ResourceTables<'a> { + literals: &'a [f64], + graphical_functions: &'a [Vec<(f64, f64)>], + module_decls: &'a [SymbolicModuleDecl], + static_views: &'a [SymbolicStaticView], + dim_lists: Vec>, + base: ContextResourceCounts, +} + +impl<'a> ResourceTables<'a> { + /// The tables a fragment's own (0-based) opcodes are addressed against. + fn of_fragment(frag: &'a PerVarBytecodes) -> Self { + ResourceTables { + literals: &frag.symbolic.literals, + graphical_functions: &frag.graphical_functions, + module_decls: &frag.module_decls, + static_views: &frag.static_views, + dim_lists: frag.dim_lists.iter().map(|dl| truncate4(dl)).collect(), + base: ContextResourceCounts::default(), + } + } + + /// The tables a merged stream's opcodes are addressed against, given the + /// `ctx_base` the merge was run with. + fn of_merged(merged: &'a ConcatenatedBytecodes, base: &ContextResourceCounts) -> Self { + ResourceTables { + literals: &merged.bytecode.literals, + graphical_functions: &merged.graphical_functions, + module_decls: &merged.module_decls, + static_views: &merged.static_views, + dim_lists: merged + .dim_lists + .iter() + .map(|(n, arr)| arr[..(*n as usize)].to_vec()) + .collect(), + base: base.clone(), + } + } +} + +/// The 4-element truncation `absorb_non_gf` applies to a dim list on the way +/// into the merged `(u8, [u16; 4])` representation. +fn truncate4(dl: &[u16]) -> Vec { + dl.iter().take(4).copied().collect() +} + +/// What one opcode denotes: the resource VALUES its ids name, plus the opcode +/// itself with every resource id blanked. +/// +/// Temps are deliberately absent. A temp id does not name a value, it names a +/// slot, and whether two fragments may share a slot is M5 -- checked by +/// `merged_temp_slot_uses_never_interleave`, not here. Blanking the temp ids +/// keeps this comparison from failing on the sharing M5 exists to permit. +#[derive(Clone, Debug, PartialEq)] +struct Denotation { + skeleton: SymbolicOpcode, + literals: Vec, + graphical_functions: Vec>, + module_decls: Vec, + static_views: Vec, + dim_lists: Vec>, +} + +/// Blank every resource id in `op`, leaving the instruction and all its +/// non-id operands (including `SymVarRef`s and jump offsets) untouched. +/// +/// The match is EXHAUSTIVE with no catch-all arm, on purpose: a new +/// `SymbolicOpcode` variant must be classified here before this file compiles, +/// where a catch-all would silently treat its ids as constants. +/// +/// That is a narrower guarantee than it looks, and the gap is worth naming. +/// Exhaustiveness forces a DECISION about a new variant; it does not force the +/// decision to be right, and it says nothing about the production side. +/// `renumber_opcode` still ends in `other => other.clone()`, so a variant +/// classified here as id-carrying but never given a renumber arm there would +/// come through unrenumbered -- and the properties would not see it either, +/// because the generators emit a fixed opcode set that a new variant is not in +/// until someone adds it. Adding a resource-bearing opcode means touching three +/// places, and only the first is compiler-enforced. +fn blank_resource_ids(op: &SymbolicOpcode) -> SymbolicOpcode { + match op { + // ── carry resource ids ────────────────────────────────────────── + SymbolicOpcode::LoadConstant { .. } => SymbolicOpcode::LoadConstant { id: 0 }, + SymbolicOpcode::AssignConstCurr { var, .. } => SymbolicOpcode::AssignConstCurr { + var: var.clone(), + literal_id: 0, + }, + SymbolicOpcode::Lookup { + table_count, mode, .. + } => SymbolicOpcode::Lookup { + base_gf: 0, + table_count: *table_count, + mode: *mode, + }, + SymbolicOpcode::LookupArray { + table_count, mode, .. + } => SymbolicOpcode::LookupArray { + base_gf: 0, + table_count: *table_count, + mode: *mode, + write_temp_id: 0, + }, + SymbolicOpcode::EvalModule { n_inputs, .. } => SymbolicOpcode::EvalModule { + id: 0, + n_inputs: *n_inputs, + }, + SymbolicOpcode::PushStaticView { .. } => SymbolicOpcode::PushStaticView { view_id: 0 }, + SymbolicOpcode::PushTempView { .. } => SymbolicOpcode::PushTempView { + temp_id: 0, + dim_list_id: 0, + }, + SymbolicOpcode::PushVarViewDirect { var, .. } => SymbolicOpcode::PushVarViewDirect { + var: var.clone(), + dim_list_id: 0, + }, + SymbolicOpcode::LoadTempConst { index, .. } => SymbolicOpcode::LoadTempConst { + temp_id: 0, + index: *index, + }, + SymbolicOpcode::LoadTempDynamic { .. } => SymbolicOpcode::LoadTempDynamic { temp_id: 0 }, + SymbolicOpcode::BeginIter { has_write_temp, .. } => SymbolicOpcode::BeginIter { + write_temp_id: 0, + has_write_temp: *has_write_temp, + }, + SymbolicOpcode::LoadIterTempElement { .. } => { + SymbolicOpcode::LoadIterTempElement { temp_id: 0 } + } + SymbolicOpcode::BeginBroadcastIter { n_sources, .. } => { + SymbolicOpcode::BeginBroadcastIter { + n_sources: *n_sources, + dest_temp_id: 0, + } + } + SymbolicOpcode::VectorElmMap { + full_source_len, .. + } => SymbolicOpcode::VectorElmMap { + write_temp_id: 0, + // NOT a resource id: an absolute element count of the source + // variable, invariant under renumbering, so it stays in the + // skeleton where a change to it fails the comparison. + full_source_len: *full_source_len, + }, + SymbolicOpcode::VectorSortOrder { .. } => { + SymbolicOpcode::VectorSortOrder { write_temp_id: 0 } + } + SymbolicOpcode::Rank { .. } => SymbolicOpcode::Rank { write_temp_id: 0 }, + SymbolicOpcode::AllocateAvailable { .. } => { + SymbolicOpcode::AllocateAvailable { write_temp_id: 0 } + } + SymbolicOpcode::AllocateByPriority { .. } => { + SymbolicOpcode::AllocateByPriority { write_temp_id: 0 } + } + + // ── carry no resource id ──────────────────────────────────────── + // + // `ViewStarRange::subdim_relation_id` indexes `subdim_relations`, which + // the fragment path never populates (`assemble_module` sets it to + // `vec![]`) and codegen never emits the opcode, so there is nothing to + // renumber and nothing this file can pin. Recorded here rather than + // silently grouped: if that opcode is ever revived, it becomes a sixth + // renumberable resource and belongs in the arm above. + SymbolicOpcode::Op2 { .. } + | SymbolicOpcode::Not { .. } + | SymbolicOpcode::LoadVar { .. } + | SymbolicOpcode::SymLoadPrev { .. } + | SymbolicOpcode::SymLoadInitial { .. } + | SymbolicOpcode::LoadGlobalVar { .. } + | SymbolicOpcode::PushSubscriptIndex { .. } + | SymbolicOpcode::LoadSubscript { .. } + | SymbolicOpcode::SetCond { .. } + | SymbolicOpcode::If { .. } + | SymbolicOpcode::Ret + | SymbolicOpcode::LoadModuleInput { .. } + | SymbolicOpcode::AssignCurr { .. } + | SymbolicOpcode::Apply { .. } + | SymbolicOpcode::BinOpAssignCurr { .. } + | SymbolicOpcode::BinOpAssignNext { .. } + | SymbolicOpcode::ViewSubscriptConst { .. } + | SymbolicOpcode::ViewSubscriptDynamic { .. } + | SymbolicOpcode::ViewRange { .. } + | SymbolicOpcode::ViewRangeDynamic { .. } + | SymbolicOpcode::ViewStarRange { .. } + | SymbolicOpcode::ViewWildcard { .. } + | SymbolicOpcode::ViewTranspose { .. } + | SymbolicOpcode::PopView { .. } + | SymbolicOpcode::DupView { .. } + | SymbolicOpcode::LoadIterElement { .. } + | SymbolicOpcode::LoadIterViewTop { .. } + | SymbolicOpcode::LoadIterViewAt { .. } + | SymbolicOpcode::StoreIterElement { .. } + | SymbolicOpcode::NextIterOrJump { .. } + | SymbolicOpcode::EndIter { .. } + | SymbolicOpcode::ArraySum { .. } + | SymbolicOpcode::ArrayMax { .. } + | SymbolicOpcode::ArrayMin { .. } + | SymbolicOpcode::ArrayMean { .. } + | SymbolicOpcode::ArrayStddev { .. } + | SymbolicOpcode::ArraySize { .. } + | SymbolicOpcode::VectorSelect { .. } + | SymbolicOpcode::LoadBroadcastElement { .. } + | SymbolicOpcode::StoreBroadcastElement { .. } + | SymbolicOpcode::NextBroadcastOrJump { .. } + | SymbolicOpcode::EndBroadcastIter { .. } => op.clone(), + } +} + +/// Bit patterns of one GF table, so tables compare by exact content. +fn table_bits(table: &[(f64, f64)]) -> Vec<(u64, u64)> { + table + .iter() + .map(|(x, y)| (x.to_bits(), y.to_bits())) + .collect() +} + +/// Resolve `op`'s resource ids against `tables`. `Err` names the id that did +/// not resolve, which is what an M1/M2/M3 break looks like from the outside: +/// an opcode pointing past the table it is supposed to index. +fn denote(op: &SymbolicOpcode, tables: &ResourceTables<'_>) -> Result { + let mut d = Denotation { + skeleton: blank_resource_ids(op), + literals: Vec::new(), + graphical_functions: Vec::new(), + module_decls: Vec::new(), + static_views: Vec::new(), + dim_lists: Vec::new(), + }; + let index = |id: u16, base: usize, len: usize, what: &str| -> Result { + let idx = (id as usize) + .checked_sub(base) + .ok_or_else(|| format!("{what} id {id} is below its ctx_base {base}"))?; + if idx >= len { + return Err(format!( + "{what} id {id} (index {idx}) is past its table of {len}" + )); + } + Ok(idx) + }; + + match op { + SymbolicOpcode::LoadConstant { id } => { + let i = index(*id, 0, tables.literals.len(), "literal")?; + d.literals.push(tables.literals[i].to_bits()); + } + SymbolicOpcode::AssignConstCurr { literal_id, .. } => { + let i = index(*literal_id, 0, tables.literals.len(), "literal")?; + d.literals.push(tables.literals[i].to_bits()); + } + SymbolicOpcode::Lookup { + base_gf, + table_count, + .. + } + | SymbolicOpcode::LookupArray { + base_gf, + table_count, + .. + } => { + for k in 0..(*table_count as usize) { + let slot = *base_gf as usize + k; + if slot >= tables.graphical_functions.len() { + return Err(format!( + "GF run [{base_gf}, {base_gf}+{table_count}) is past its table of {}", + tables.graphical_functions.len() + )); + } + d.graphical_functions + .push(table_bits(&tables.graphical_functions[slot])); + } + } + SymbolicOpcode::EvalModule { id, .. } => { + let i = index( + *id, + tables.base.modules, + tables.module_decls.len(), + "module", + )?; + d.module_decls.push(tables.module_decls[i].clone()); + } + SymbolicOpcode::PushStaticView { view_id } => { + let i = index( + *view_id, + tables.base.views, + tables.static_views.len(), + "view", + )?; + d.static_views.push(tables.static_views[i].clone()); + } + SymbolicOpcode::PushTempView { dim_list_id, .. } + | SymbolicOpcode::PushVarViewDirect { dim_list_id, .. } => { + let i = index( + *dim_list_id, + tables.base.dim_lists, + tables.dim_lists.len(), + "dim list", + )?; + d.dim_lists.push(tables.dim_lists[i].clone()); + } + _ => {} + } + Ok(d) +} + +/// A `SymbolicStaticView` with a `Temp` base normalized away. +/// +/// A view's `Temp` base is a temp reference, so it is renumbered like one and +/// its identity is M5's business; everything else about the view is M1's. +fn view_shape(view: &SymbolicStaticView) -> SymbolicStaticView { + SymbolicStaticView { + base: match &view.base { + SymStaticViewBase::Temp(_) => SymStaticViewBase::Temp(0), + other => other.clone(), + }, + ..view.clone() + } +} + +fn denote_shaped(op: &SymbolicOpcode, tables: &ResourceTables<'_>) -> Result { + let mut d = denote(op, tables)?; + d.static_views = d.static_views.iter().map(view_shape).collect(); + Ok(d) +} + +// ============================================================================ +// M1 + M6 + M7: every merged opcode names its own fragment's resources +// ============================================================================ + +proptest! { + #![proptest_config(ProptestConfig::with_cases(256))] + + /// M1, M6 and M7 together, which is the merge's whole reason to exist. + /// + /// M6 gives the slicing: the merged stream is each fragment's Ret-stripped + /// opcodes, contiguous and in order, plus one terminal `Ret`. Walking the + /// original opcodes beside their merged twins, M1 says each pair must + /// DENOTE the same resources -- the same literal value, the same module + /// declaration, the same run of GF tables, the same view, the same + /// dimension list -- even though the ids spelling them differ. M7 says the + /// rest of the opcode, `SymVarRef` operands and jump offsets included, is + /// byte-identical. + /// + /// The two halves are what a merge bug actually looks like: an id that + /// resolves to something (so nothing crashes) but to the WRONG something. + #[test] + fn merged_ids_dereference_to_their_own_fragments_resources(specs in frag_specs()) { + check_denotations_and_ret(&build_fragments(&specs))?; + } +} + +proptest! { + #![proptest_config(ProptestConfig::with_cases(64))] + + /// Non-vacuity guard for the property above (the GH #739 pattern). + /// + /// The base strategy draws resource counts from zero upwards, so an + /// individual case can carry no modules, no views, no temps and no GF + /// blocks at all -- and a property that only ever saw those cases would + /// pass no matter what the merger did with the resources it never got. + /// `forced_rich_specs` guarantees every fragment carries every resource + /// kind, and this property additionally ASSERTS that the merged output + /// contains all six, so coverage cannot silently drain away. + #[test] + fn forced_rich_fragments_exercise_every_resource_kind(specs in forced_rich_specs()) { + let frags = build_fragments(&specs); + let merged = check_denotations_and_ret(&frags)?; + + prop_assert!(!merged.bytecode.literals.is_empty(), "no literals generated"); + prop_assert!( + !merged.graphical_functions.is_empty(), + "no graphical functions generated" + ); + prop_assert!(!merged.module_decls.is_empty(), "no module decls generated"); + prop_assert!(!merged.static_views.is_empty(), "no static views generated"); + prop_assert!(!merged.temp_offsets.is_empty(), "no temps generated"); + prop_assert!(!merged.dim_lists.is_empty(), "no dim lists generated"); + prop_assert!( + merged + .bytecode + .code + .iter() + .any(|op| matches!(op, SymbolicOpcode::NextIterOrJump { .. })), + "no backward jump generated, so M7's jump-offset half is untested" + ); + // At least two fragments must share a temp slot, or M5's whole reason + // to exist is untested by this case. + prop_assert!( + frags.iter().filter(|f| !f.temp_sizes.is_empty()).count() >= 2, + "fewer than two fragments carry temps" + ); + } +} + +/// M1, M6 and M7 together, which is the merge's whole reason to exist. Shared +/// by the arbitrary and the forced-rich properties. +/// +/// M6 gives the slicing: the merged stream is each fragment's Ret-stripped +/// opcodes, contiguous and in order, plus one terminal `Ret`. Walking the +/// original opcodes beside their merged twins, M1 says each pair must DENOTE +/// the same resources -- the same literal value, the same module declaration, +/// the same run of GF tables, the same view, the same dimension list -- even +/// though the ids spelling them differ. M7 says the rest of the opcode, +/// `SymVarRef` operands and jump offsets included, is byte-identical. +/// +/// The two halves are what a merge bug actually looks like: an id that +/// resolves to something (so nothing crashes) but to the WRONG something. +fn check_denotations_and_ret( + frags: &[PerVarBytecodes], +) -> Result { + let refs = as_refs(frags); + let no_base = ContextResourceCounts::default(); + let merged = concatenate_fragments(&refs, &no_base) + .map_err(|e| TestCaseError::fail(format!("well-formed fragments must merge: {e}")))?; + + { + let merged_tables = ResourceTables::of_merged(&merged, &no_base); + let mut pos = 0usize; + for frag in frags { + let n = ret_stripped_len(frag); + let frag_tables = ResourceTables::of_fragment(frag); + for k in 0..n { + let orig = &frag.symbolic.code[k]; + let got = &merged.bytecode.code[pos + k]; + let want = denote_shaped(orig, &frag_tables) + .map_err(|e| TestCaseError::fail(format!("fragment opcode {k}: {e}")))?; + let have = denote_shaped(got, &merged_tables) + .map_err(|e| TestCaseError::fail(format!("merged opcode {}: {e}", pos + k)))?; + prop_assert_eq!( + &want, + &have, + "merged opcode {} does not denote what fragment opcode {} denoted", + pos + k, + k + ); + } + pos += n; + } + + // M6: exactly the fragments' opcodes plus one terminal `Ret`. + prop_assert_eq!( + merged.bytecode.code.len(), + pos + 1, + "merged length must be the sum of the Ret-stripped fragment lengths plus one Ret" + ); + prop_assert_eq!( + merged.bytecode.code.last(), + Some(&SymbolicOpcode::Ret), + "the merged stream must end in a single Ret" + ); + let rets = merged + .bytecode + .code + .iter() + .filter(|op| matches!(op, SymbolicOpcode::Ret)) + .count(); + prop_assert_eq!(rets, 1, "no interior Ret may survive the merge"); + } + + Ok(merged) +} + +// ============================================================================ +// M2: flat resources are appended, so their ranges tile the merged table +// ============================================================================ + +proptest! { + #![proptest_config(ProptestConfig::with_cases(256))] + + /// M2 stated positively: each flat merged table is EXACTLY the + /// concatenation of the fragments' tables, in fragment order, with the two + /// documented normalizations (a dim list truncated to four elements, a + /// `Temp`-based view's base shifted onto the merged temp slot). That is + /// disjointness and tiling in one statement -- a fragment whose entries + /// overlapped another's, or that was dropped, or that was reordered, fails + /// it. + /// + /// Graphical functions are deliberately absent: they are the one resource + /// that shares rather than tiles (M4), and they have their own property. + #[test] + fn flat_resource_ranges_are_disjoint_and_tile_the_merged_table(specs in frag_specs()) { + let frags = build_fragments(&specs); + let refs = as_refs(&frags); + // A non-zero recycle base, so the view/temp shift is exercised rather + // than being the identity. + let base = ContextResourceCounts { + temps: 2, + ..ContextResourceCounts::default() + }; + let merged = concatenate_fragments(&refs, &base) + .map_err(|e| TestCaseError::fail(format!("well-formed fragments must merge: {e}")))?; + + let want_literals: Vec = frags + .iter() + .flat_map(|f| f.symbolic.literals.iter().copied()) + .collect(); + prop_assert_eq!(&merged.bytecode.literals, &want_literals); + + let want_modules: Vec = frags + .iter() + .flat_map(|f| f.module_decls.iter().cloned()) + .collect(); + prop_assert_eq!(&merged.module_decls, &want_modules); + + let want_views: Vec = frags + .iter() + .flat_map(|f| { + f.static_views.iter().map(|v| SymbolicStaticView { + base: match &v.base { + SymStaticViewBase::Temp(id) => SymStaticViewBase::Temp(id + base.temps), + other => other.clone(), + }, + ..v.clone() + }) + }) + .collect(); + prop_assert_eq!(&merged.static_views, &want_views); + + let want_dim_lists: Vec> = frags + .iter() + .flat_map(|f| f.dim_lists.iter().map(|dl| truncate4(dl))) + .collect(); + let got_dim_lists: Vec> = merged + .dim_lists + .iter() + .map(|(n, arr)| arr[..(*n as usize)].to_vec()) + .collect(); + prop_assert_eq!(&got_dim_lists, &want_dim_lists); + } +} + +// ============================================================================ +// M4: graphical functions share by content and only by content +// ============================================================================ + +proptest! { + #![proptest_config(ProptestConfig::with_cases(256))] + + /// M4 in three parts. + /// + /// **Total content preservation.** The remap covers `[0, gf_len)`, and for + /// EVERY local slot -- including one no opcode reads -- the merged table at + /// the remapped id holds that slot's content. This is the safety direction: + /// it means two slots can share an id only when their content is + /// bit-identical, so a `Lookup` can never be redirected onto a different + /// table. + /// + /// **Run contiguity.** For every `Lookup`/`LookupArray` run + /// `[base, base + table_count)`, the remap shifts the whole run by one + /// delta. `LookupArray` reads `graphical_functions[base_gf ..]` at runtime, + /// so a run scattered by de-duplication would read the wrong tables even + /// though each individual table is present somewhere. + /// + /// **De-duplication actually happens.** Two fragments built from identical + /// specs must get identical remaps -- otherwise "de-duplicate" could be + /// satisfied by never sharing anything, and the property above would still + /// pass. + #[test] + fn gf_dedup_preserves_runs_and_never_merges_distinct_content(specs in frag_specs()) { + // Duplicate the first fragment at the end, so at least one pair of + // fragments always carries bit-identical GF blocks. + let mut specs = specs; + let mut dup = specs[0].clone(); + dup.tag += 100; + let dup_index = specs.len(); + specs.push(dup); + + let frags = build_fragments(&specs); + let refs = as_refs(&frags); + let dedup = GfDedup::build(&refs) + .map_err(|e| TestCaseError::fail(format!("well-formed fragments must dedup: {e}")))?; + + for (i, frag) in frags.iter().enumerate() { + let remap = dedup.remap(i); + prop_assert_eq!( + remap.len(), + frag.graphical_functions.len(), + "fragment {}'s remap must cover every local GF slot, read or not", + i + ); + for (local, table) in frag.graphical_functions.iter().enumerate() { + let global = remap[local] as usize; + prop_assert!( + global < dedup.tables.len(), + "fragment {} slot {} remapped past the deduped table", + i, + local + ); + prop_assert_eq!( + table_bits(&dedup.tables[global]), + table_bits(table), + "fragment {} slot {} lost its content under de-duplication", + i, + local + ); + } + for op in &frag.symbolic.code { + let (base, count) = match op { + SymbolicOpcode::Lookup { + base_gf, + table_count, + .. + } + | SymbolicOpcode::LookupArray { + base_gf, + table_count, + .. + } => (*base_gf as usize, *table_count as usize), + _ => continue, + }; + for k in 0..count { + prop_assert_eq!( + remap[base + k] as usize, + remap[base] as usize + k, + "fragment {}'s GF run [{}, {}+{}) was scattered by de-duplication", + i, + base, + base, + count + ); + } + } + } + + prop_assert_eq!( + dedup.remap(0), + dedup.remap(dup_index), + "two fragments carrying bit-identical GF blocks must share them" + ); + } +} + +// ============================================================================ +// M5: temps +// ============================================================================ + +/// Every merged temp slot `code` references, paired with the position of the +/// opcode that references it. +/// +/// There are TWO channels by which an opcode reaches a temp, and a scan that +/// covers only the first is blind to half of M5: +/// +/// 1. an opcode carrying a temp id directly (`VectorSortOrder`, `BeginIter` +/// with a write temp, `LoadTempConst`, ...); +/// 2. a `PushStaticView { view_id }` whose `static_views[view_id].base` is a +/// `Temp`. That base is renumbered by `absorb_non_gf` alongside the opcodes, +/// with the SAME per-fragment `temp_offset`, and it is how an +/// array-producing builtin's scratch array is read back as a view. +/// +/// Channel 2 is why this takes the view table. A merger that shifted a view's +/// `Temp` base by the wrong offset would leave every opcode-carried id correct +/// and still hand one fragment another's scratch array -- and until that +/// channel was scanned, nothing in the repository noticed. +fn temp_uses(code: &[SymbolicOpcode], views: &[SymbolicStaticView]) -> Vec<(usize, u32)> { + code.iter() + .enumerate() + .filter_map(|(pc, op)| { + let id: u32 = match op { + SymbolicOpcode::PushTempView { temp_id, .. } + | SymbolicOpcode::LoadTempConst { temp_id, .. } + | SymbolicOpcode::LoadTempDynamic { temp_id } + | SymbolicOpcode::LoadIterTempElement { temp_id } => *temp_id as u32, + SymbolicOpcode::BeginIter { + write_temp_id, + has_write_temp: true, + } => *write_temp_id as u32, + SymbolicOpcode::BeginBroadcastIter { dest_temp_id, .. } => *dest_temp_id as u32, + SymbolicOpcode::VectorElmMap { write_temp_id, .. } + | SymbolicOpcode::VectorSortOrder { write_temp_id } + | SymbolicOpcode::Rank { write_temp_id } + | SymbolicOpcode::LookupArray { write_temp_id, .. } + | SymbolicOpcode::AllocateAvailable { write_temp_id } + | SymbolicOpcode::AllocateByPriority { write_temp_id } => *write_temp_id as u32, + // Channel 2: the view this opcode pushes may itself be based on + // a temp. + SymbolicOpcode::PushStaticView { view_id } => { + // Loud on an id the view table cannot answer. A silent + // `None` here is the exact shape this scan exists to + // eliminate -- a temp channel that goes unread -- so an + // out-of-range id must not look like "this opcode uses no + // temp". (M1's `denote` reports it too; this is the + // belt-and-braces inside the scan itself.) + let view = views.get(*view_id as usize).unwrap_or_else(|| { + panic!( + "PushStaticView names view {view_id} but the table holds {}", + views.len() + ) + }); + match &view.base { + SymStaticViewBase::Temp(id) => *id, + SymStaticViewBase::Var(_) => return None, + } + } + _ => return None, + }; + Some((pc, id)) + }) + .collect() +} + +/// The per-slot sizes behind a `ConcatenatedBytecodes`'s `temp_offsets` / +/// `temp_total_size` pair. +fn merged_temp_sizes(merged: &ConcatenatedBytecodes) -> Vec { + let offs = &merged.temp_offsets; + (0..offs.len()) + .map(|i| { + let end = offs.get(i + 1).copied().unwrap_or(merged.temp_total_size); + end - offs[i] + }) + .collect() +} + +proptest! { + #![proptest_config(ProptestConfig::with_cases(256))] + + /// M5 for the sequential emitter, stated as the live-range condition + /// itself rather than as a layout. + /// + /// Tag every temp use in the merged stream with the fragment that produced + /// it. For each merged temp SLOT, the tagged uses must form runs: once the + /// stream leaves fragment A's uses of a slot and enters fragment B's, it + /// must never come back to A. That is exactly "no two fragments' uses of a + /// slot interleave", which is what makes identity sharing safe. The same + /// property is what forbids `Recycle` under an interleaving emitter, where + /// it fails by construction -- see + /// `db::combined_fragment_proptest::interleaved_members_never_share_a_temp_slot` + /// for the other half. + /// + /// Also pinned here, because they are the rest of what "share a slot" + /// means: storage regions derived from `temp_offsets` are disjoint and + /// cover `temp_total_size`, and every referenced slot exists. + #[test] + fn merged_temp_slot_uses_never_interleave(specs in frag_specs()) { + let frags = build_fragments(&specs); + let refs = as_refs(&frags); + let no_base = ContextResourceCounts::default(); + let merged = concatenate_fragments(&refs, &no_base) + .map_err(|e| TestCaseError::fail(format!("well-formed fragments must merge: {e}")))?; + + // Which merged positions belong to which fragment (M6's slicing). + let mut owner_of_pc: Vec = Vec::new(); + for (i, frag) in frags.iter().enumerate() { + owner_of_pc.extend((0..ret_stripped_len(frag)).map(|_| i)); + } + + let mut seen_owners: HashMap> = HashMap::new(); + for (pc, temp) in temp_uses(&merged.bytecode.code, &merged.static_views) { + if pc >= owner_of_pc.len() { + continue; // the terminal Ret carries no temp + } + let owner = owner_of_pc[pc]; + let runs = seen_owners.entry(temp).or_default(); + if runs.last() != Some(&owner) { + prop_assert!( + !runs.contains(&owner), + "merged temp slot {} is used by fragment {} again after another \ + fragment used it -- two fragments' live ranges interleave on a \ + shared slot", + temp, + owner + ); + runs.push(owner); + } + } + + // Storage: `temp_offsets` is a prefix sum, so slots never overlap. + let sizes = merged_temp_sizes(&merged); + let mut running = 0usize; + for (i, size) in sizes.iter().enumerate() { + prop_assert_eq!( + merged.temp_offsets[i], + running, + "temp slot {} does not start where the previous slot ended", + i + ); + running += size; + } + prop_assert_eq!(running, merged.temp_total_size); + + // Every referenced slot must exist, or the VM indexes `temp_offsets` + // out of bounds. + for (_, temp) in temp_uses(&merged.bytecode.code, &merged.static_views) { + prop_assert!( + (temp as usize) < merged.temp_offsets.len(), + "merged opcode references temp slot {} but the pool has {}", + temp, + merged.temp_offsets.len() + ); + } + } +} + +proptest! { + #![proptest_config(ProptestConfig::with_cases(256))] + + /// The `Recycle` half of M5: sharing is by IDENTITY (fragment-local temp + /// `t` becomes merged slot `t + ctx_base.temps`, whichever fragment it came + /// from), and a shared slot is sized to the LARGEST of its users. + /// + /// Both halves are load-bearing and fail differently. Identity is what + /// bounds the merged count by the widest single fragment instead of the sum + /// -- the `u8` `TempId` capacity #583 blew through. Max sizing is what + /// keeps the fragment that needs the most elements from writing past its + /// storage into the next slot. + #[test] + fn recycle_shares_by_identity_and_sizes_by_max(specs in frag_specs()) { + let frags = build_fragments(&specs); + let refs = as_refs(&frags); + let base = ContextResourceCounts { + temps: 1, + ..ContextResourceCounts::default() + }; + let merged = concatenate_fragments(&refs, &base) + .map_err(|e| TestCaseError::fail(format!("well-formed fragments must merge: {e}")))?; + + // Identity: read each fragment's temp uses back out of its slice of the + // merged stream and check the shift is the fixed base, not a per- + // fragment running sum. + let mut pos = 0usize; + for frag in &frags { + let n = ret_stripped_len(frag); + let local = temp_uses(&frag.symbolic.code[..n], &frag.static_views); + let global = temp_uses(&merged.bytecode.code[pos..pos + n], &merged.static_views); + prop_assert_eq!(local.len(), global.len()); + for ((_, l), (_, g)) in local.iter().zip(global.iter()) { + prop_assert_eq!( + *g, + *l + base.temps, + "recycle must shift every fragment's temps by the same fixed base" + ); + } + pos += n; + } + + // Max sizing: the slot for local id `t` must be at least as large as + // every fragment that declared `t`, and no larger than the largest of + // them (no silent inflation). + let sizes = merged_temp_sizes(&merged); + let mut want: HashMap = HashMap::new(); + for frag in &frags { + for (id, size) in &frag.temp_sizes { + let slot = *id + base.temps; + let e = want.entry(slot).or_insert(0); + *e = (*e).max(*size); + } + } + for (slot, size) in &want { + prop_assert!( + (*slot as usize) < sizes.len(), + "slot {} was never allocated", + slot + ); + prop_assert_eq!( + sizes[*slot as usize], + *size, + "shared temp slot {} must hold exactly the largest of its users", + slot + ); + } + } +} + +// ============================================================================ +// M6: a prefix of the fragment list is a prefix of the merged stream +// ============================================================================ + +proptest! { + #![proptest_config(ProptestConfig::with_cases(128))] + + /// M6's consequence, and the reason `db::assemble` may compute the + /// run-invariant flow prefix (GH #712) as a SUM of fragment lengths before + /// the merge has run: merging only the first `k` fragments produces exactly + /// the first `sum(ret_stripped[..k])` opcodes of merging all of them. + /// + /// Nothing about a fragment's renumbering may depend on what FOLLOWS it, + /// which is a stronger and more useful statement than "the counts add up". + /// A merger that, say, assigned GF ids by global frequency would still + /// preserve opcode counts but would move the boundary's contents. + #[test] + fn merge_is_one_to_one_on_opcodes_and_prefix_lengths_are_boundaries( + specs in frag_specs(), + ) { + let frags = build_fragments(&specs); + let refs = as_refs(&frags); + let no_base = ContextResourceCounts::default(); + let full = concatenate_fragments(&refs, &no_base) + .map_err(|e| TestCaseError::fail(format!("well-formed fragments must merge: {e}")))?; + + let mut boundary = 0usize; + for k in 1..=frags.len() { + boundary += ret_stripped_len(&frags[k - 1]); + let prefix = concatenate_fragments(&refs[..k], &no_base) + .map_err(|e| TestCaseError::fail(format!("prefix merge failed: {e}")))?; + prop_assert_eq!( + prefix.bytecode.code.len(), + boundary + 1, + "a {}-fragment merge must be {} opcodes plus a Ret", + k, + boundary + ); + prop_assert_eq!( + &prefix.bytecode.code[..boundary], + &full.bytecode.code[..boundary], + "merging {} fragments alone must reproduce the first {} opcodes of \ + merging all {}", + k, + boundary, + frags.len() + ); + } + } +} + +// ============================================================================ +// M8: the phase split agrees with the whole-model merge +// ============================================================================ + +proptest! { + #![proptest_config(ProptestConfig::with_cases(128))] + + /// M8, stated the way the VM experiences it. + /// + /// `assemble_module` renumbers initials, flows and stocks SEPARATELY (each + /// against a `ctx_base` summed from the preceding phases) but builds the + /// module's `module_decls` / `static_views` / `dim_lists` / `temp_offsets` + /// from ONE all-phases merge. So the test derefs a phase's renumbered + /// opcodes against the ALL-PHASES tables and requires them to name what the + /// fragment named -- which is literally what happens at run time. + /// + /// Literals are the deliberate exception, and the property is written to + /// show why rather than to skip them: each phase's bytecode carries its own + /// literal pool and the all-phases pool is discarded, so a phase's literal + /// ids are checked against the PHASE's pool. + #[test] + fn phase_split_assigns_the_same_ids_as_the_all_phases_merge(specs in frag_specs()) { + check_phase_split(&build_fragments(&specs), false)?; + } +} + +proptest! { + #![proptest_config(ProptestConfig::with_cases(64))] + + /// Non-vacuity guard for the property above, and it matters more here than + /// anywhere else in this file. + /// + /// A phase whose fragments carry no modules, no views and no dim lists + /// hands the NEXT phase a `ctx_base` of all zeros, and with a zero base the + /// phase split is trivially the all-phases merge -- the property passes + /// without testing anything. The base strategy produces that shape often. + /// `forced_rich_specs` guarantees each fragment carries all three + /// resources, and `check_phase_split(.., require_nonzero_base = true)` + /// asserts the derived bases really are non-zero before checking anything. + /// + /// Worth the extra property specifically because this is the one obligation + /// whose violation nothing else in the default (`--lib`) test suite + /// notices: dropping `ctx_base` from `absorb_non_gf` leaves all 5363 other + /// tests green. + #[test] + fn forced_rich_phase_split_uses_non_zero_bases(specs in forced_rich_specs()) { + check_phase_split(&build_fragments(&specs), true)?; + } +} + +/// M8, stated the way the VM experiences it. Shared by the arbitrary and the +/// forced-rich properties. +/// +/// `assemble_module` renumbers initials, flows and stocks SEPARATELY (each +/// against a `ctx_base` summed from the preceding phases) but builds the +/// module's `module_decls` / `static_views` / `dim_lists` / `temp_offsets` from +/// ONE all-phases merge. So this derefs a phase's renumbered opcodes against +/// the ALL-PHASES tables and requires them to name what the fragment named -- +/// which is literally what happens at run time. +/// +/// Literals are the deliberate exception, and the check is written to show why +/// rather than to skip them: each phase's bytecode carries its own literal pool +/// and the all-phases pool is discarded, so a phase's literal ids are resolved +/// against the PHASE's pool. +fn check_phase_split( + frags: &[PerVarBytecodes], + require_nonzero_base: bool, +) -> Result<(), TestCaseError> { + let refs = as_refs(frags); + // Split into the three phases assembly uses. An empty phase is a real + // shape (a model with no stocks), so the split is allowed to produce + // one. + let n = refs.len(); + let a = n / 3; + let b = a + (n - a) / 2; + let phases: [&[&PerVarBytecodes]; 3] = [&refs[..a], &refs[a..b], &refs[b..]]; + + let counts0 = ContextResourceCounts::from_fragments(phases[0]); + let counts1 = ContextResourceCounts::from_fragments(phases[1]); + // Exactly `assemble_module`'s bases: modules / views / dim-lists sum + // across preceding phases, temps stay at 0 because the temp pool is + // shared by every phase rather than partitioned. + let bases = [ + ContextResourceCounts::default(), + ContextResourceCounts { + temps: 0, + ..counts0.clone() + }, + ContextResourceCounts { + modules: counts0.modules + counts1.modules, + views: counts0.views + counts1.views, + temps: 0, + dim_lists: counts0.dim_lists + counts1.dim_lists, + }, + ]; + + if require_nonzero_base { + prop_assert!( + bases[1].modules > 0 && bases[1].views > 0 && bases[1].dim_lists > 0, + "the second phase's base must be non-zero or the split is \ + trivially the all-phases merge: {:?}", + bases[1] + ); + prop_assert!( + bases[2].modules > bases[1].modules, + "the third phase's base must exceed the second's: {:?} vs {:?}", + bases[2], + bases[1] + ); + } + + let dedup = GfDedup::build(&refs).map_err(|e| TestCaseError::fail(format!("dedup: {e}")))?; + let all = concatenate_fragments_with_gf(&refs, &ContextResourceCounts::default(), &dedup, 0) + .map_err(|e| TestCaseError::fail(format!("all-phases merge: {e}")))?; + + // The INITIALS phase does not go through a concat: `eval_initials` runs + // each initial separately, so assembly renumbers them one at a time through + // `renumber_initials_phase`, each at literal offset 0. Drive the REAL + // function and deref its output against the same all-phases tables, so this + // property covers the path assembly actually takes rather than a concat + // standing in for it. As an inline loop that function was unreachable from + // a test, and freezing two of its three accumulators left the whole + // repository green. + // + // Run it over EVERY fragment rather than over `phases[0]`: the thirds + // split leaves the initials phase with at most one fragment, and with one + // initial the running offsets are never used at all -- freezing two of the + // three would still pass. A model in which every variable has an initial is + // both a real shape and the one that exercises the accumulators, and it + // keeps the all-phases tables the correct thing to deref against (the + // fragment list and its order are the same). + let named_initials: Vec<(String, &PerVarBytecodes)> = refs + .iter() + .enumerate() + .map(|(i, f)| (format!("init{i}"), *f)) + .collect(); + if require_nonzero_base { + prop_assert!( + named_initials.len() >= 2, + "one initial never advances the per-initial offsets, so the check \ + below would be vacuous" + ); + } + let initials = crate::db::renumber_initials_phase(&named_initials, &dedup) + .map_err(|e| TestCaseError::fail(format!("initials renumber: {e}")))?; + for (frag, compiled) in refs.iter().zip(initials.iter()) { + let frag_tables = ResourceTables::of_fragment(frag); + let tables = ResourceTables { + literals: &compiled.bytecode.literals, + graphical_functions: &all.graphical_functions, + module_decls: &all.module_decls, + static_views: &all.static_views, + dim_lists: all + .dim_lists + .iter() + .map(|(n, arr)| arr[..(*n as usize)].to_vec()) + .collect(), + base: ContextResourceCounts::default(), + }; + let n = ret_stripped_len(frag); + for k in 0..n { + let want = denote_shaped(&frag.symbolic.code[k], &frag_tables) + .map_err(|e| TestCaseError::fail(format!("initial fragment: {e}")))?; + let have = denote_shaped(&compiled.bytecode.code[k], &tables).map_err(|e| { + TestCaseError::fail(format!( + "initial opcode {k} against the all-phases tables: {e}" + )) + })?; + prop_assert_eq!( + &want, + &have, + "initial opcode {} does not name its fragment's resource in the \ + module's all-phases tables", + k + ); + } + } + + let mut gf_index_base = 0usize; + let mut frag_index = 0usize; + for (p, phase) in phases.iter().enumerate() { + let concat = concatenate_fragments_with_gf(phase, &bases[p], &dedup, gf_index_base) + .map_err(|e| TestCaseError::fail(format!("phase {p} merge: {e}")))?; + // The tables the VM would consult: the phase's OWN literal pool, + // every other table from the all-phases merge. + let tables = ResourceTables { + literals: &concat.bytecode.literals, + graphical_functions: &all.graphical_functions, + module_decls: &all.module_decls, + static_views: &all.static_views, + dim_lists: all + .dim_lists + .iter() + .map(|(n, arr)| arr[..(*n as usize)].to_vec()) + .collect(), + base: ContextResourceCounts::default(), + }; + + let mut pos = 0usize; + for frag in phase.iter() { + let n = ret_stripped_len(frag); + let frag_tables = ResourceTables::of_fragment(frag); + for k in 0..n { + let want = denote_shaped(&frag.symbolic.code[k], &frag_tables) + .map_err(|e| TestCaseError::fail(format!("fragment: {e}")))?; + let have = denote_shaped(&concat.bytecode.code[pos + k], &tables).map_err(|e| { + TestCaseError::fail(format!( + "phase {p} opcode {} against the all-phases tables: {e}", + pos + k + )) + })?; + prop_assert_eq!( + &want, + &have, + "phase {} opcode {} does not name its fragment's resource in the \ + module's all-phases tables", + p, + pos + k + ); + } + pos += n; + frag_index += 1; + } + gf_index_base = frag_index; + } + + Ok(()) +} diff --git a/src/simlin-engine/src/db.rs b/src/simlin-engine/src/db.rs index cb70bdfdb..2f7b85214 100644 --- a/src/simlin-engine/src/db.rs +++ b/src/simlin-engine/src/db.rs @@ -77,7 +77,6 @@ pub use input::{ }; mod query; -pub(crate) use query::canonical_module_input_set; pub use query::{ ImplicitVarMeta, ModuleReferenceGraph, ParsedVariableResult, UnitsContextResult, VariableDeps, model_implicit_var_info, model_module_ident_context, model_module_map, @@ -86,6 +85,9 @@ pub use query::{ project_units_context, project_units_context_result, variable_dimensions, variable_direct_dependencies, variable_relevant_dimensions, variable_size, }; +pub(crate) use query::{ + canonical_module_input_set, model_implicit_var_by_name, model_variable_by_name, +}; mod sync; pub use sync::{ @@ -102,12 +104,21 @@ pub use fragment_compile::compile_var_fragment; pub(crate) use fragment_compile::{ compile_implicit_var_fragment, compile_implicit_var_phase_bytecodes, }; +// Test-only: the per-thread record of which fragment-compiler bodies ran, so +// `fragment_char_tests` can prove a layout-only edit did or did not recompile +// a fragment. Pointer equality of a memo cannot prove that -- salsa backdates +// a re-executed query whose value compares equal -- and a fragment is designed +// to compare equal across a layout edit, so the record is the only evidence. +#[cfg(test)] +pub(crate) use fragment_compile::{ + FragmentExecKind, fragment_executions, note_fragment_execution, reset_fragment_executions, +}; mod assemble; pub(crate) use assemble::{ - PerVarOffsetMap, VarFragmentResult, build_module_inputs, build_stub_variable, + PerVarSizes, VarFragmentResult, build_module_inputs, build_stub_variable, build_submodel_metadata, compile_phase_to_per_var_bytecodes, extract_tables_from_source_var, - var_phase_symbolic_fragment_prod, + fragment_emit_ctx, var_phase_symbolic_fragment_prod, }; pub use assemble::{assemble_module, assemble_simulation}; // `combine_scc_fragment` and `calc_flattened_offsets_incremental` are @@ -117,8 +128,19 @@ pub use assemble::{assemble_module, assemble_simulation}; // `crate::db::...` / `super::...`. #[cfg(test)] pub(crate) use assemble::{calc_flattened_offsets_incremental, combine_scc_fragment}; +// `renumber_initials_phase` is the initials half of assembly's phase renumber. +// The root re-export exists so the merger's M8 property +// (`compiler::symbolic_merge_proptest`) can drive the REAL function rather than +// re-deriving it: as an inline loop inside `assemble_module` it was unreachable +// from a test, and freezing two of its three accumulators left the whole +// repository green. +#[cfg(test)] +pub(crate) use assemble::renumber_initials_phase; -pub use dep_graph::{ModelDepGraphResult, ResolvedScc, SccPhase, model_dependency_graph}; +pub use dep_graph::{ + ModelDepGraphResult, ResolvedScc, RunlistMembership, SccPhase, model_dependency_graph, + var_runlist_membership, +}; mod ltm; use ltm::*; @@ -1363,6 +1385,8 @@ pub fn compile_project_incremental( } } +#[cfg(test)] +mod combined_fragment_proptest; #[cfg(test)] mod combined_fragment_tests; #[cfg(test)] @@ -1376,6 +1400,10 @@ mod dimension_invalidation_tests; #[cfg(test)] mod fragment_cache_tests; #[cfg(test)] +mod fragment_char_tests; +#[cfg(test)] +mod fragment_determinism_tests; +#[cfg(test)] mod incremental_compile_tests; #[cfg(test)] mod ltm_char_tests; diff --git a/src/simlin-engine/src/db/assemble.rs b/src/simlin-engine/src/db/assemble.rs index d26383cb9..ad9d1c44d 100644 --- a/src/simlin-engine/src/db/assemble.rs +++ b/src/simlin-engine/src/db/assemble.rs @@ -8,8 +8,8 @@ //! Holds the table/metadata extraction helpers //! (`extract_tables_from_source_var`, `build_module_inputs`, //! `build_stub_variable`, `build_submodel_metadata`), the per-variable -//! compile+symbolize tail (`compile_phase_to_per_var_bytecodes` and the -//! `VarFragmentResult`/`PerVarOffsetMap` values), the production +//! emission tail (`compile_phase_to_per_var_bytecodes` and the +//! `VarFragmentResult`/`PerVarSizes` values), the production //! element-graph source `var_phase_symbolic_fragment_prod`, the resolved //! recurrence-SCC interleaver (`segment_member_by_element` / //! `combine_scc_fragment`), the salsa-tracked `assemble_module` / @@ -229,7 +229,7 @@ pub(crate) fn build_submodel_metadata<'arena>( sub_metadata.insert( var_ident.clone(), crate::compiler::VariableMetadata { - offset, + offset: Some(offset), size, var: stub, }, @@ -306,7 +306,7 @@ pub(crate) fn build_submodel_metadata<'arena>( sub_metadata.insert( var_ident, crate::compiler::VariableMetadata { - offset: entry.offset, + offset: Some(entry.offset), size: entry.size, var: stub, }, @@ -360,7 +360,7 @@ pub(crate) fn build_submodel_metadata<'arena>( sub_metadata.insert( var_ident, crate::compiler::VariableMetadata { - offset: entry.offset, + offset: Some(entry.offset), size: entry.size, var: stub, }, @@ -404,30 +404,30 @@ pub(crate) struct VarFragmentResult { pub flow_invariance: Option, } -/// Walk `exprs` and push every base slot offset referenced by a `Var`, +/// Walk `exprs` and push the NAME of every variable referenced by a `Var`, /// `Subscript`, or `StaticSubscript` node into `out`. /// -/// This is NOT the same as calling `exprs_are_invariant`: it collects offsets +/// This is NOT the same as calling `exprs_are_invariant`: it collects names /// without returning a verdict. It is used by `compute_flow_invariance_support` -/// to determine which mini-layout entries are actually referenced in the *flow* -/// expression (as opposed to the init expression, which also contributes to -/// the mini-layout but must not pollute the dep_names set). +/// to determine which variables are actually referenced in the *flow* +/// expression (as opposed to the init expression, which must not pollute the +/// dep_names set). /// /// The walk is exhaustive over every `Expr`/`BuiltinFn` variant. -fn collect_expr_offsets(exprs: &[crate::compiler::Expr], out: &mut HashSet) { +fn collect_expr_refs(exprs: &[crate::compiler::Expr], out: &mut HashSet>) { use crate::builtins::BuiltinFn; use crate::compiler::Expr; use crate::compiler::expr::SubscriptIndex; - fn walk(expr: &Expr, out: &mut HashSet) { + fn walk(expr: &Expr, out: &mut HashSet>) { match expr { - // Leaf: referenced slot. - Expr::Var(off, _) - | Expr::Subscript(off, _, _, _) - | Expr::StaticSubscript(off, _, _) => { - out.insert(*off); + // Leaf: referenced variable. + Expr::Var(var, _) + | Expr::Subscript(var, _, _, _) + | Expr::StaticSubscript(var, _, _) => { + out.insert(var.name.clone()); // For Subscript, also walk the index expressions (they may - // reference other slots). + // reference other variables). if let Expr::Subscript(_, indices, _, _) = expr { for idx in indices { match idx { @@ -592,24 +592,21 @@ fn collect_expr_offsets(exprs: &[crate::compiler::Expr], out: &mut HashSet, - offsets: &PerVarOffsetMap, - model_name_ident: &Ident, var_ident_canonical: &Ident, ) -> Option { - use crate::compiler::invariance::{OffsetClass, exprs_are_invariant}; + use crate::compiler::invariance::{RefClass, exprs_are_invariant}; use std::collections::BTreeSet; use std::sync::Arc; @@ -623,59 +620,27 @@ pub(crate) fn compute_flow_invariance_support( // Structural purity check: does the expression contain any variant // builtins (TIME, PULSE, RAMP, STEP, PREVIOUS, EvalModule, ModuleInput)? - // All offset lookups return Invariant so only the builtin arms matter. - let locally_pure = exprs_are_invariant(&flow_var.ast, &|_off| OffsetClass::Invariant); - - // Build the reverse-lookup: offset -> variable name from the mini-layout. - // An offset belongs to the variable whose range [base, base+size) contains it. - let model_offsets = offsets.get(model_name_ident); - - // Walk the flow expression to collect only the offsets actually referenced - // there (not init-only deps that also appear in the mini-layout). - let mut referenced_offsets: HashSet = HashSet::new(); - collect_expr_offsets(&flow_var.ast, &mut referenced_offsets); - - // Resolve each referenced offset to its owning variable name, excluding the - // variable itself (its own slot is offset 0 in the mini-layout, never a dep). - let var_name_str = var_ident_canonical.as_str(); - let dep_names: BTreeSet = match model_offsets { - None => BTreeSet::new(), - Some(mo) => { - // Build an (offset, name) list from the mini-layout for range lookup. - let ranges: Vec<(&Ident, usize, usize)> = mo - .iter() - .map(|(name, &(base, size))| (name, base, size)) - .collect(); - - referenced_offsets - .iter() - .filter_map(|&off| { - // Find the mini-layout entry whose [base, base+size) contains - // this offset. - let owner = ranges - .iter() - .find(|(_, base, size)| off >= *base && off < *base + *size) - .map(|(name, _, _)| name.as_str()); - // The mini-layout is contiguous over [0, total) and the time - // globals lower via LoadGlobalVar/Dt/App (never Expr::Var), - // so every flow-referenced offset must resolve to an owner. - // A silently dropped offset would make that dependency - // INVISIBLE to the invariance fixpoint -- the - // over-classification direction (stale values once B2 skips - // re-evaluating the invariant prefix) -- so be loud in debug - // builds; the end-to-end bit-constancy oracle is the - // release-build backstop. - debug_assert!( - owner.is_some(), - "flow-referenced offset {off} of '{var_name_str}' resolves to no mini-layout owner" - ); - owner - .filter(|name| *name != var_name_str) - .map(|name| name.to_string()) - }) - .collect() - } - }; + // All reference lookups return Invariant so only the builtin arms matter. + let locally_pure = exprs_are_invariant(&flow_var.ast, &|_var| RefClass::Invariant); + + // Walk the flow expression to collect only the variables actually + // referenced there (not init-only deps). + // + // This used to reverse-map each referenced slot offset through the + // fragment's private offset layout, with a `debug_assert!` guarding the case + // where an offset resolved to no owner -- a silently dropped dependency + // would be INVISIBLE to the invariance fixpoint, the over-classification + // direction. Reading the name off the reference removes the failure mode + // rather than guarding it: there is no lookup left to fail. + let mut referenced: HashSet> = HashSet::new(); + collect_expr_refs(&flow_var.ast, &mut referenced); + + // The variable's own references are self-references, never dependencies. + let dep_names: BTreeSet = referenced + .iter() + .filter(|name| *name != var_ident_canonical) + .map(|name| name.as_str().to_string()) + .collect(); Some(FlowInvarianceSupport { locally_pure, @@ -683,51 +648,111 @@ pub(crate) fn compute_flow_invariance_support( }) } -/// `model_name -> (var_name -> (offset, size))`: the per-variable mini- -/// layout offset map `lower_var_fragment` produces and the minimal -/// per-phase `crate::compiler::Module` consumes. Structurally identical to -/// `compiler::VariableOffsetMap` / `var_fragment::VarOffsets` (both -/// private aliases in their modules); named here so the factored -/// `compile_phase_to_per_var_bytecodes` signature is self-documenting -/// rather than an inline nested-`HashMap` (which clippy flags as a very -/// complex type). -pub(crate) type PerVarOffsetMap = - HashMap, HashMap, (usize, usize)>>; - -/// Compile one phase's lowered `Vec` for a single variable through -/// its own correct mini-context and symbolize the result into a +/// `reference -> extent of the variable it addresses in whole`: the per-fragment +/// size table `lower_var_fragment` produces, and which BOTH halves of the +/// compile borrow -- the lowering context (`compiler::ContextCore`) and the +/// per-phase emission context (`compiler::ModuleCtx`). This IS +/// `compiler::VarSizes` -- the fragment-side name for the same type, kept so the +/// db-side signatures read in fragment terms. +/// +/// Its predecessor carried `(offset, size)` per variable over a private +/// per-fragment layout that existed only so symbolization could undo it. There +/// are no fragment-local offsets any more: emission reads names, and the only +/// surviving question is how big a variable is. +pub(crate) type PerVarSizes = crate::compiler::VarSizes; + +/// Flatten a phase's temp-id -> size map into the `(temp_id, size)` vector +/// `PerVarBytecodes::temp_sizes` carries, **ordered by temp id**. +/// +/// The ordering is load-bearing, not cosmetic. `temp_sizes` rides on a +/// `PerVarBytecodes`, which is a salsa-cached value with a *derived* +/// `PartialEq`, and `Vec` equality is order-sensitive. Building it straight +/// out of `HashMap::iter` therefore made two otherwise-identical compiles of +/// the same fragment compare unequal whenever the per-process hash seed +/// reordered the map: salsa's backdating stops firing (every downstream +/// consumer re-executes), and the compiled artifact itself stops being +/// reproducible run to run -- the nondeterminism class GH #595 tracks. +/// +/// Nothing downstream ever *depended* on the order: the sole consumer, +/// `FragmentMerger::absorb`, folds each entry into a resize-and-`max` over a +/// dense `merged_temp_sizes` vector, which is order-independent, and the +/// merged result is re-emitted densely by `into_per_var_bytecodes`. So this is +/// a pure determinism fix with no bytecode consequence -- which is exactly why +/// it survived undetected. +pub(crate) fn temp_sizes_by_id(temp_sizes_map: &HashMap) -> Vec<(u32, usize)> { + let mut temp_sizes: Vec<(u32, usize)> = temp_sizes_map + .iter() + .map(|(&id, &size)| (id, size)) + .collect(); + temp_sizes.sort_unstable_by_key(|(id, _)| *id); + temp_sizes +} + +/// Assemble the phase-INVARIANT emission context for one variable's +/// fragment: everything `compile_phase_to_per_var_bytecodes` needs except +/// the phase's own lowered expressions (and the temp sizes derived from +/// them, which that function fills in). +/// +/// Every field is a borrow with the caller's lifetime -- the fragment's size +/// table and tables, and the salsa-cached project-global dimension context and +/// converted dimensions, are read in place, never copied into a per-fragment +/// container. +/// +/// The empty `runlist_flows`/`temp_sizes` placeholders exist in exactly one +/// place, here, so no emission site can forget which runlist a fragment's +/// expressions belong in. +pub(crate) fn fragment_emit_ctx<'a>( + model_name: &'a Ident, + inputs: &'a BTreeSet>, + var_sizes: &'a PerVarSizes, + tables: &'a HashMap, Vec>, + dimensions: &'a [crate::dimensions::Dimension], + dimensions_ctx: &'a crate::dimensions::DimensionsContext, +) -> crate::compiler::ModuleCtx<'a> { + crate::compiler::ModuleCtx { + ident: model_name, + inputs, + temp_sizes: &[], + runlist_initials_by_var: &[], + runlist_flows: &[], + runlist_stocks: &[], + var_sizes, + tables, + dimensions, + dimensions_ctx, + } +} + +/// Compile one phase's lowered `Vec` for a single variable into a /// layout-independent `PerVarBytecodes`. /// -/// This is the exact body of `compile_var_fragment`'s former -/// `compile_phase` closure, factored out so the element-cycle SCC graph -/// builder (`crate::db::dep_graph` via `var_phase_symbolic_fragment_prod`) -/// reuses the *exact* production compile+symbolize path rather than a -/// re-derivation. `compile_var_fragment` calls this for each phase; the -/// SCC accessor `var_phase_symbolic_fragment_prod` builds the caller-owned -/// context byte-identically to `compile_var_fragment` and calls this with -/// the phase's production-lowered exprs. +/// **This is the single fragment emission entry point** (GH #964's "explicit, +/// implicit, and LTM variables share one fragment emission implementation"). +/// Five call sites reach it: `compile_var_fragment` (explicit variables), +/// `compile_implicit_var_phase_bytecodes` (SMOOTH/DELAY/TREND helpers, and +/// through it `var_phase_symbolic_fragment_prod`'s parent-sourced arm), +/// `var_phase_symbolic_fragment_prod` itself (the element-cycle SCC graph +/// builder, which must reuse the *exact* production path rather than a +/// re-derivation), and both LTM emitters in `db/ltm/compile.rs`. The two LTM +/// sites used to carry hand-copied 97-line duplicates of this body, which is +/// how the same `temp_sizes` ordering defect came to need fixing in three +/// places at once. +/// +/// `base` is the phase-INVARIANT half of the emission context, built once +/// per variable by the caller: its `runlist_flows`/`temp_sizes` are ignored +/// (this function fills both in per phase), and everything else -- the +/// fragment's `var_sizes` and `tables`, the project-global +/// `dimensions`/`dimensions_ctx`, and the module-input set -- is borrowed for +/// the call and never cloned. /// -/// The caller owns and supplies the lowering-independent context -/// (`offsets`, `rmap`, `tables`, `module_refs`, `mini_offset`, -/// `converted_dims`, `dim_context`, `model_name_ident`, `inputs`) exactly -/// as `compile_var_fragment` constructs it. `var_ident_canonical` is the -/// single-variable runlist-order entry the minimal `Module` is built -/// around. Returns `None` (loud-safe, never panics) when `exprs` is -/// empty, the minimal `Module::compile()` fails, or any symbolization -/// step fails -- exactly the closure's original `None` arms. -#[allow(clippy::too_many_arguments)] +/// What comes back is codegen's output verbatim: it is already symbolic, so +/// there is nothing between emission and the salsa-cached fragment. +/// +/// Returns `None` (loud-safe, never panics) when `exprs` is empty or codegen +/// fails. pub(crate) fn compile_phase_to_per_var_bytecodes( + base: &crate::compiler::ModuleCtx<'_>, exprs: &[crate::compiler::Expr], - offsets: &PerVarOffsetMap, - rmap: &crate::compiler::symbolic::ReverseOffsetMap, - tables: &HashMap, Vec>, - module_refs: &HashMap, crate::vm::ModuleKey>, - mini_offset: usize, - converted_dims: &[crate::dimensions::Dimension], - dim_context: &crate::dimensions::DimensionsContext, - model_name_ident: &Ident, - var_ident_canonical: &Ident, - inputs: &BTreeSet>, ) -> Option { use crate::compiler::symbolic::PerVarBytecodes; @@ -735,92 +760,45 @@ pub(crate) fn compile_phase_to_per_var_bytecodes( return None; } - // Build a minimal Module for this phase - let runlist_initials_by_var = vec![]; - let module_inputs: HashSet> = inputs.iter().cloned().collect(); - let module = crate::compiler::Module { - ident: model_name_ident.clone(), - inputs: module_inputs, - n_slots: mini_offset, - n_temps: 0, - temp_sizes: vec![], - runlist_initials: vec![], - runlist_initials_by_var, - runlist_flows: exprs.to_vec(), - runlist_stocks: vec![], - offsets: offsets.clone(), - runlist_order: vec![var_ident_canonical.clone()], - tables: tables.clone(), - dimensions: converted_dims.to_vec(), - dimensions_ctx: dim_context.clone(), - module_refs: module_refs.clone(), - }; - // Extract temp sizes from expressions let mut temp_sizes_map: HashMap = HashMap::new(); for expr in exprs { crate::compiler::extract_temp_sizes_pub(expr, &mut temp_sizes_map); } - let n_temps = temp_sizes_map.len(); - let mut temp_sizes: Vec = vec![0; n_temps]; + let mut temp_sizes: Vec = vec![0; temp_sizes_map.len()]; for (id, size) in &temp_sizes_map { if (*id as usize) < temp_sizes.len() { temp_sizes[*id as usize] = *size; } } - // Update Module with temp info - let module = crate::compiler::Module { - n_temps, - temp_sizes: temp_sizes.clone(), - ..module + // A fragment is one variable's one phase, so the whole phase goes in the + // flows runlist; the initials/stocks runlists stay empty and the phase + // distinction is the caller's (which lowered `Vec` it passes). + let emit_ctx = crate::compiler::ModuleCtx { + runlist_flows: exprs, + temp_sizes: &temp_sizes, + ..*base }; - match module.compile() { - Ok(compiled) => { - // Symbolize the flows bytecode (we put everything in flows) - let sym_bc = - crate::compiler::symbolic::symbolize_bytecode(&compiled.compiled_flows, rmap) - .ok()?; - - let ctx = &*compiled.context; - let sym_views: Vec<_> = ctx - .static_views - .iter() - .map(|sv| crate::compiler::symbolic::symbolize_static_view(sv, rmap)) - .collect::, _>>() - .ok()?; - let sym_mods: Vec<_> = ctx - .modules - .iter() - .map(|md| crate::compiler::symbolic::symbolize_module_decl(md, rmap)) - .collect::, _>>() - .ok()?; - - let temp_sizes_vec: Vec<(u32, usize)> = - temp_sizes_map.iter().map(|(&k, &v)| (k, v)).collect(); - - let dim_lists: Vec> = ctx - .dim_lists - .iter() - .map(|(n, arr)| arr[..(*n as usize)].to_vec()) - .collect(); - - Some(PerVarBytecodes { - symbolic: sym_bc, - graphical_functions: ctx.graphical_functions.clone(), - module_decls: sym_mods, - static_views: sym_views, - temp_sizes: temp_sizes_vec, - dim_lists, - }) - } - Err(_) => None, - } + let compiled = emit_ctx.compile().ok()?; + + Some(PerVarBytecodes { + symbolic: compiled.compiled_flows, + graphical_functions: compiled.graphical_functions, + module_decls: compiled.module_decls, + static_views: compiled.static_views, + temp_sizes: temp_sizes_by_id(&temp_sizes_map), + dim_lists: compiled + .dim_lists + .iter() + .map(|(n, arr)| arr[..(*n as usize)].to_vec()) + .collect(), + }) } /// A variable's *symbolic* `PerVarBytecodes` for a phase, sourced through -/// the exact production compile+symbolize path (`lower_var_fragment` + +/// the exact production lowering + emission path (`lower_var_fragment` + /// `compile_phase_to_per_var_bytecodes`), never a re-derivation. /// /// This is the cross-member-comparable substrate the element-cycle SCC @@ -857,7 +835,7 @@ pub(crate) fn compile_phase_to_per_var_bytecodes( /// A synthetic helper (`$\u{205A}` prefix, absent from `model.variables`) /// that lands in a recurrence SCC is **parent-sourced**: its symbolic /// `PerVarBytecodes` is the parent variable's `implicit_vars[index]` -/// compiled+symbolized through the shared per-phase relation +/// compiled through the shared per-phase relation /// `compile_implicit_var_phase_bytecodes` (the same chain /// `compile_implicit_var_fragment` runs), so the element-graph builder /// consumes it exactly like a real member (element-cycle Phase 3 Task 2 / @@ -874,8 +852,7 @@ pub(crate) fn compile_phase_to_per_var_bytecodes( /// explicit `return None`; /// - the requested phase's `Var::new` errored (`phase_var.ok()?`); /// - any `compile_phase_to_per_var_bytecodes` failure (empty exprs, the -/// minimal `Module::compile()`, or any `symbolize_*` step) -- that -/// function is itself total-and-`None`-on-failure. +/// codegen) -- that function is itself total-and-`None`-on-failure. /// /// `None` propagates loud-safe and all-or-nothing: any in-SCC node that /// cannot be element-sourced makes `symbolic_phase_element_order` return @@ -921,11 +898,11 @@ pub(crate) fn var_phase_symbolic_fragment_prod( // (element-cycle Phase 3 Task 2 / AC3.1). A synthetic helper that // lands in a recurrence SCC has no `SourceVariable` but DOES resolve // in `model_implicit_var_info`; its symbolic `PerVarBytecodes` is the - // parent variable's `implicit_vars[index]` compiled+symbolized through + // parent variable's `implicit_vars[index]` compiled through // the SAME shared per-phase relation the production per-variable // assembly uses (`compile_implicit_var_phase_bytecodes` -- the exact // `parent → parsed.implicit_vars[i] → parse_var → lower_variable → - // compile → symbolize` chain `compile_implicit_var_fragment` runs), so + // compile` chain `compile_implicit_var_fragment` runs), so // the element-graph builder consumes it exactly like a real member // (same layout-independent `SymVarRef` form). The element-cycle SCC // identification uses the default no-module-input wiring, so source the @@ -944,7 +921,6 @@ pub(crate) fn var_phase_symbolic_fragment_prod( let is_initial = matches!(phase, SccPhase::Initial); return compile_implicit_var_phase_bytecodes(db, meta, model, project, &[], is_initial); }; - let var_ident_canonical: Ident = Ident::new(var_name); // Caller-owned, lowering-independent context, read EXACTLY as // `compile_var_fragment` reads it (mirror byte-for-byte): the @@ -968,27 +944,17 @@ pub(crate) fn var_phase_symbolic_fragment_prod( &inputs, ); - let (per_phase_lowered, tables, offsets, rmap, mini_offset) = match lowered { + let (per_phase_lowered, tables, var_sizes) = match lowered { LoweredVarFragment::Lowered { per_phase_lowered, tables, - offsets, - rmap, - mini_offset, + var_sizes, .. - } => (per_phase_lowered, tables, offsets, rmap, mini_offset), + } => (per_phase_lowered, tables, var_sizes), // The variable did not lower at all => `None` (loud-safe). LoweredVarFragment::Fatal { .. } => return None, }; - // The element-cycle SCC identification uses the default no-module- - // input wiring, so the module-ref reconstruction must match that - // wiring too (mirrors `compile_var_fragment`'s - // `build_caller_module_refs(.., &module_input_names)` with empty - // inputs). - let module_refs = - crate::db::var_fragment::build_caller_module_refs(db, *sv, model, project, &[]); - // `SccPhase::Dt` selects the non-initial (dt/flow) lowering; // `SccPhase::Initial` selects the initial lowering -- the same // selection `compile_var_fragment` makes per phase. @@ -1000,19 +966,15 @@ pub(crate) fn var_phase_symbolic_fragment_prod( // lowered exprs => `None` (loud-safe). let var = phase_var.ok()?; - compile_phase_to_per_var_bytecodes( - &var.ast, - &offsets, - &rmap, + let base_ctx = fragment_emit_ctx( + &model_name_ident, + &inputs, + &var_sizes, &tables, - &module_refs, - mini_offset, converted_dims, dim_context, - &model_name_ident, - &var_ident_canonical, - &inputs, - ) + ); + compile_phase_to_per_var_bytecodes(&base_ctx, &var.ast) } /// Segment one member's symbolic opcode stream into per-element slices, @@ -1039,7 +1001,18 @@ pub(crate) fn var_phase_symbolic_fragment_prod( /// - a duplicate write for the same element (ambiguous segmentation); /// - opcodes present but no per-element write at all (not element- /// sourceable in the simple per-element shape, mirroring -/// `symbolic_phase_element_order`'s `saw_write` guard). +/// `symbolic_phase_element_order`'s `saw_write` guard); +/// - a backward jump whose target lies in an EARLIER segment. Segments are +/// emitted in `element_order`, not in their original order, and a jump +/// offset is relative, so a jump that escaped its own segment would land on +/// whatever opcode happened to sit that far back after the interleave -- a +/// silent miscompile with no bad id and no bad reference to notice. Codegen +/// cannot currently produce one (a `BeginIter` loop writes a TEMP through +/// `StoreIterElement`, and a member's per-element `AssignCurr` is emitted +/// after `EndIter`, so a loop is always wholly inside one segment), which is +/// exactly why it is worth checking rather than asserting in prose: this is +/// an assumption about a DIFFERENT file's emission shape, and nothing else +/// would notice it changing. /// /// Consumed by `combine_scc_fragment`, which `assemble_module` invokes /// for every resolved recurrence SCC (the Subcomponent B Task 6 @@ -1059,28 +1032,74 @@ fn segment_member_by_element( }; let body = &code[..end]; - let mut segments: HashMap> = HashMap::new(); - let mut current: Vec = Vec::new(); - let mut last_written_elem: Option = None; - - for op in body { - current.push(op.clone()); - let write_elem = match op { + // The opcode that terminates one of THIS member's per-element segments. A + // write to a *different* member, or a `BinOpAssignNext` (a stock update, + // not a per-element current-value write of this member), does not -- + // exactly the `symbolic_phase_element_order` rule. + let write_element = |op: &SymbolicOpcode| -> Option { + match op { SymbolicOpcode::AssignCurr { var } | SymbolicOpcode::AssignConstCurr { var, .. } | SymbolicOpcode::BinOpAssignCurr { var, .. } - if var.name == member => + if var.name.as_str() == member => { Some(var.element_offset) } - // A write to a *different* member, or AssignNext/ - // BinOpAssignNext (a stock-update, not a per-element - // current-value write of THIS member) does not terminate this - // member's element segment -- exactly the - // `symbolic_phase_element_order` rule. _ => None, + } + }; + + // Jump containment. `lower_bound[pc]` is the first index of the segment + // `pc` ends up in: segments start just after the previous per-element + // write, except that opcodes trailing the FINAL write are appended to the + // last segment rather than starting a new one (see below), so their bound + // is that segment's start. + let mut lower_bound: Vec = Vec::with_capacity(body.len()); + let mut start = 0usize; + for (pc, op) in body.iter().enumerate() { + lower_bound.push(start); + if write_element(op).is_some() { + start = pc + 1; + } + } + if let Some(last_write) = body.iter().rposition(|op| write_element(op).is_some()) + && last_write + 1 < body.len() + { + let last_segment_start = lower_bound[last_write]; + for bound in &mut lower_bound[last_write + 1..] { + *bound = last_segment_start; + } + } + for (pc, op) in body.iter().enumerate() { + let Some(offset) = op.jump_offset() else { + continue; }; - if let Some(elem) = write_elem { + let target = pc as isize + offset as isize; + // Backward-or-self and not before the segment's own start. The second + // half is what segment reordering needs; the first is what makes the + // lower bound sufficient, and it holds for every jump the instruction + // set can express (`jump_offset` reports only backward jumps). A + // future forward jump would need its own upper-bound reasoning, and + // should trip this rather than silently inherit an argument that does + // not cover it. + if target < lower_bound[pc] as isize || target > pc as isize { + return Err(format!( + "SCC member `{member}` has a jump at opcode {pc} targeting \ + {target}, outside the per-element segment starting at {}; \ + the segments are reordered by element_order, so a jump that \ + leaves its own segment cannot be relocated safely", + lower_bound[pc] + )); + } + } + + let mut segments: HashMap> = HashMap::new(); + let mut current: Vec = Vec::new(); + let mut last_written_elem: Option = None; + + for op in body { + current.push(op.clone()); + if let Some(elem) = write_element(op) { if segments.contains_key(&elem) { return Err(format!( "SCC member `{member}` has a duplicate per-element \ @@ -1124,7 +1143,7 @@ fn segment_member_by_element( /// `member_fragments` maps each SCC member's canonical name to its /// *symbolic* `PerVarBytecodes` for the SCC's phase (obtained by the /// caller via `var_phase_symbolic_fragment_prod(.., scc.phase)` -- the -/// exact production compile+symbolize path, never a re-derivation). The +/// exact production emission path, never a re-derivation). The /// result is a single fragment whose per-element writes appear in /// `scc.element_order`, with each write keeping its **original** /// `SymVarRef { name, element_offset }` (only segment ordering changes). @@ -1248,6 +1267,73 @@ pub(crate) fn combine_scc_fragment( Ok(merger.into_per_var_bytecodes(combined_code)) } +/// Renumber the INITIALS phase into one `SymbolicCompiledInitial` per fragment. +/// +/// The other two phases go through `concatenate_fragments_with_gf`, which +/// merges their fragments into a single stream. Initials cannot: `eval_initials` +/// runs them one at a time, so each keeps its own bytecode -- and therefore its +/// own literal pool, which is why every initial is renumbered at literal offset +/// 0 rather than at a running one. +/// +/// Everything else must still land where the all-phases merge put it (M8): the +/// module / view / dim-list offsets are the running counts over the preceding +/// initials, exactly the bases `FragmentMerger::absorb_non_gf` would hand out, +/// and the GF base comes from the shared dedup (initial `i` is `all_frags[i]`, +/// since initials come first). Temps are NOT advanced (#583) -- they recycle +/// into the one identity pool every phase shares. +/// +/// This is a third hand-rolled copy of the merger's flat accounting, kept +/// because of the literal-pool difference above. It is a free function rather +/// than an inline loop specifically so the M8 property can drive it: as an +/// inline loop, freezing `init_view_off` or `init_dl_off` left the entire +/// repository green. +/// +/// `Err` on M3, for the same reason `absorb_non_gf` checks its own bases: these +/// offsets index the all-phases tables the module reports, so a silently +/// truncated one produces a well-formed initial that names a different module, +/// view or dimension list. The bound itself is not restated here -- the running +/// offsets are `usize` and each is narrowed by `symbolic::resource_base`, the +/// one place that knows both a base and the length of the fragment about to use +/// it, so an initial that carries none of a resource is exempt exactly as a +/// merged fragment is. +pub(crate) fn renumber_initials_phase( + initial_frags: &[(String, &crate::compiler::symbolic::PerVarBytecodes)], + gf_dedup: &crate::compiler::symbolic::GfDedup, +) -> Result, String> { + use crate::compiler::symbolic::{ + SymbolicByteCode, SymbolicCompiledInitial, renumber_opcode, resource_base, + }; + + let mut compiled_initials: Vec = Vec::new(); + let mut mod_off: usize = 0; + let mut view_off: usize = 0; + let temp_off: u32 = 0; + let mut dl_off: usize = 0; + for (i, (name, bc)) in initial_frags.iter().enumerate() { + let gf_remap = gf_dedup.remap(i); + let mod_base = resource_base(0, mod_off, bc.module_decls.len(), "module declaration")?; + let view_base = resource_base(0, view_off, bc.static_views.len(), "static view")?; + let dl_base = resource_base(0, dl_off, bc.dim_lists.len(), "dimension list")?; + let renumbered_code = bc + .symbolic + .code + .iter() + .map(|op| renumber_opcode(op, 0, gf_remap, mod_base, view_base, temp_off, dl_base)) + .collect::, _>>()?; + compiled_initials.push(SymbolicCompiledInitial { + ident: Ident::new(name), + bytecode: SymbolicByteCode { + literals: bc.symbolic.literals.clone(), + code: renumbered_code, + }, + }); + mod_off += bc.module_decls.len(); + view_off += bc.static_views.len(); + dl_off += bc.dim_lists.len(); + } + Ok(compiled_initials) +} + /// Assemble a complete CompiledModule from per-variable fragments. /// /// Salsa-tracked: the per-module assembly (fragment concatenation, SCC @@ -1272,8 +1358,8 @@ pub fn assemble_module<'db>( module_inputs: ModuleInputSet<'db>, ) -> Result, String> { use crate::compiler::symbolic::{ - ContextResourceCounts, SymbolicCompiledInitial, SymbolicCompiledModule, - concatenate_fragments_with_gf, resolve_module, + ContextResourceCounts, SymbolicCompiledModule, concatenate_fragments_with_gf, + merge_context_side_channels, resolve_module, }; // The interned set stores the sorted canonical names; the plain lowering @@ -1440,7 +1526,7 @@ pub fn assemble_module<'db>( // one-contiguous-block-per-variable fragments -- the latter cannot // express the required cross-member per-element interleaving. Each // member's symbolic fragment is sourced via the EXACT production - // compile+symbolize path (`var_phase_symbolic_fragment_prod`, the + // emission path (`var_phase_symbolic_fragment_prod`, the // Task 4 accessor -- never a re-derivation), so every write keeps its // original `SymVarRef { name, element_offset }`; `resolve_module` // therefore maps each write to the same model slot the acyclic layout @@ -1737,6 +1823,13 @@ pub fn assemble_module<'db>( temps: 0, ..initial_counts.clone() }; + // Summed in `usize` and left there. M3 -- every ASSIGNED id is + // representable -- is discharged by `symbolic::resource_base` when a stock + // fragment actually consumes one of these bases, which is the only point + // where "the table is full" and "this fragment wants an id past the end" + // can be told apart. Bounding the sum here instead would reject a model + // whose initials and flows fill the table exactly and whose stocks phase + // then names nothing -- a program in which every id is valid. let stock_base = ContextResourceCounts { modules: initial_counts.modules + flow_counts.modules, views: initial_counts.views + flow_counts.views, @@ -1771,57 +1864,19 @@ pub fn assemble_module<'db>( let stocks_concat = concatenate_fragments_with_gf(&stock_frags, &stock_base, &gf_dedup, n_init + n_flow)?; - // Build SymbolicCompiledInitial for each initial variable, renumbered - // so context resource IDs (GFs, modules, views, temps, dim_lists) match - // the all-phases merge. Literal IDs are local to each initial's bytecode - // so they get no base offset. The GF base comes from the shared dedup - // (initial `i` is `all_frags[i]`); the other resources stay flat. - let mut compiled_initials: Vec = Vec::new(); - let mut init_mod_off: u16 = 0; - let mut init_view_off: u16 = 0; - // #583: temps recycle into the shared identity pool (the same pool the - // `merged` table below builds), so each initial's temp ids stay - // fragment-local (offset 0) -- they are NOT advanced per initial. - let init_temp_off: u32 = 0; - let mut init_dl_off: u16 = 0; - for (i, (name, bc)) in initial_frags.iter().enumerate() { - let gf_remap = gf_dedup.remap(i); - let renumbered_code: Vec = bc - .symbolic - .code - .iter() - .map(|op| { - crate::compiler::symbolic::renumber_opcode( - op, - 0, // literals are local to each initial's bytecode - gf_remap, - init_mod_off, - init_view_off, - init_temp_off, - init_dl_off, - ) - }) - .collect::, _>>()?; - compiled_initials.push(SymbolicCompiledInitial { - ident: Ident::new(name), - bytecode: crate::compiler::symbolic::SymbolicByteCode { - literals: bc.symbolic.literals.clone(), - code: renumbered_code, - }, - }); - init_mod_off += bc.module_decls.len() as u16; - init_view_off += bc.static_views.len() as u16; - // `init_temp_off` is NOT advanced (#583): temps recycle into the - // shared identity pool, so every initial's temp ids stay - // fragment-local and index the same `merged.temp_offsets` table. - init_dl_off += bc.dim_lists.len() as u16; - } + let compiled_initials = renumber_initials_phase(&initial_frags, &gf_dedup)?; - // The all-phases merge for the shared context side-channels (modules, - // views, temps, dim_lists); its `graphical_functions` is the dedup's - // single table (set by `concatenate_fragments_with_gf`), shared by all - // three phases. - let merged = concatenate_fragments_with_gf(&all_frags, &no_base, &gf_dedup, 0)?; + // The all-phases aggregation of the shared context side-channels (modules, + // views, temps, dim_lists); its `graphical_functions` is the dedup's single + // table, shared by all three phases. + // + // This is `merge_context_side_channels` and not a full merge because the + // three phases above already keep every stream this module retains. A full + // merge additionally built an opcode stream and a literal pool spanning the + // whole model, both dropped on the floor here -- and bounded the model's + // AGGREGATE literal count against the `u16` id capacity in the process, + // failing assembly for models whose every retained pool was fine. + let merged = merge_context_side_channels(&all_frags, &no_base, &gf_dedup)?; // Build dimension metadata from project dimensions (mirrors // Compiler::populate_dimension_metadata). Read the project-global converted @@ -1867,7 +1922,6 @@ pub fn assemble_module<'db>( // Build the symbolic compiled module let sym_module = SymbolicCompiledModule { ident: Ident::new(&model_name), - n_slots: layout.n_slots, compiled_initials, compiled_flows: flows_concat.bytecode, compiled_stocks: stocks_concat.bytecode, @@ -1885,8 +1939,7 @@ pub fn assemble_module<'db>( }; // Resolve symbolic -> concrete offsets. The CompiledModule stays a pure, - // symbolizable artifact (the symbolic roundtrip tests symbolize it again, - // and salsa caches it); the 3-address fusion (R2) is applied later, at + // salsa-cached artifact; the 3-address fusion (R2) is applied later, at // Vm::new, to the execution copy of the bytecode. The success payload is // wrapped in an `Arc` so this tracked fn's return type is `salsa::Update` // and salsa's clone-out is a refcount bump (the inner bytecode is large). diff --git a/src/simlin-engine/src/db/combined_fragment_proptest.rs b/src/simlin-engine/src/db/combined_fragment_proptest.rs new file mode 100644 index 000000000..df7d68661 --- /dev/null +++ b/src/simlin-engine/src/db/combined_fragment_proptest.rs @@ -0,0 +1,610 @@ +// Copyright 2026 The Simlin Authors. All rights reserved. +// Use of this source code is governed by the Apache License, +// Version 2.0, that can be found in the LICENSE file. + +//! Property tests for `combine_scc_fragment` -- the INTERLEAVING consumer of +//! `FragmentMerger`, and therefore the other half of obligations M5 and M6. +//! +//! `compiler::symbolic_merge_proptest` covers the sequential consumer, where +//! each fragment's opcodes are one contiguous run. Everything that makes that +//! emission safe changes here: a resolved recurrence SCC's members are cut into +//! per-element segments and re-emitted in `element_order`, so `ce[0]` runs, +//! then `ecc[0]`, then `ce[1]`. Members' temp live ranges therefore OVERLAP, +//! and the two properties this file adds are exactly the ones that overlap +//! makes non-trivial: +//! +//! * **M5** -- interleaved members must never share a temp slot. In the +//! sequential case sharing is safe and necessary; here it would alias two +//! simultaneously-live scratch arrays and silently miscompile the SCC. Stated +//! as the same non-interleaving condition the sequential property uses, so +//! the two are the same rule applied to two emission shapes rather than two +//! different beliefs. +//! +//! * **M6** -- the interleave is a pure REORDERING. Every member opcode +//! survives exactly once, each per-element write keeps its original +//! `SymVarRef` (which is what lets `resolve_module` place it at the same slot +//! the acyclic layout would, AC2.3), and the writes come out in +//! `element_order`. +//! +//! The third obligation the reordering creates -- that a relative backward jump +//! must not leave its own segment -- is pinned by hand-built fixtures in +//! `combined_fragment_tests.rs` instead, beside that file's other loud-safe +//! segmentation tests: the shape needs an exactly-placed jump, which a +//! generator would only reach by accident. +//! +//! `combined_fragment_tests.rs` pins the same contract on two hand-built +//! `ref.mdl`-shaped members. These properties widen it to arbitrary member +//! counts, element counts, resource loads and interleave orders -- the axes a +//! two-member fixture cannot vary. + +use proptest::prelude::*; +use proptest::test_runner::TestCaseError; + +use super::combine_scc_fragment; +use crate::common::{Canonical, Ident}; +use crate::compiler::symbolic::{ + PerVarBytecodes, SymStaticViewBase, SymVarRef, SymbolicByteCode, SymbolicModuleDecl, + SymbolicOpcode, SymbolicStaticView, +}; +use crate::db::{ResolvedScc, SccPhase}; +use smallvec::SmallVec; +use std::collections::{BTreeSet, HashMap, HashSet}; + +/// One SCC member: `n_elements` per-element segments, each carrying a slice of +/// the member's resources and terminated by that element's write. +#[derive(Clone, Debug)] +struct MemberSpec { + /// 0-based index; the member is named `m{index}`. + index: usize, + n_elements: usize, + n_literals: usize, + n_temps: usize, + n_modules: usize, + n_views: usize, + n_gf_tables: usize, + /// Emit a self-contained iteration loop inside the first element's + /// segment, so the properties see a backward jump. + with_loop: bool, +} + +fn member_spec() -> impl Strategy { + ( + 1usize..4, // n_elements + 0usize..3, // n_literals + 0usize..3, // n_temps + 0usize..2, // n_modules + 0usize..3, // n_views + 0usize..3, // n_gf_tables + any::(), + ) + .prop_map( + |(n_elements, n_literals, n_temps, n_modules, n_views, n_gf_tables, with_loop)| { + MemberSpec { + index: 0, + n_elements, + n_literals, + n_temps, + n_modules, + n_views, + n_gf_tables, + with_loop, + } + }, + ) +} + +/// Two to four members, indexed so their names and resource values differ. +fn member_specs() -> impl Strategy> { + prop::collection::vec(member_spec(), 2..5).prop_map(|mut specs| { + for (i, spec) in specs.iter_mut().enumerate() { + spec.index = i; + } + specs + }) +} + +fn member_name(index: usize) -> Ident { + Ident::new(&format!("m{index}")) +} + +fn vref(name: &Ident, element_offset: usize) -> SymVarRef { + SymVarRef { + name: name.clone(), + element_offset, + } +} + +fn build_member(spec: &MemberSpec) -> PerVarBytecodes { + let name = member_name(spec.index); + let tag = spec.index + 1; + + let literals: Vec = (0..spec.n_literals) + .map(|i| (tag * 1000 + i) as f64) + .collect(); + let graphical_functions: Vec> = (0..spec.n_gf_tables) + .map(|i| vec![(i as f64, (tag * 10 + i) as f64)]) + .collect(); + let module_decls: Vec = (0..spec.n_modules) + .map(|i| SymbolicModuleDecl { + model_name: Ident::new(&format!("sub{tag}_{i}")), + input_set: BTreeSet::new(), + var: vref(&name, i), + }) + .collect(); + let static_views: Vec = (0..spec.n_views) + .map(|i| SymbolicStaticView { + base: if spec.n_temps > 0 && i % 2 == 0 { + SymStaticViewBase::Temp((i % spec.n_temps) as u32) + } else { + SymStaticViewBase::Var(vref(&name, i)) + }, + dims: smallvec::smallvec![(tag * 10 + i) as u16], + strides: smallvec::smallvec![1], + offset: i as u32, + sparse: SmallVec::new(), + dim_ids: SmallVec::new(), + }) + .collect(); + let temp_sizes: Vec<(u32, usize)> = (0..spec.n_temps).map(|i| (i as u32, i + 1)).collect(); + let dim_lists: Vec> = (0..spec.n_elements) + .map(|i| vec![(tag + i) as u16]) + .collect(); + + let mut code: Vec = Vec::new(); + for e in 0..spec.n_elements { + if e < literals.len() { + code.push(SymbolicOpcode::LoadConstant { id: e as u16 }); + } + if e < graphical_functions.len() { + code.push(SymbolicOpcode::Lookup { + base_gf: e as u8, + table_count: 1, + mode: crate::bytecode::LookupMode::Interpolate, + }); + } + if e < module_decls.len() { + code.push(SymbolicOpcode::EvalModule { + id: e as u16, + n_inputs: 0, + }); + } + if e < static_views.len() { + code.push(SymbolicOpcode::PushStaticView { view_id: e as u16 }); + code.push(SymbolicOpcode::PopView {}); + } + if e < dim_lists.len() { + code.push(SymbolicOpcode::PushVarViewDirect { + var: SymVarRef::base(name.clone()), + dim_list_id: e as u16, + }); + code.push(SymbolicOpcode::PopView {}); + } + if e < temp_sizes.len() { + code.push(SymbolicOpcode::VectorSortOrder { + write_temp_id: e as u8, + }); + code.push(SymbolicOpcode::LoadTempConst { + temp_id: e as u8, + index: 0, + }); + } + if spec.with_loop && e == 0 && spec.n_temps > 0 { + // Wholly inside this element's segment: `BeginIter` and the + // backward jump that targets inside it both precede the write. + code.push(SymbolicOpcode::BeginIter { + write_temp_id: 0, + has_write_temp: true, + }); + code.push(SymbolicOpcode::LoadIterViewAt { offset: 0 }); + code.push(SymbolicOpcode::StoreIterElement {}); + code.push(SymbolicOpcode::NextIterOrJump { jump_back: -2 }); + code.push(SymbolicOpcode::EndIter {}); + } + code.push(SymbolicOpcode::AssignCurr { + var: vref(&name, e), + }); + } + code.push(SymbolicOpcode::Ret); + + PerVarBytecodes { + symbolic: SymbolicByteCode { literals, code }, + graphical_functions, + module_decls, + static_views, + temp_sizes, + dim_lists, + } +} + +fn build_members(specs: &[MemberSpec]) -> HashMap, PerVarBytecodes> { + specs + .iter() + .map(|spec| (member_name(spec.index), build_member(spec))) + .collect() +} + +/// An element order that INTERLEAVES the members: element 0 of every member, +/// then element 1 of every member that has one, and so on. This is the shape +/// that makes temp sharing unsound, so generating it (rather than a +/// member-at-a-time order) is what gives the M5 property its teeth. +fn interleaved_order(specs: &[MemberSpec]) -> Vec<(Ident, usize)> { + let max_elements = specs.iter().map(|s| s.n_elements).max().unwrap_or(0); + let mut order = Vec::new(); + for e in 0..max_elements { + for spec in specs { + if e < spec.n_elements { + order.push((member_name(spec.index), e)); + } + } + } + order +} + +fn scc_of(order: Vec<(Ident, usize)>) -> ResolvedScc { + ResolvedScc { + members: order.iter().map(|(m, _)| m.clone()).collect(), + element_order: order, + phase: SccPhase::Dt, + } +} + +/// Every temp slot the combined stream references, paired with the position of +/// the opcode that references it. +/// +/// Two channels, and the second is the one that matters here. An opcode can +/// carry a temp id directly, OR a `PushStaticView { view_id }` can push a view +/// whose base is a `Temp` -- how an array-producing builtin's scratch array is +/// read back. `absorb_non_gf` renumbers that base with the same per-fragment +/// `temp_offset` it renumbers opcodes with, so under `Sum` it must advance per +/// member exactly as the opcode ids do. +/// +/// Scanning only channel 1 left a real defect green across the whole +/// repository: a merger that shifted a view's `Temp` base by `ctx_base.temps` +/// instead of the per-fragment offset (correct under `Recycle`, wrong under +/// `Sum`) passed lib 5365/0 and `file_io` 637/0. Every member after the first +/// in a resolved recurrence SCC would read the FIRST member's scratch array -- +/// two simultaneously-live temps aliased, M5's exact failure mode, arriving +/// through the channel the scan did not look at. +fn temp_uses(code: &[SymbolicOpcode], views: &[SymbolicStaticView]) -> Vec<(usize, u32)> { + code.iter() + .enumerate() + .filter_map(|(pc, op)| { + let id: u32 = match op { + SymbolicOpcode::PushTempView { temp_id, .. } + | SymbolicOpcode::LoadTempConst { temp_id, .. } + | SymbolicOpcode::LoadTempDynamic { temp_id } + | SymbolicOpcode::LoadIterTempElement { temp_id } => *temp_id as u32, + SymbolicOpcode::BeginIter { + write_temp_id, + has_write_temp: true, + } => *write_temp_id as u32, + SymbolicOpcode::BeginBroadcastIter { dest_temp_id, .. } => *dest_temp_id as u32, + SymbolicOpcode::VectorElmMap { write_temp_id, .. } + | SymbolicOpcode::VectorSortOrder { write_temp_id } + | SymbolicOpcode::Rank { write_temp_id } + | SymbolicOpcode::LookupArray { write_temp_id, .. } + | SymbolicOpcode::AllocateAvailable { write_temp_id } + | SymbolicOpcode::AllocateByPriority { write_temp_id } => *write_temp_id as u32, + SymbolicOpcode::PushStaticView { view_id } => { + // Loud on an id the view table cannot answer. A silent + // `None` here is the exact shape this scan exists to + // eliminate -- a temp channel that goes unread -- so an + // out-of-range id must not look like "this opcode uses no + // temp". (M1's `denote` reports it too; this is the + // belt-and-braces inside the scan itself.) + let view = views.get(*view_id as usize).unwrap_or_else(|| { + panic!( + "PushStaticView names view {view_id} but the table holds {}", + views.len() + ) + }); + match &view.base { + SymStaticViewBase::Temp(id) => *id, + SymStaticViewBase::Var(_) => return None, + } + } + _ => return None, + }; + Some((pc, id)) + }) + .collect() +} + +/// Which member each opcode of the combined stream came from, read off the +/// per-element write that terminates each segment. +/// +/// Derived from `element_order` rather than from the combined stream's own +/// writes, so it does not assume the thing M6 is about to check. +fn owner_per_opcode( + combined: &PerVarBytecodes, + order: &[(Ident, usize)], + members: &HashMap, PerVarBytecodes>, +) -> Vec> { + // Segment lengths, recomputed from each member's own fragment the same way + // the combiner cuts them: up to and including that element's write, with + // any tail after the final write joining the last segment. + let mut owners = Vec::new(); + for (member, elem) in order { + let frag = &members[member]; + let len = segment_len(frag, member.as_str(), *elem); + for _ in 0..len { + owners.push(member.clone()); + } + } + // The combined fragment's single terminal `Ret` has no owner. + debug_assert_eq!(owners.len() + 1, combined.symbolic.code.len()); + owners +} + +/// The length of `member`'s segment for element `elem`, computed from the +/// member's OWN fragment. +/// +/// This re-derives the segmentation rule `segment_member_by_element` applies, +/// which is the pattern this file's header criticises in the tests it replaces. +/// It is admissible here for a reason that did not hold there, and the +/// difference is worth stating rather than leaving the reader to infer. +/// +/// The deleted temp tests used their re-derivation as the ORACLE: the value it +/// produced was the expected answer, so the test could only fail if the two +/// implementations disagreed -- and since that oracle's output was itself +/// asserted equal to a literal, it could not even do that. Here the +/// re-derivation is not the answer to anything. It only attributes merged +/// opcodes to the member that produced them, so the real assertions (no shared +/// temp slot; writes in `element_order`) have owners to talk about. +/// +/// And it is self-checking rather than trusted: `owner_per_opcode` ends in +/// `debug_assert_eq!(owners.len() + 1, combined.symbolic.code.len())`, so if +/// this derivation drifts from production's -- for any reason, including +/// production changing -- the totals stop matching and the test panics rather +/// than quietly attributing opcodes to the wrong member. +fn segment_len(frag: &PerVarBytecodes, member: &str, elem: usize) -> usize { + let code = &frag.symbolic.code; + let end = if code.last() == Some(&SymbolicOpcode::Ret) { + code.len() - 1 + } else { + code.len() + }; + let body = &code[..end]; + let is_write = |op: &SymbolicOpcode| -> Option { + match op { + SymbolicOpcode::AssignCurr { var } + | SymbolicOpcode::AssignConstCurr { var, .. } + | SymbolicOpcode::BinOpAssignCurr { var, .. } + if var.name.as_str() == member => + { + Some(var.element_offset) + } + _ => None, + } + }; + let mut start = 0usize; + let last_write = body.iter().rposition(|op| is_write(op).is_some()); + for (pc, op) in body.iter().enumerate() { + if let Some(e) = is_write(op) { + let is_last = Some(pc) == last_write; + let end = if is_last { body.len() } else { pc + 1 }; + if e == elem { + return end - start; + } + start = pc + 1; + } + } + 0 +} + +proptest! { + #![proptest_config(ProptestConfig::with_cases(128))] + + /// M5 for the interleaving emitter, stated as the SAME non-interleaving + /// condition the sequential property uses: tag every temp use in the + /// combined stream with the member that produced it, and require that no + /// merged temp slot is used by a member, then another, then the first + /// again. + /// + /// Under `element_order` the members' segments alternate, so the ONLY way + /// to satisfy this is for members not to share slots at all -- which is + /// what `TempStrategy::Sum` gives them. The property is therefore not a + /// restatement of "use Sum here"; it is the reason Sum is required, and it + /// would fail immediately if this call site were switched to `Recycle`. + #[test] + fn interleaved_members_never_share_a_temp_slot(specs in member_specs()) { + check_no_shared_temp_slot(&specs, false)?; + } +} + +proptest! { + #![proptest_config(ProptestConfig::with_cases(64))] + + /// Non-vacuity guard for the property above, aimed squarely at the SECOND + /// temp channel. + /// + /// The base strategy only sometimes draws a member with both temps and + /// views, and a case with no `Temp`-based static view exercises nothing of + /// `temp_uses`' view arm. `forced_view_backed_temp_specs` gives every + /// member both, and `check_no_shared_temp_slot(.., require_view_temps = + /// true)` asserts a `Temp`-based view is actually present in the combined + /// output before checking anything. + /// + /// This is the guard the missing channel needed: without it, a scan that + /// silently stopped resolving views would keep passing on whatever the + /// draw happened to produce. + #[test] + fn forced_view_backed_temps_are_never_shared(specs in forced_view_backed_temp_specs()) { + check_no_shared_temp_slot(&specs, true)?; + } +} + +/// M5 for the interleaving emitter, over both temp channels. Shared by the +/// arbitrary and the forced-view properties. +fn check_no_shared_temp_slot( + specs: &[MemberSpec], + require_view_temps: bool, +) -> Result<(), TestCaseError> { + let members = build_members(specs); + let order = interleaved_order(specs); + let scc = scc_of(order.clone()); + let combined = combine_scc_fragment(&scc, &members) + .map_err(|e| TestCaseError::fail(format!("well-formed members must combine: {e}")))?; + + if require_view_temps { + // Assert the COUPLING, not two independent counts. "Some view is + // Temp-based" and "some view is pushed" can both hold while the pushed + // ones are all `Var`-based and the view arm of `temp_uses` never + // resolves anything -- the guard would report covered with the channel + // unreached, which is the exact failure mode this property exists to + // prevent. So: count the entries `temp_uses` produced that sit at a + // `PushStaticView` position. Those are, by construction, temps reached + // THROUGH a view. + let via_view = temp_uses(&combined.symbolic.code, &combined.static_views) + .into_iter() + .filter(|(pc, _)| { + matches!( + combined.symbolic.code[*pc], + SymbolicOpcode::PushStaticView { .. } + ) + }) + .count(); + prop_assert!( + via_view >= 2, + "the view arm of `temp_uses` must resolve at least two temps, or the \ + second temp channel is untested; it resolved {}", + via_view + ); + } + + let owners = owner_per_opcode(&combined, &order, &members); + let mut runs: HashMap>> = HashMap::new(); + for (pc, temp) in temp_uses(&combined.symbolic.code, &combined.static_views) { + if pc >= owners.len() { + continue; // the terminal Ret + } + let owner = &owners[pc]; + let seen = runs.entry(temp).or_default(); + if seen.last() != Some(owner) { + prop_assert!( + !seen.contains(owner), + "combined temp slot {} is used by `{}` again after another member \ + used it -- interleaved members' live ranges overlap, so a shared \ + slot aliases two live scratch arrays", + temp, + owner.as_str() + ); + seen.push(owner.clone()); + } + } + + // The stronger consequence for this emitter: no slot has two owners at + // all. Kept alongside the general condition rather than instead of it, + // because it is what `Sum` specifically buys and it fails on a + // different mutation (a `Sum` that forgot to advance). + for (temp, seen) in &runs { + prop_assert_eq!( + seen.len(), + 1, + "combined temp slot {} has {} distinct member owners", + temp, + seen.len() + ); + } + + Ok(()) +} + +/// Every member carries temps AND views based on them, so the second temp +/// channel is exercised in every case. +fn forced_view_backed_temp_specs() -> impl Strategy> { + let member = + (1usize..3, 1usize..3, 1usize..4).prop_map(|(n_elements, n_temps, n_views)| MemberSpec { + index: 0, + n_elements, + n_literals: 1, + n_temps, + n_modules: 0, + // Deliberately NOT pinned to 1. `build_member` bases view `i` on a + // temp only when `i % 2 == 0`, so a range here mixes Temp-based and + // Var-based views -- which is what makes the coupling assertion + // above load-bearing rather than incidentally true. + n_views, + n_gf_tables: 0, + with_loop: false, + }); + prop::collection::vec(member, 2..4).prop_map(|mut specs| { + for (i, spec) in specs.iter_mut().enumerate() { + spec.index = i; + } + specs + }) +} + +proptest! { + #![proptest_config(ProptestConfig::with_cases(128))] + + /// M6 for the interleaving emitter: the combine is a pure REORDERING. + /// + /// Opcode count is conserved (every member's Ret-stripped opcodes appear, + /// plus one terminal `Ret`); each per-element write appears exactly once + /// with its ORIGINAL `SymVarRef`, so `resolve_module` still places it where + /// the acyclic layout would (AC2.3); and the writes come out in exactly + /// `element_order`. + #[test] + fn interleave_conserves_opcodes_and_follows_element_order(specs in member_specs()) { + let members = build_members(&specs); + let order = interleaved_order(&specs); + let scc = scc_of(order.clone()); + let combined = combine_scc_fragment(&scc, &members).map_err(|e| { + TestCaseError::fail(format!("well-formed members must combine: {e}")) + })?; + + let want_len: usize = members + .values() + .map(|f| { + let code = &f.symbolic.code; + if code.last() == Some(&SymbolicOpcode::Ret) { + code.len() - 1 + } else { + code.len() + } + }) + .sum(); + prop_assert_eq!( + combined.symbolic.code.len(), + want_len + 1, + "the interleave must conserve every member opcode and add one Ret" + ); + prop_assert_eq!( + combined.symbolic.code.last(), + Some(&SymbolicOpcode::Ret), + "exactly one terminal Ret" + ); + + let member_names: HashSet = specs + .iter() + .map(|s| member_name(s.index).as_str().to_string()) + .collect(); + let writes: Vec = combined + .symbolic + .code + .iter() + .filter_map(|op| match op { + SymbolicOpcode::AssignCurr { var } + | SymbolicOpcode::AssignConstCurr { var, .. } + | SymbolicOpcode::BinOpAssignCurr { var, .. } + if member_names.contains(var.name.as_str()) => + { + Some(var.clone()) + } + _ => None, + }) + .collect(); + let want_writes: Vec = order + .iter() + .map(|(member, elem)| vref(member, *elem)) + .collect(); + prop_assert_eq!( + &writes, + &want_writes, + "the combined writes must follow element_order, each keeping its \ + original (name, element_offset)" + ); + } +} diff --git a/src/simlin-engine/src/db/combined_fragment_tests.rs b/src/simlin-engine/src/db/combined_fragment_tests.rs index 0f53fc1bb..72a398810 100644 --- a/src/simlin-engine/src/db/combined_fragment_tests.rs +++ b/src/simlin-engine/src/db/combined_fragment_tests.rs @@ -33,7 +33,7 @@ fn id(s: &str) -> Ident { fn vref(name: &str, element_offset: usize) -> SymVarRef { SymVarRef { - name: name.to_string(), + name: crate::common::Ident::new(name), element_offset, } } @@ -518,6 +518,127 @@ fn combined_fragment_trailing_non_write_opcodes_join_last_segment() { ); } +// ── Jump containment ──────────────────────────────────────────────────── +// +// `combine_scc_fragment` REORDERS a member's per-element segments into +// `element_order`, and a jump offset is relative. A jump that left its own +// segment would, after the interleave, land on whatever opcode happened to sit +// that far back -- a silent miscompile with no bad id and no bad reference for +// anything downstream to notice. `segment_member_by_element` rejects it. +// +// Codegen cannot emit that shape today: `BeginIter`/`NextIterOrJump`/`EndIter` +// come from one place (the `Expr::AssignTemp` arm, whose loop writes a TEMP +// through `StoreIterElement`), and a member's per-element `AssignCurr` is +// emitted by the statement-level arm after `EndIter` -- so a loop is always +// wholly inside one segment. That is precisely why it is checked rather than +// asserted in prose: the assumption lives in a different file, and nothing else +// would notice it changing. +// +// The same statement-boundary argument is what keeps the view stack and the +// value stack balanced across a segment cut, so a future emitter that broke +// this would want re-examining on all three counts, not just the jump. + +/// Splice a self-contained iteration loop into `ce`'s element-0 segment: the +/// `BeginIter` and the backward jump that targets inside it both precede the +/// element-0 write, so the whole loop moves as one unit. +fn ce_with_contained_loop() -> HashMap, PerVarBytecodes> { + let mut frags = two_member_fragments(); + let ce = frags.get_mut(&id("ce")).unwrap(); + ce.symbolic.code = vec![ + SymbolicOpcode::BeginIter { + write_temp_id: 0, + has_write_temp: true, + }, + SymbolicOpcode::LoadIterViewAt { offset: 0 }, + SymbolicOpcode::StoreIterElement {}, + SymbolicOpcode::NextIterOrJump { jump_back: -2 }, + SymbolicOpcode::EndIter {}, + SymbolicOpcode::AssignConstCurr { + var: vref("ce", 0), + literal_id: 0, + }, + SymbolicOpcode::LoadVar { + var: vref("ecc", 0), + }, + SymbolicOpcode::AssignCurr { var: vref("ce", 1) }, + SymbolicOpcode::Ret, + ]; + frags +} + +#[test] +fn a_jump_inside_its_own_segment_survives_the_interleave() { + let combined = combine_scc_fragment(&scc(interleaved_order()), &ce_with_contained_loop()) + .expect("a self-contained loop must not block the interleave"); + let jumps = combined + .symbolic + .code + .iter() + .filter(|op| matches!(op, SymbolicOpcode::NextIterOrJump { .. })) + .count(); + assert_eq!(jumps, 1, "the loop must survive the interleave"); + assert_eq!( + write_refs(&combined), + vec![vref("ce", 0), vref("ecc", 0), vref("ce", 1), vref("ecc", 1)], + "and the writes must still follow element_order" + ); +} + +#[test] +fn a_jump_escaping_its_segment_is_a_loud_safe_err() { + let mut frags = two_member_fragments(); + // `ce`'s element-1 segment jumps back across the element-0 write. + let ce = frags.get_mut(&id("ce")).unwrap(); + ce.symbolic.code = vec![ + SymbolicOpcode::LoadVar { + var: vref("ecc", 0), + }, + SymbolicOpcode::AssignCurr { var: vref("ce", 0) }, + SymbolicOpcode::LoadVar { + var: vref("ecc", 0), + }, + // Targets opcode 0, which is in element 0's segment. + SymbolicOpcode::NextIterOrJump { jump_back: -3 }, + SymbolicOpcode::AssignCurr { var: vref("ce", 1) }, + SymbolicOpcode::Ret, + ]; + + let err = combine_scc_fragment(&scc(interleaved_order()), &frags) + .expect_err("a jump crossing a segment boundary must be rejected"); + assert!( + err.contains("outside the per-element segment"), + "expected a loud jump-containment error, got: {err}" + ); +} + +#[test] +fn a_jump_in_the_trailing_tail_targets_its_own_merged_segment() { + // Opcodes after the FINAL per-element write join the last segment rather + // than starting one of their own, so a jump among them is contained when it + // targets inside that segment. The guard's lower bound has to account for + // the tail merging backwards; a naive "segment starts after the last write" + // bound rejects this. + let mut frags = two_member_fragments(); + let ce = frags.get_mut(&id("ce")).unwrap(); + ce.symbolic.code = vec![ + SymbolicOpcode::AssignConstCurr { + var: vref("ce", 0), + literal_id: 0, + }, + SymbolicOpcode::LoadVar { + var: vref("ecc", 0), + }, + SymbolicOpcode::AssignCurr { var: vref("ce", 1) }, + // Tail: joins element 1's segment, and targets opcode 1, inside it. + SymbolicOpcode::PopView {}, + SymbolicOpcode::NextIterOrJump { jump_back: -3 }, + SymbolicOpcode::Ret, + ]; + + combine_scc_fragment(&scc(interleaved_order()), &frags) + .expect("a tail jump into the segment the tail joins is contained"); +} + // ── AC2.3: combined-fragment injection is layout-transparent ──────────── // // End-to-end through `assemble_module` (the Task 6 production consumer). diff --git a/src/simlin-engine/src/db/dep_graph.rs b/src/simlin-engine/src/db/dep_graph.rs index 908d8e6dd..08bbd5e39 100644 --- a/src/simlin-engine/src/db/dep_graph.rs +++ b/src/simlin-engine/src/db/dep_graph.rs @@ -44,8 +44,8 @@ use crate::canonicalize; use crate::common::{Canonical, Ident}; use crate::db::{ CompilationDiagnostic, Db, Diagnostic, DiagnosticError, DiagnosticSeverity, ModuleInputSet, - SourceModel, SourceProject, SourceVariableKind, VariableDeps, model_module_ident_context, - variable_direct_dependencies, + SourceModel, SourceProject, SourceVariable, SourceVariableKind, VariableDeps, + model_module_ident_context, variable_direct_dependencies, }; /// Per-variable dependency facts used to build the model dependency @@ -271,9 +271,11 @@ pub(crate) fn build_var_info( // A `submodel·subvar` dependency whose `subvar` is a Stock is read // from the PRIOR timestep in the dt phase (a stock breaks the // dependency chain), so it must NOT impose a same-step ordering edge. - // This mirrors the legacy `model.rs::module_output_deps` gate + // It used to mirror the same rule in `model.rs::module_output_deps` // (`if ctx.is_initial || !output_var.is_stock()` -- the dt-phase case - // omits the module dependency for a stock output). It applies to EVERY + // omitted the module dependency for a stock output); that second + // dependency walk is gone (GH #568) and this is the only statement of + // the rule left. It applies to EVERY // reader, not just module variables: a NON-module variable that reads // a stock submodel output (e.g. `v = SMOOTH(...)·output`, the SMOOTH // output being an INTEG stock) must likewise drop the dt edge. @@ -777,17 +779,16 @@ enum SccVerdict { /// by the old `members.len() != 1` short-circuit). This builder instead /// consumes each member's *symbolic* `PerVarBytecodes` /// (`var_phase_symbolic_fragment_prod`, the exact production -/// compile+symbolize path -- never a re-derivation), where every variable +/// emission path -- never a re-derivation), where every variable /// reference is a layout-independent `SymVarRef { name, element_offset }`. -/// `SymVarRef.name` is the canonical variable name (the mini-layout keys -/// are `Ident` -- see `layout_from_metadata`), so it is -/// directly comparable to an SCC member's `Ident`. The N=1 and +/// `SymVarRef.name` IS an `Ident` -- codegen copies it straight +/// out of the lowered expression -- so it is directly comparable to an SCC +/// member's `Ident`. The N=1 and /// N>=2 cases are the same builder; N=1 is byte-identical to before (a -/// single member's `AssignCurr(member_base+e, rhs)` symbolizes to one -/// write op with `element_offset == e`, and the reads the prior -/// `collect_read_slots` mapped to `(member, e')` via the mini-`rmap` -/// become exactly the symbolic reads with `name == member, -/// element_offset == e'`; same `element_node_key`, same +/// single member's per-element write emits one write op with +/// `element_offset == e`, and the reads the prior `collect_read_slots` +/// mapped to `(member, e')` are exactly the symbolic reads with +/// `name == member, element_offset == e'`; same `element_node_key`, same /// `scc_components`, same sorted Kahn => same `element_order`). /// /// The edges are the literal current-value data-flow reads of each @@ -815,9 +816,11 @@ enum SccVerdict { /// and does NOT strip INIT-refs (an `INIT(x)` read during the /// initial-value computation is a genuine init-phase ordering edge; /// `init_referenced_vars` feeds the Initials runlist, not a strip). -/// - `LoadVar`/`LoadSubscript`/`PushVarView`/`PushVarViewDirect` and a -/// `Var`-based `PushStaticView`: current-value reads, kept unchanged -- -/// the reads `build_var_info` never strips. +/// - `LoadVar`/`LoadSubscript`/`PushVarViewDirect` and a `Var`-based +/// `PushStaticView`: current-value reads, kept unchanged -- the reads +/// `build_var_info` never strips. (These are ALL the `SymVarRef`-carrying +/// read opcodes the compiler can emit; codegen has no whole-array +/// `PushVarView` form.) /// /// **Why this is the CORRECT relation, not a new over/under-approximation /// (the AC4 soundness argument).** A genuine current-(phase-)timestep @@ -857,8 +860,7 @@ fn symbolic_phase_element_order( // The set of member canonical names, for the "is this read an in-SCC // member?" test. `SymVarRef.name` is the canonical variable name - // (mini-layout keys are `Ident`), so a member's - // `as_str()` compares directly. + // (an `Ident`), so a member's `as_str()` compares directly. let member_names: BTreeSet<&str> = members.iter().map(|m| m.as_str()).collect(); // Build the induced element graph by segmenting each member's @@ -884,7 +886,7 @@ fn symbolic_phase_element_order( // Reads accumulated since the previous per-element write of THIS // member, as (read-name, read-element) pairs. A read is an // in-SCC edge source only if its name is an SCC member. - let mut pending_reads: BTreeSet<(String, usize)> = BTreeSet::new(); + let mut pending_reads: BTreeSet<(Ident, usize)> = BTreeSet::new(); // True once at least one per-element write of this member has // been seen: a malformed fragment with no write for the member // means it is not element-sourceable in the simple per-element @@ -900,7 +902,7 @@ fn symbolic_phase_element_order( SymbolicOpcode::AssignCurr { var } | SymbolicOpcode::AssignConstCurr { var, .. } | SymbolicOpcode::BinOpAssignCurr { var, .. } - if var.name == member_name => + if var.name.as_str() == member_name => { saw_write = true; let node = element_node_key(member_name, var.element_offset); @@ -910,7 +912,7 @@ fn symbolic_phase_element_order( let mut preds: BTreeSet> = BTreeSet::new(); for (rname, relem) in &pending_reads { if member_names.contains(rname.as_str()) { - preds.insert(element_node_key(rname, *relem)); + preds.insert(element_node_key(rname.as_str(), *relem)); } } for pred in preds { @@ -932,7 +934,6 @@ fn symbolic_phase_element_order( // (`build_var_info` never strips a current-value dep). SymbolicOpcode::LoadVar { var } | SymbolicOpcode::LoadSubscript { var } - | SymbolicOpcode::PushVarView { var, .. } | SymbolicOpcode::PushVarViewDirect { var, .. } => { pending_reads.insert((var.name.clone(), var.element_offset)); } @@ -987,8 +988,8 @@ fn symbolic_phase_element_order( } } } - // Other write targets (a different member, or `AssignNext` - // / `BinOpAssignNext` -- a stock-update, not a per-element + // Other write targets (a different member, or + // `BinOpAssignNext` -- a stock-update, not a per-element // current-value write of THIS member) do not terminate // this member's element segment and carry no read; ignore. _ => {} @@ -1101,7 +1102,7 @@ fn symbolic_phase_element_order( /// recurrence has an init self-loop with **no corresponding dt cycle**. /// Here only the **init** induced element graph is relevant: the dt /// precondition the `Dt` branch applies would be *wrong* (a stock has no -/// dt element graph -- its dt lowering is `AssignNext`, not the +/// dt element graph -- its dt lowering is `BinOpAssignNext`, not the /// per-element `AssignCurr` the element graph reads -- so requiring dt /// element-acyclicity would spuriously reject every init-only /// recurrence). The init verdict therefore verifies `SccPhase::Initial` @@ -1275,12 +1276,11 @@ pub(crate) struct DtSccResolution { /// fragment injection (Task 6, `assemble_module` -> /// `var_phase_symbolic_fragment_prod`) deliberately consumes the *same* /// no-input wiring: the symbolic per-member fragments are lowered with -/// `lower_var_fragment(.., &[], ..)` / `inputs = BTreeSet::new()` / -/// `build_caller_module_refs(.., &[])`, matching this SCC identification's -/// `build_var_info(.., &[])`, so the verdict's `element_order` and the -/// combined fragment's per-element segmentation agree by construction. The -/// real `module_input_names` are intentionally NOT plumbed into either -/// side, because the with-inputs `compute_transitive` re-run is the +/// `lower_var_fragment(.., &[], ..)` / `inputs = BTreeSet::new()`, matching +/// this SCC identification's `build_var_info(.., &[])`, so the verdict's +/// `element_order` and the combined fragment's per-element segmentation +/// agree by construction. The real `module_input_names` are intentionally NOT +/// plumbed into either side, because the with-inputs `compute_transitive` re-run is the /// soundness backstop for the multi-member (N>=2) SCCs Subcomponent B /// resolves *exactly as it is for the N=1 single-variable self-recurrence /// case*: that re-run's `.unwrap_or_else` clears `resolved_sccs` and sets @@ -1681,6 +1681,45 @@ pub fn model_dependency_graph<'db>( model_dependency_graph_impl(db, model, project, module_input_names) } +/// Which of the three phase runlists one variable appears in. +#[derive(Clone, Copy, Debug, PartialEq, Eq, salsa::Update)] +pub struct RunlistMembership { + pub initials: bool, + pub flows: bool, + pub stocks: bool, +} + +/// Project `model_dependency_graph` down to the three runlist-membership bits +/// a per-variable fragment compiler actually reads -- a salsa *firewall*. +/// +/// `compile_var_fragment` needs only "is this variable in the initials / +/// flows / stocks runlist"; reading the whole `ModelDepGraphResult` for that +/// made every fragment in a model depend on every other variable's +/// dependencies, so any dep-graph change re-executed all of them. This +/// projection re-executes on the same changes but returns three booleans, so +/// salsa backdates it whenever the variable's own membership is unchanged -- +/// which is the case for every variable a layout-only edit does not touch. +/// +/// Keyed on the `SourceVariable` handle rather than a name so it is invalidated +/// by a rename of *this* variable through the same field read every other +/// per-variable query uses. +#[salsa::tracked] +pub fn var_runlist_membership<'db>( + db: &'db dyn Db, + var: SourceVariable, + model: SourceModel, + project: SourceProject, + module_inputs: ModuleInputSet<'db>, +) -> RunlistMembership { + let dep_graph = model_dependency_graph(db, model, project, module_inputs); + let name = canonicalize(var.ident(db)).into_owned(); + RunlistMembership { + initials: dep_graph.runlist_initials.contains(&name), + flows: dep_graph.runlist_flows.contains(&name), + stocks: dep_graph.runlist_stocks.contains(&name), + } +} + // ── Model dependency graph (the cycle gate) ──────────────────────────── // // `model_dependency_graph_impl` is the production consumer of this diff --git a/src/simlin-engine/src/db/dep_graph_tests.rs b/src/simlin-engine/src/db/dep_graph_tests.rs index bcd37ed7b..974645589 100644 --- a/src/simlin-engine/src/db/dep_graph_tests.rs +++ b/src/simlin-engine/src/db/dep_graph_tests.rs @@ -2333,7 +2333,7 @@ fn synthetic_helper_symbolic_fragment_is_parent_sourced() { SymbolicOpcode::AssignCurr { var } | SymbolicOpcode::AssignConstCurr { var, .. } | SymbolicOpcode::BinOpAssignCurr { var, .. } - if var.name == helper.as_str() + if var.name.as_str() == helper.as_str() )), "the parent-sourced helper fragment must write the helper's own \ per-element value (so symbolic_phase_element_order can define its \ diff --git a/src/simlin-engine/src/db/diagnostic_tests.rs b/src/simlin-engine/src/db/diagnostic_tests.rs index 3d0c097ce..3619390ce 100644 --- a/src/simlin-engine/src/db/diagnostic_tests.rs +++ b/src/simlin-engine/src/db/diagnostic_tests.rs @@ -2926,3 +2926,112 @@ fn unit_definition_errors_survive_an_unrelated_input_change() { "the full diagnostic set must be identical across an unrelated input change" ); } + +/// `Variable::errors` and `Variable::unit_errors` are the CHANNEL by which +/// parsing and lowering report a failure to the salsa path, not residue from +/// the monolithic compiler. +/// +/// `docs/tech-debt.md` item 17 claimed all four embedded error fields were +/// "dead weight carried through the monolithic compilation path", redundant +/// with the salsa pipeline. For these two that is backwards: the salsa +/// pipeline's diagnostics are DOWNSTREAM of them -- +/// `db::var_fragment::lower_var_fragment` reads +/// `parsed.variable.unit_errors()`, `parsed.variable.equation_errors()` and +/// `lowered.equation_errors()` and turns each entry into a `Diagnostic`. Acting +/// on the claim would silently drop those diagnostics, so it is pinned here +/// rather than left as prose: each half asserts BOTH that the stage's value +/// carries the error in the field AND that the matching diagnostic comes out of +/// `collect_all_diagnostics`. +/// +/// Emptying the `unit_errors()` read or the `lowered.equation_errors()` read +/// reds THIS test. Emptying the `parsed.variable.equation_errors()` read does +/// NOT -- `lower_variable` clones parse errors forward in all three arms, so the +/// lowered read catches the same error and this test stays green. What that read +/// uniquely carries is the conveyor/queue driven-flow `EmptyEquation` +/// suppression, and dropping it reds +/// `test_conveyor_driven_flow_empty_equation_suppressed`, +/// `test_conveyor_marker_removal_reinstates_empty_equation` and +/// `test_queue_driven_outflow_empty_equation_suppressed` instead. Measured, not +/// assumed. +#[test] +fn variable_error_fields_are_the_lowering_channel() { + use crate::test_common::TestProject; + + // ── parse-time: a malformed `` string, and a syntax error ── + let dm = TestProject::new("parse_channel") + .aux("bad_unit_var", "1", Some("bad units here!!!")) + .aux("bad_eqn_var", "1 +", None) + .build_datamodel(); + let db = SimlinDb::default(); + let sync = sync_from_datamodel(&db, &dm); + let stage0 = crate::db::model_stage0(&db, sync.models["main"].source, sync.project); + + assert!( + stage0.variables[&crate::common::Ident::new("bad_unit_var")] + .unit_errors() + .is_some(), + "parsing must record the malformed unit string on the variable" + ); + assert!( + stage0.variables[&crate::common::Ident::new("bad_eqn_var")] + .equation_errors() + .is_some(), + "parsing must record the equation syntax error on the variable" + ); + + let diags = collect_all_diagnostics(&db, sync.project); + assert!( + diags.iter().any(|d| { + d.variable.as_deref() == Some("bad_unit_var") + && matches!(&d.error, DiagnosticError::Unit(_)) + }), + "the recorded unit error must reach collect_all_diagnostics; got: {diags:?}" + ); + assert!( + diags.iter().any(|d| { + d.variable.as_deref() == Some("bad_eqn_var") + && matches!(&d.error, DiagnosticError::Equation(_)) + }), + "the recorded equation error must reach collect_all_diagnostics; got: {diags:?}" + ); + + // ── lowering-time: an error `lower_ast` raises, which the parsed + // variable cannot carry because it does not exist yet ── + let dm = TestProject::new("lower_channel") + .named_dimension("Cities", &["Boston", "Seattle"]) + .named_dimension("Products", &["Widgets", "Gadgets"]) + .array_aux("sales[Cities]", "1") + .array_aux("prices[Products]", "1") + .array_aux("bad[Cities]", "sales + prices") + .build_datamodel(); + let db = SimlinDb::default(); + let sync = sync_from_datamodel(&db, &dm); + let bad = crate::common::Ident::new("bad"); + + assert!( + crate::db::model_stage0(&db, sync.models["main"].source, sync.project).variables[&bad] + .equation_errors() + .is_none(), + "the fixture must isolate a LOWERING error: parsing sees nothing wrong" + ); + let lowered_errors = crate::db::model_stage1(&db, sync.models["main"].source, sync.project) + .variables[&bad] + .equation_errors() + .expect("lowering must record the dimension mismatch on the variable"); + assert!( + lowered_errors + .iter() + .any(|e| e.code == crate::common::ErrorCode::MismatchedDimensions), + "expected MismatchedDimensions, got: {lowered_errors:?}" + ); + + let diags = collect_all_diagnostics(&db, sync.project); + assert!( + diags.iter().any(|d| { + d.variable.as_deref() == Some("bad") + && matches!(&d.error, DiagnosticError::Equation(e) + if e.code == crate::common::ErrorCode::MismatchedDimensions) + }), + "the recorded lowering error must reach collect_all_diagnostics; got: {diags:?}" + ); +} diff --git a/src/simlin-engine/src/db/fragment_char_golden/array_ops.txt b/src/simlin-engine/src/db/fragment_char_golden/array_ops.txt new file mode 100644 index 000000000..b7f2a2124 --- /dev/null +++ b/src/simlin-engine/src/db/fragment_char_golden/array_ops.txt @@ -0,0 +1,165 @@ +########## model main (module inputs: []) ########## +== layout == + n_slots: 16 + arr: [0, 3) + combo: [3, 6) + order: [6, 9) + ranked: [9, 12) + sorted: [12, 15) + total: [15, 16) +== main::arr [explicit] : flow == + initial: + flow: + literals: [30.0, 10.0, 20.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 AssignConstCurr arr@0 #0 (=30.0) + 0001 AssignConstCurr arr@1 #1 (=10.0) + 0002 AssignConstCurr arr@2 #2 (=20.0) + 0003 Ret + stock: +== main::combo [explicit] : flow == + initial: + flow: + literals: [1.0] + temp_sizes: [temp0:3, temp1:3] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: + view[0]: base=arr@0 dims=[3] strides=[1] offset=0 dim_ids=[0] sparse=[] + view[1]: base=arr@0 dims=[3] strides=[1] offset=0 dim_ids=[0] sparse=[] + code: + 0000 PushStaticView view=0 + 0001 LoadConstant #0 (=1.0) + 0002 VectorSortOrder write_temp=0 + 0003 PopView + 0004 PushStaticView view=1 + 0005 LoadConstant #0 (=1.0) + 0006 Rank write_temp=1 + 0007 PopView + 0008 LoadTempConst temp=0 index=0 + 0009 LoadTempConst temp=1 index=0 + 0010 BinOpAssignCurr Add combo@0 + 0011 LoadTempConst temp=0 index=1 + 0012 LoadTempConst temp=1 index=1 + 0013 BinOpAssignCurr Add combo@1 + 0014 LoadTempConst temp=0 index=2 + 0015 LoadTempConst temp=1 index=2 + 0016 BinOpAssignCurr Add combo@2 + 0017 Ret + stock: +== main::order [explicit] : flow == + initial: + flow: + literals: [1.0] + temp_sizes: [temp0:3] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: + view[0]: base=arr@0 dims=[3] strides=[1] offset=0 dim_ids=[0] sparse=[] + code: + 0000 PushStaticView view=0 + 0001 LoadConstant #0 (=1.0) + 0002 VectorSortOrder write_temp=0 + 0003 PopView + 0004 LoadTempConst temp=0 index=0 + 0005 AssignCurr order@0 + 0006 LoadTempConst temp=0 index=1 + 0007 AssignCurr order@1 + 0008 LoadTempConst temp=0 index=2 + 0009 AssignCurr order@2 + 0010 Ret + stock: +== main::ranked [explicit] : flow == + initial: + flow: + literals: [1.0] + temp_sizes: [temp0:3] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: + view[0]: base=arr@0 dims=[3] strides=[1] offset=0 dim_ids=[0] sparse=[] + code: + 0000 PushStaticView view=0 + 0001 LoadConstant #0 (=1.0) + 0002 Rank write_temp=0 + 0003 PopView + 0004 LoadTempConst temp=0 index=0 + 0005 AssignCurr ranked@0 + 0006 LoadTempConst temp=0 index=1 + 0007 AssignCurr ranked@1 + 0008 LoadTempConst temp=0 index=2 + 0009 AssignCurr ranked@2 + 0010 Ret + stock: +== main::sorted [explicit] : flow == + initial: + flow: + literals: [] + temp_sizes: [temp0:3] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: + view[0]: base=arr@0 dims=[] strides=[] offset=0 dim_ids=[] sparse=[] + view[1]: base=order@0 dims=[3] strides=[1] offset=0 dim_ids=[0] sparse=[] + code: + 0000 PushStaticView view=0 + 0001 PushStaticView view=1 + 0002 VectorElmMap write_temp=0 full_source_len=3 + 0003 PopView + 0004 PopView + 0005 LoadTempConst temp=0 index=0 + 0006 AssignCurr sorted@0 + 0007 LoadTempConst temp=0 index=1 + 0008 AssignCurr sorted@1 + 0009 LoadTempConst temp=0 index=2 + 0010 AssignCurr sorted@2 + 0011 Ret + stock: +== main::total [explicit] : flow == + initial: + flow: + literals: [] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: + view[0]: base=arr@0 dims=[3] strides=[1] offset=0 dim_ids=[0] sparse=[] + code: + 0000 PushStaticView view=0 + 0001 ArraySum + 0002 PopView + 0003 AssignCurr total@0 + 0004 Ret + stock: +########## runtime ########## + step 0: + arr[east] = 30.0 + arr[north] = 20.0 + arr[west] = 10.0 + combo[east] = 4.0 + combo[north] = 2.0 + combo[west] = 3.0 + dt = 1.0 + final_time = 1.0 + initial_time = 0.0 + order[east] = 1.0 + order[north] = 0.0 + order[west] = 2.0 + ranked[east] = 3.0 + ranked[north] = 2.0 + ranked[west] = 1.0 + sorted[east] = 10.0 + sorted[north] = 30.0 + sorted[west] = 20.0 + time = 0.0 + total = 60.0 diff --git a/src/simlin-engine/src/db/fragment_char_golden/arrayed_shapes.txt b/src/simlin-engine/src/db/fragment_char_golden/arrayed_shapes.txt new file mode 100644 index 000000000..957bd0650 --- /dev/null +++ b/src/simlin-engine/src/db/fragment_char_golden/arrayed_shapes.txt @@ -0,0 +1,64 @@ +########## model main (module inputs: []) ########## +== layout == + n_slots: 6 + base: [0, 2) + exceptdef: [2, 4) + perelem: [4, 6) +== main::base [explicit] : flow == + initial: + flow: + literals: [10.0, 10.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 AssignConstCurr base@0 #0 (=10.0) + 0001 AssignConstCurr base@1 #1 (=10.0) + 0002 Ret + stock: +== main::exceptdef [explicit] : flow == + initial: + flow: + literals: [7.0, 1.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 AssignConstCurr exceptdef@0 #0 (=7.0) + 0001 AssignConstCurr exceptdef@1 #1 (=1.0) + 0002 Ret + stock: +== main::perelem [explicit] : flow == + initial: + flow: + literals: [2.0, 5.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadVar base@0 + 0001 LoadConstant #0 (=2.0) + 0002 BinOpAssignCurr Mul perelem@0 + 0003 LoadVar base@1 + 0004 LoadConstant #1 (=5.0) + 0005 BinOpAssignCurr Add perelem@1 + 0006 Ret + stock: +########## runtime ########## + step 0: + base[east] = 10.0 + base[west] = 10.0 + dt = 1.0 + exceptdef[east] = 7.0 + exceptdef[west] = 1.0 + final_time = 1.0 + initial_time = 0.0 + perelem[east] = 20.0 + perelem[west] = 15.0 + time = 0.0 diff --git a/src/simlin-engine/src/db/fragment_char_golden/dynamic_view.txt b/src/simlin-engine/src/db/fragment_char_golden/dynamic_view.txt new file mode 100644 index 000000000..ce12e5b5e --- /dev/null +++ b/src/simlin-engine/src/db/fragment_char_golden/dynamic_view.txt @@ -0,0 +1,185 @@ +########## model main (module inputs: []) ########## +== layout == + n_slots: 16 + acc: [0, 3) + arr: [3, 6) + grow: [6, 9) + idx: [9, 10) + mapped: [10, 13) + off1: [13, 16) +== main::acc [explicit] : initial+stock == + initial: + literals: [0.0, 0.0, 0.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 AssignConstCurr acc@0 #0 (=0.0) + 0001 AssignConstCurr acc@1 #1 (=0.0) + 0002 AssignConstCurr acc@2 #2 (=0.0) + 0003 Ret + flow: + stock: + literals: [0.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadVar acc@0 + 0001 LoadVar grow@0 + 0002 LoadConstant #0 (=0.0) + 0003 Op2 Sub + 0004 LoadGlobalVar off=1 (dt) + 0005 Op2 Mul + 0006 BinOpAssignNext Add acc@0 + 0007 LoadVar acc@1 + 0008 LoadVar grow@1 + 0009 LoadConstant #0 (=0.0) + 0010 Op2 Sub + 0011 LoadGlobalVar off=1 (dt) + 0012 Op2 Mul + 0013 BinOpAssignNext Add acc@1 + 0014 LoadVar acc@2 + 0015 LoadVar grow@2 + 0016 LoadConstant #0 (=0.0) + 0017 Op2 Sub + 0018 LoadGlobalVar off=1 (dt) + 0019 Op2 Mul + 0020 BinOpAssignNext Add acc@2 + 0021 Ret +== main::arr [explicit] : flow == + initial: + flow: + literals: [1.0, 2.0, 3.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 AssignConstCurr arr@0 #0 (=1.0) + 0001 AssignConstCurr arr@1 #1 (=2.0) + 0002 AssignConstCurr arr@2 #2 (=3.0) + 0003 Ret + stock: +== main::grow [explicit] : flow == + initial: + flow: + literals: [] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadVar arr@0 + 0001 AssignCurr grow@0 + 0002 LoadVar arr@1 + 0003 AssignCurr grow@1 + 0004 LoadVar arr@2 + 0005 AssignCurr grow@2 + 0006 Ret + stock: +== main::idx [explicit] : flow == + initial: + flow: + literals: [1.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadConstant #0 (=1.0) + 0001 LoadGlobalVar off=0 (time) + 0002 BinOpAssignCurr Add idx@0 + 0003 Ret + stock: +== main::mapped [explicit] : flow == + initial: + flow: + literals: [] + temp_sizes: [temp0:3] + dim_lists: [[3]] + graphical_functions: [] + module_decls: [] + static_views: + view[0]: base=off1@0 dims=[3] strides=[1] offset=0 dim_ids=[0] sparse=[] + code: + 0000 PushVarViewDirect arr@0 dim_list=0 + 0001 LoadVar idx@0 + 0002 ViewSubscriptDynamic dim=0 + 0003 PushStaticView view=0 + 0004 VectorElmMap write_temp=0 full_source_len=3 + 0005 PopView + 0006 PopView + 0007 LoadTempConst temp=0 index=0 + 0008 AssignCurr mapped@0 + 0009 LoadTempConst temp=0 index=1 + 0010 AssignCurr mapped@1 + 0011 LoadTempConst temp=0 index=2 + 0012 AssignCurr mapped@2 + 0013 Ret + stock: +== main::off1 [explicit] : flow == + initial: + flow: + literals: [0.0, 0.0, 0.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 AssignConstCurr off1@0 #0 (=0.0) + 0001 AssignConstCurr off1@1 #1 (=0.0) + 0002 AssignConstCurr off1@2 #2 (=0.0) + 0003 Ret + stock: +########## runtime ########## + step 0: + acc[east] = 0.0 + acc[north] = 0.0 + acc[west] = 0.0 + arr[east] = 1.0 + arr[north] = 3.0 + arr[west] = 2.0 + dt = 1.0 + final_time = 1.0 + grow[east] = 1.0 + grow[north] = 3.0 + grow[west] = 2.0 + idx = 1.0 + initial_time = 0.0 + mapped[east] = 1.0 + mapped[north] = 1.0 + mapped[west] = 1.0 + off1[east] = 0.0 + off1[north] = 0.0 + off1[west] = 0.0 + time = 0.0 + step 1: + acc[east] = 1.0 + acc[north] = 3.0 + acc[west] = 2.0 + arr[east] = 1.0 + arr[north] = 3.0 + arr[west] = 2.0 + dt = 1.0 + final_time = 1.0 + grow[east] = 1.0 + grow[north] = 3.0 + grow[west] = 2.0 + idx = 2.0 + initial_time = 0.0 + mapped[east] = 2.0 + mapped[north] = 2.0 + mapped[west] = 2.0 + off1[east] = 0.0 + off1[north] = 0.0 + off1[west] = 0.0 + time = 1.0 diff --git a/src/simlin-engine/src/db/fragment_char_golden/graphical_functions.txt b/src/simlin-engine/src/db/fragment_char_golden/graphical_functions.txt new file mode 100644 index 000000000..0d02f76e8 --- /dev/null +++ b/src/simlin-engine/src/db/fragment_char_golden/graphical_functions.txt @@ -0,0 +1,132 @@ +########## model main (module inputs: []) ########## +== layout == + n_slots: 7 + curve: [0, 1) + drive: [1, 2) + g: [2, 4) + gtotal: [4, 5) + out: [5, 7) +== main::curve [explicit] : flow == + initial: + flow: + literals: [0.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: + gf[0]: (0.0, 5.0) (1.0, 15.0) + module_decls: [] + static_views: [] + code: + 0000 LoadConstant #0 (=0.0) + 0001 LoadVar drive@0 + 0002 Lookup base_gf=0 table_count=1 mode=Interpolate + 0003 AssignCurr curve@0 + 0004 Ret + stock: +== main::drive [explicit] : flow == + initial: + flow: + literals: [] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadGlobalVar off=0 (time) + 0001 AssignCurr drive@0 + 0002 Ret + stock: +== main::g [explicit] : flow == + initial: + flow: + literals: [0.0, 1.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: + gf[0]: (0.0, 1.0) (1.0, 2.0) + gf[1]: (0.0, 2.0) (1.0, 4.0) + module_decls: [] + static_views: [] + code: + 0000 LoadConstant #0 (=0.0) + 0001 LoadGlobalVar off=0 (time) + 0002 Lookup base_gf=0 table_count=2 mode=Interpolate + 0003 AssignCurr g@0 + 0004 LoadConstant #1 (=1.0) + 0005 LoadGlobalVar off=0 (time) + 0006 Lookup base_gf=0 table_count=2 mode=Interpolate + 0007 AssignCurr g@1 + 0008 Ret + stock: +== main::gtotal [explicit] : flow == + initial: + flow: + literals: [] + temp_sizes: [temp0:2] + dim_lists: [] + graphical_functions: + gf[0]: (0.0, 1.0) (1.0, 2.0) + gf[1]: (0.0, 2.0) (1.0, 4.0) + module_decls: [] + static_views: + view[0]: base=g@0 dims=[2] strides=[1] offset=0 dim_ids=[0] sparse=[] + view[1]: base=temp0 dims=[2] strides=[1] offset=0 dim_ids=[0] sparse=[] + code: + 0000 PushStaticView view=0 + 0001 LoadGlobalVar off=0 (time) + 0002 LookupArray base_gf=0 table_count=2 mode=Interpolate write_temp=0 + 0003 PopView + 0004 PushStaticView view=1 + 0005 ArraySum + 0006 PopView + 0007 AssignCurr gtotal@0 + 0008 Ret + stock: +== main::out [explicit] : flow == + initial: + flow: + literals: [0.0, 1.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: + gf[0]: (0.0, 1.0) (1.0, 2.0) + gf[1]: (0.0, 2.0) (1.0, 4.0) + module_decls: [] + static_views: [] + code: + 0000 LoadConstant #0 (=0.0) + 0001 LoadGlobalVar off=0 (time) + 0002 Lookup base_gf=0 table_count=2 mode=Interpolate + 0003 AssignCurr out@0 + 0004 LoadConstant #1 (=1.0) + 0005 LoadGlobalVar off=0 (time) + 0006 Lookup base_gf=0 table_count=2 mode=Interpolate + 0007 AssignCurr out@1 + 0008 Ret + stock: +########## runtime ########## + step 0: + curve = 5.0 + drive = 0.0 + dt = 1.0 + final_time = 1.0 + g[east] = 1.0 + g[west] = 2.0 + gtotal = 3.0 + initial_time = 0.0 + out[east] = 1.0 + out[west] = 2.0 + time = 0.0 + step 1: + curve = 15.0 + drive = 1.0 + dt = 1.0 + final_time = 1.0 + g[east] = 2.0 + g[west] = 4.0 + gtotal = 6.0 + initial_time = 0.0 + out[east] = 2.0 + out[west] = 4.0 + time = 1.0 diff --git a/src/simlin-engine/src/db/fragment_char_golden/lookup_only_table.txt b/src/simlin-engine/src/db/fragment_char_golden/lookup_only_table.txt new file mode 100644 index 000000000..7d4f15634 --- /dev/null +++ b/src/simlin-engine/src/db/fragment_char_golden/lookup_only_table.txt @@ -0,0 +1,59 @@ +########## model main (module inputs: []) ########## +== layout == + n_slots: 3 + at_half: [0, 1) + at_time: [1, 2) + table: [2, 3) +== main::at_half [explicit] : flow == + initial: + flow: + literals: [0.0, 0.5] + temp_sizes: [] + dim_lists: [] + graphical_functions: + gf[0]: (0.0, 3.0) (1.0, 7.0) + module_decls: [] + static_views: [] + code: + 0000 LoadConstant #0 (=0.0) + 0001 LoadConstant #1 (=0.5) + 0002 Lookup base_gf=0 table_count=1 mode=Interpolate + 0003 AssignCurr at_half@0 + 0004 Ret + stock: +== main::at_time [explicit] : flow == + initial: + flow: + literals: [0.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: + gf[0]: (0.0, 3.0) (1.0, 7.0) + module_decls: [] + static_views: [] + code: + 0000 LoadConstant #0 (=0.0) + 0001 LoadGlobalVar off=0 (time) + 0002 Lookup base_gf=0 table_count=1 mode=Interpolate + 0003 AssignCurr at_time@0 + 0004 Ret + stock: +== main::table [explicit] : none == + initial: + flow: + stock: +########## runtime ########## + step 0: + at_half = 5.0 + at_time = 3.0 + dt = 1.0 + final_time = 1.0 + initial_time = 0.0 + time = 0.0 + step 1: + at_half = 5.0 + at_time = 7.0 + dt = 1.0 + final_time = 1.0 + initial_time = 0.0 + time = 1.0 diff --git a/src/simlin-engine/src/db/fragment_char_golden/ltm_loop_discovery.txt b/src/simlin-engine/src/db/fragment_char_golden/ltm_loop_discovery.txt new file mode 100644 index 000000000..5198f08a3 --- /dev/null +++ b/src/simlin-engine/src/db/fragment_char_golden/ltm_loop_discovery.txt @@ -0,0 +1,351 @@ +########## model main (module inputs: []) ########## +== layout == + n_slots: 9 + $⁚$⁚ltm⁚link_score⁚growth→level⁚0⁚arg0: [6, 7) + $⁚$⁚ltm⁚link_score⁚growth→level⁚1⁚arg0: [7, 8) + $⁚$⁚ltm⁚link_score⁚growth→level⁚2⁚arg0: [8, 9) + $⁚ltm⁚link_score⁚growth→level: [3, 4) + $⁚ltm⁚link_score⁚level→growth: [4, 5) + $⁚ltm⁚link_score⁚rate→growth: [5, 6) + growth: [0, 1) + level: [1, 2) + rate: [2, 3) +== main::$⁚$⁚ltm⁚link_score⁚growth→level⁚0⁚arg0 [ltm-implicit-helper] : initial+flow == + initial: + literals: [] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadGlobalVar off=0 (time) + 0001 AssignCurr $⁚$⁚ltm⁚link_score⁚growth→level⁚0⁚arg0@0 + 0002 Ret + flow: + literals: [] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadGlobalVar off=0 (time) + 0001 AssignCurr $⁚$⁚ltm⁚link_score⁚growth→level⁚0⁚arg0@0 + 0002 Ret + stock: +== main::$⁚$⁚ltm⁚link_score⁚growth→level⁚1⁚arg0 [ltm-implicit-helper] : initial+flow == + initial: + literals: [0.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadConstant #0 (=0.0) + 0001 LoadPrev growth@0 + 0002 AssignCurr $⁚$⁚ltm⁚link_score⁚growth→level⁚1⁚arg0@0 + 0003 Ret + flow: + literals: [0.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadConstant #0 (=0.0) + 0001 LoadPrev growth@0 + 0002 AssignCurr $⁚$⁚ltm⁚link_score⁚growth→level⁚1⁚arg0@0 + 0003 Ret + stock: +== main::$⁚$⁚ltm⁚link_score⁚growth→level⁚2⁚arg0 [ltm-implicit-helper] : initial+flow == + initial: + literals: [0.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadConstant #0 (=0.0) + 0001 LoadPrev level@0 + 0002 AssignCurr $⁚$⁚ltm⁚link_score⁚growth→level⁚2⁚arg0@0 + 0003 Ret + flow: + literals: [0.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadConstant #0 (=0.0) + 0001 LoadPrev level@0 + 0002 AssignCurr $⁚$⁚ltm⁚link_score⁚growth→level⁚2⁚arg0@0 + 0003 Ret + stock: +== main::$⁚ltm⁚link_score⁚growth→level [ltm-synthetic] : flow == + initial: + flow: + literals: [0.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadConstant #0 (=0.0) + 0001 LoadGlobalVar off=1 (dt) + 0002 LoadConstant #0 (=0.0) + 0003 LoadPrev growth@0 + 0004 LoadConstant #0 (=0.0) + 0005 LoadPrev $⁚$⁚ltm⁚link_score⁚growth→level⁚1⁚arg0@0 + 0006 Op2 Sub + 0007 Op2 Mul + 0008 LoadVar level@0 + 0009 LoadConstant #0 (=0.0) + 0010 LoadPrev level@0 + 0011 Op2 Sub + 0012 LoadConstant #0 (=0.0) + 0013 LoadPrev level@0 + 0014 LoadConstant #0 (=0.0) + 0015 LoadPrev $⁚$⁚ltm⁚link_score⁚growth→level⁚2⁚arg0@0 + 0016 Op2 Sub + 0017 Op2 Sub + 0018 LoadConstant #0 (=0.0) + 0019 Apply SafeDiv + 0020 LoadConstant #0 (=0.0) + 0021 LoadConstant #0 (=0.0) + 0022 Apply Abs + 0023 LoadGlobalVar off=0 (time) + 0024 LoadGlobalVar off=2 (initial_time) + 0025 Op2 Eq + 0026 LoadGlobalVar off=2 (initial_time) + 0027 LoadPrev $⁚$⁚ltm⁚link_score⁚growth→level⁚0⁚arg0@0 + 0028 LoadGlobalVar off=2 (initial_time) + 0029 Op2 Eq + 0030 Op2 Or + 0031 SetCond + 0032 If + 0033 AssignCurr $⁚ltm⁚link_score⁚growth→level@0 + 0034 Ret + stock: +== main::$⁚ltm⁚link_score⁚level→growth [ltm-synthetic] : flow == + initial: + flow: + literals: [0.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadConstant #0 (=0.0) + 0001 LoadConstant #0 (=0.0) + 0002 LoadVar level@0 + 0003 LoadConstant #0 (=0.0) + 0004 LoadPrev rate@0 + 0005 Op2 Mul + 0006 LoadConstant #0 (=0.0) + 0007 LoadPrev growth@0 + 0008 Op2 Sub + 0009 LoadVar growth@0 + 0010 LoadConstant #0 (=0.0) + 0011 LoadPrev growth@0 + 0012 Op2 Sub + 0013 LoadConstant #0 (=0.0) + 0014 LoadConstant #0 (=0.0) + 0015 Apply Abs + 0016 LoadConstant #0 (=0.0) + 0017 Apply SafeDiv + 0018 LoadVar level@0 + 0019 LoadConstant #0 (=0.0) + 0020 LoadPrev level@0 + 0021 Op2 Sub + 0022 LoadConstant #0 (=0.0) + 0023 LoadConstant #0 (=0.0) + 0024 Apply Sign + 0025 Op2 Mul + 0026 LoadVar growth@0 + 0027 LoadConstant #0 (=0.0) + 0028 LoadPrev growth@0 + 0029 Op2 Sub + 0030 LoadConstant #0 (=0.0) + 0031 Op2 Eq + 0032 LoadVar level@0 + 0033 LoadConstant #0 (=0.0) + 0034 LoadPrev level@0 + 0035 Op2 Sub + 0036 LoadConstant #0 (=0.0) + 0037 Op2 Eq + 0038 Op2 Or + 0039 SetCond + 0040 If + 0041 LoadGlobalVar off=0 (time) + 0042 LoadGlobalVar off=2 (initial_time) + 0043 Op2 Eq + 0044 SetCond + 0045 If + 0046 AssignCurr $⁚ltm⁚link_score⁚level→growth@0 + 0047 Ret + stock: +== main::$⁚ltm⁚link_score⁚rate→growth [ltm-synthetic] : flow == + initial: + flow: + literals: [0.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadConstant #0 (=0.0) + 0001 LoadConstant #0 (=0.0) + 0002 LoadConstant #0 (=0.0) + 0003 LoadPrev level@0 + 0004 LoadVar rate@0 + 0005 Op2 Mul + 0006 LoadConstant #0 (=0.0) + 0007 LoadPrev growth@0 + 0008 Op2 Sub + 0009 LoadVar growth@0 + 0010 LoadConstant #0 (=0.0) + 0011 LoadPrev growth@0 + 0012 Op2 Sub + 0013 LoadConstant #0 (=0.0) + 0014 LoadConstant #0 (=0.0) + 0015 Apply Abs + 0016 LoadConstant #0 (=0.0) + 0017 Apply SafeDiv + 0018 LoadVar rate@0 + 0019 LoadConstant #0 (=0.0) + 0020 LoadPrev rate@0 + 0021 Op2 Sub + 0022 LoadConstant #0 (=0.0) + 0023 LoadConstant #0 (=0.0) + 0024 Apply Sign + 0025 Op2 Mul + 0026 LoadVar growth@0 + 0027 LoadConstant #0 (=0.0) + 0028 LoadPrev growth@0 + 0029 Op2 Sub + 0030 LoadConstant #0 (=0.0) + 0031 Op2 Eq + 0032 LoadVar rate@0 + 0033 LoadConstant #0 (=0.0) + 0034 LoadPrev rate@0 + 0035 Op2 Sub + 0036 LoadConstant #0 (=0.0) + 0037 Op2 Eq + 0038 Op2 Or + 0039 SetCond + 0040 If + 0041 LoadGlobalVar off=0 (time) + 0042 LoadGlobalVar off=2 (initial_time) + 0043 Op2 Eq + 0044 SetCond + 0045 If + 0046 AssignCurr $⁚ltm⁚link_score⁚rate→growth@0 + 0047 Ret + stock: +== main::growth [explicit] : flow == + initial: + flow: + literals: [] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadVar level@0 + 0001 LoadVar rate@0 + 0002 BinOpAssignCurr Mul growth@0 + 0003 Ret + stock: +== main::level [explicit] : initial+stock == + initial: + literals: [10.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 AssignConstCurr level@0 #0 (=10.0) + 0001 Ret + flow: + stock: + literals: [0.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadVar level@0 + 0001 LoadVar growth@0 + 0002 LoadConstant #0 (=0.0) + 0003 Op2 Sub + 0004 LoadGlobalVar off=1 (dt) + 0005 Op2 Mul + 0006 BinOpAssignNext Add level@0 + 0007 Ret +== main::rate [explicit] : flow == + initial: + flow: + literals: [0.1] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 AssignConstCurr rate@0 #0 (=0.1) + 0001 Ret + stock: +########## runtime ########## + step 0: + $⁚$⁚ltm⁚link_score⁚growth→level⁚0⁚arg0 = 0.0 + $⁚$⁚ltm⁚link_score⁚growth→level⁚1⁚arg0 = 0.0 + $⁚$⁚ltm⁚link_score⁚growth→level⁚2⁚arg0 = 0.0 + $⁚ltm⁚link_score⁚growth→level = 0.0 + $⁚ltm⁚link_score⁚level→growth = 0.0 + $⁚ltm⁚link_score⁚rate→growth = 0.0 + dt = 1.0 + final_time = 2.0 + growth = 1.0 + initial_time = 0.0 + level = 10.0 + rate = 0.1 + time = 0.0 + step 1: + $⁚$⁚ltm⁚link_score⁚growth→level⁚0⁚arg0 = 1.0 + $⁚$⁚ltm⁚link_score⁚growth→level⁚1⁚arg0 = 1.0 + $⁚$⁚ltm⁚link_score⁚growth→level⁚2⁚arg0 = 10.0 + $⁚ltm⁚link_score⁚growth→level = 0.0 + $⁚ltm⁚link_score⁚level→growth = 1.0 + $⁚ltm⁚link_score⁚rate→growth = 0.0 + dt = 1.0 + final_time = 2.0 + growth = 1.1 + initial_time = 0.0 + level = 11.0 + rate = 0.1 + time = 1.0 + step 2: + $⁚$⁚ltm⁚link_score⁚growth→level⁚0⁚arg0 = 2.0 + $⁚$⁚ltm⁚link_score⁚growth→level⁚1⁚arg0 = 1.1 + $⁚$⁚ltm⁚link_score⁚growth→level⁚2⁚arg0 = 11.0 + $⁚ltm⁚link_score⁚growth→level = 1.0000000000000044 + $⁚ltm⁚link_score⁚level→growth = 1.0 + $⁚ltm⁚link_score⁚rate→growth = 0.0 + dt = 1.0 + final_time = 2.0 + growth = 1.21 + initial_time = 0.0 + level = 12.1 + rate = 0.1 + time = 2.0 diff --git a/src/simlin-engine/src/db/fragment_char_golden/ltm_loop_exhaustive.txt b/src/simlin-engine/src/db/fragment_char_golden/ltm_loop_exhaustive.txt new file mode 100644 index 000000000..1678cab78 --- /dev/null +++ b/src/simlin-engine/src/db/fragment_char_golden/ltm_loop_exhaustive.txt @@ -0,0 +1,307 @@ +########## model main (module inputs: []) ########## +== layout == + n_slots: 9 + $⁚$⁚ltm⁚link_score⁚growth→level⁚0⁚arg0: [6, 7) + $⁚$⁚ltm⁚link_score⁚growth→level⁚1⁚arg0: [7, 8) + $⁚$⁚ltm⁚link_score⁚growth→level⁚2⁚arg0: [8, 9) + $⁚ltm⁚link_score⁚growth→level: [3, 4) + $⁚ltm⁚link_score⁚level→growth: [4, 5) + $⁚ltm⁚loop_score⁚r1: [5, 6) + growth: [0, 1) + level: [1, 2) + rate: [2, 3) +== main::$⁚$⁚ltm⁚link_score⁚growth→level⁚0⁚arg0 [ltm-implicit-helper] : initial+flow == + initial: + literals: [] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadGlobalVar off=0 (time) + 0001 AssignCurr $⁚$⁚ltm⁚link_score⁚growth→level⁚0⁚arg0@0 + 0002 Ret + flow: + literals: [] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadGlobalVar off=0 (time) + 0001 AssignCurr $⁚$⁚ltm⁚link_score⁚growth→level⁚0⁚arg0@0 + 0002 Ret + stock: +== main::$⁚$⁚ltm⁚link_score⁚growth→level⁚1⁚arg0 [ltm-implicit-helper] : initial+flow == + initial: + literals: [0.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadConstant #0 (=0.0) + 0001 LoadPrev growth@0 + 0002 AssignCurr $⁚$⁚ltm⁚link_score⁚growth→level⁚1⁚arg0@0 + 0003 Ret + flow: + literals: [0.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadConstant #0 (=0.0) + 0001 LoadPrev growth@0 + 0002 AssignCurr $⁚$⁚ltm⁚link_score⁚growth→level⁚1⁚arg0@0 + 0003 Ret + stock: +== main::$⁚$⁚ltm⁚link_score⁚growth→level⁚2⁚arg0 [ltm-implicit-helper] : initial+flow == + initial: + literals: [0.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadConstant #0 (=0.0) + 0001 LoadPrev level@0 + 0002 AssignCurr $⁚$⁚ltm⁚link_score⁚growth→level⁚2⁚arg0@0 + 0003 Ret + flow: + literals: [0.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadConstant #0 (=0.0) + 0001 LoadPrev level@0 + 0002 AssignCurr $⁚$⁚ltm⁚link_score⁚growth→level⁚2⁚arg0@0 + 0003 Ret + stock: +== main::$⁚ltm⁚link_score⁚growth→level [ltm-synthetic] : flow == + initial: + flow: + literals: [0.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadConstant #0 (=0.0) + 0001 LoadGlobalVar off=1 (dt) + 0002 LoadConstant #0 (=0.0) + 0003 LoadPrev growth@0 + 0004 LoadConstant #0 (=0.0) + 0005 LoadPrev $⁚$⁚ltm⁚link_score⁚growth→level⁚1⁚arg0@0 + 0006 Op2 Sub + 0007 Op2 Mul + 0008 LoadVar level@0 + 0009 LoadConstant #0 (=0.0) + 0010 LoadPrev level@0 + 0011 Op2 Sub + 0012 LoadConstant #0 (=0.0) + 0013 LoadPrev level@0 + 0014 LoadConstant #0 (=0.0) + 0015 LoadPrev $⁚$⁚ltm⁚link_score⁚growth→level⁚2⁚arg0@0 + 0016 Op2 Sub + 0017 Op2 Sub + 0018 LoadConstant #0 (=0.0) + 0019 Apply SafeDiv + 0020 LoadConstant #0 (=0.0) + 0021 LoadConstant #0 (=0.0) + 0022 Apply Abs + 0023 LoadGlobalVar off=0 (time) + 0024 LoadGlobalVar off=2 (initial_time) + 0025 Op2 Eq + 0026 LoadGlobalVar off=2 (initial_time) + 0027 LoadPrev $⁚$⁚ltm⁚link_score⁚growth→level⁚0⁚arg0@0 + 0028 LoadGlobalVar off=2 (initial_time) + 0029 Op2 Eq + 0030 Op2 Or + 0031 SetCond + 0032 If + 0033 AssignCurr $⁚ltm⁚link_score⁚growth→level@0 + 0034 Ret + stock: +== main::$⁚ltm⁚link_score⁚level→growth [ltm-synthetic] : flow == + initial: + flow: + literals: [0.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadConstant #0 (=0.0) + 0001 LoadConstant #0 (=0.0) + 0002 LoadVar level@0 + 0003 LoadConstant #0 (=0.0) + 0004 LoadPrev rate@0 + 0005 Op2 Mul + 0006 LoadConstant #0 (=0.0) + 0007 LoadPrev growth@0 + 0008 Op2 Sub + 0009 LoadVar growth@0 + 0010 LoadConstant #0 (=0.0) + 0011 LoadPrev growth@0 + 0012 Op2 Sub + 0013 LoadConstant #0 (=0.0) + 0014 LoadConstant #0 (=0.0) + 0015 Apply Abs + 0016 LoadConstant #0 (=0.0) + 0017 Apply SafeDiv + 0018 LoadVar level@0 + 0019 LoadConstant #0 (=0.0) + 0020 LoadPrev level@0 + 0021 Op2 Sub + 0022 LoadConstant #0 (=0.0) + 0023 LoadConstant #0 (=0.0) + 0024 Apply Sign + 0025 Op2 Mul + 0026 LoadVar growth@0 + 0027 LoadConstant #0 (=0.0) + 0028 LoadPrev growth@0 + 0029 Op2 Sub + 0030 LoadConstant #0 (=0.0) + 0031 Op2 Eq + 0032 LoadVar level@0 + 0033 LoadConstant #0 (=0.0) + 0034 LoadPrev level@0 + 0035 Op2 Sub + 0036 LoadConstant #0 (=0.0) + 0037 Op2 Eq + 0038 Op2 Or + 0039 SetCond + 0040 If + 0041 LoadGlobalVar off=0 (time) + 0042 LoadGlobalVar off=2 (initial_time) + 0043 Op2 Eq + 0044 SetCond + 0045 If + 0046 AssignCurr $⁚ltm⁚link_score⁚level→growth@0 + 0047 Ret + stock: +== main::$⁚ltm⁚loop_score⁚r1 [ltm-synthetic] : flow == + initial: + flow: + literals: [] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadVar $⁚ltm⁚link_score⁚growth→level@0 + 0001 LoadVar $⁚ltm⁚link_score⁚level→growth@0 + 0002 BinOpAssignCurr Mul $⁚ltm⁚loop_score⁚r1@0 + 0003 Ret + stock: +== main::growth [explicit] : flow == + initial: + flow: + literals: [] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadVar level@0 + 0001 LoadVar rate@0 + 0002 BinOpAssignCurr Mul growth@0 + 0003 Ret + stock: +== main::level [explicit] : initial+stock == + initial: + literals: [10.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 AssignConstCurr level@0 #0 (=10.0) + 0001 Ret + flow: + stock: + literals: [0.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadVar level@0 + 0001 LoadVar growth@0 + 0002 LoadConstant #0 (=0.0) + 0003 Op2 Sub + 0004 LoadGlobalVar off=1 (dt) + 0005 Op2 Mul + 0006 BinOpAssignNext Add level@0 + 0007 Ret +== main::rate [explicit] : flow == + initial: + flow: + literals: [0.1] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 AssignConstCurr rate@0 #0 (=0.1) + 0001 Ret + stock: +########## runtime ########## + step 0: + $⁚$⁚ltm⁚link_score⁚growth→level⁚0⁚arg0 = 0.0 + $⁚$⁚ltm⁚link_score⁚growth→level⁚1⁚arg0 = 0.0 + $⁚$⁚ltm⁚link_score⁚growth→level⁚2⁚arg0 = 0.0 + $⁚ltm⁚link_score⁚growth→level = 0.0 + $⁚ltm⁚link_score⁚level→growth = 0.0 + $⁚ltm⁚loop_score⁚r1 = 0.0 + dt = 1.0 + final_time = 2.0 + growth = 1.0 + initial_time = 0.0 + level = 10.0 + rate = 0.1 + time = 0.0 + step 1: + $⁚$⁚ltm⁚link_score⁚growth→level⁚0⁚arg0 = 1.0 + $⁚$⁚ltm⁚link_score⁚growth→level⁚1⁚arg0 = 1.0 + $⁚$⁚ltm⁚link_score⁚growth→level⁚2⁚arg0 = 10.0 + $⁚ltm⁚link_score⁚growth→level = 0.0 + $⁚ltm⁚link_score⁚level→growth = 1.0 + $⁚ltm⁚loop_score⁚r1 = 0.0 + dt = 1.0 + final_time = 2.0 + growth = 1.1 + initial_time = 0.0 + level = 11.0 + rate = 0.1 + time = 1.0 + step 2: + $⁚$⁚ltm⁚link_score⁚growth→level⁚0⁚arg0 = 2.0 + $⁚$⁚ltm⁚link_score⁚growth→level⁚1⁚arg0 = 1.1 + $⁚$⁚ltm⁚link_score⁚growth→level⁚2⁚arg0 = 11.0 + $⁚ltm⁚link_score⁚growth→level = 1.0000000000000044 + $⁚ltm⁚link_score⁚level→growth = 1.0 + $⁚ltm⁚loop_score⁚r1 = 1.0000000000000044 + dt = 1.0 + final_time = 2.0 + growth = 1.21 + initial_time = 0.0 + level = 12.1 + rate = 0.1 + time = 2.0 diff --git a/src/simlin-engine/src/db/fragment_char_golden/modules.txt b/src/simlin-engine/src/db/fragment_char_golden/modules.txt new file mode 100644 index 000000000..8c9313aba --- /dev/null +++ b/src/simlin-engine/src/db/fragment_char_golden/modules.txt @@ -0,0 +1,269 @@ +########## model main (module inputs: []) ########## +== layout == + n_slots: 11 + $⁚smoothed⁚0⁚arg1: [5, 6) + $⁚smoothed⁚0⁚smth1: [6, 11) + smoothed: [0, 1) + src: [1, 2) + sub: [2, 4) + usesub: [4, 5) +== main::$⁚smoothed⁚0⁚arg1 [implicit-helper] : initial+flow == + initial: + literals: [2.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 AssignConstCurr $⁚smoothed⁚0⁚arg1@0 #0 (=2.0) + 0001 Ret + flow: + literals: [2.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 AssignConstCurr $⁚smoothed⁚0⁚arg1@0 #0 (=2.0) + 0001 Ret + stock: +== main::$⁚smoothed⁚0⁚smth1 [implicit-helper] : initial+flow+stock == + initial: + literals: [] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: + decl[0]: model=stdlib⁚smth1 inputs=[delay_time, input] var=$⁚smoothed⁚0⁚smth1@0 + static_views: [] + code: + 0000 LoadVar $⁚smoothed⁚0⁚arg1@0 + 0001 LoadVar src@0 + 0002 EvalModule module=0 n_inputs=2 + 0003 Ret + flow: + literals: [] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: + decl[0]: model=stdlib⁚smth1 inputs=[delay_time, input] var=$⁚smoothed⁚0⁚smth1@0 + static_views: [] + code: + 0000 LoadVar $⁚smoothed⁚0⁚arg1@0 + 0001 LoadVar src@0 + 0002 EvalModule module=0 n_inputs=2 + 0003 Ret + stock: + literals: [] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: + decl[0]: model=stdlib⁚smth1 inputs=[delay_time, input] var=$⁚smoothed⁚0⁚smth1@0 + static_views: [] + code: + 0000 LoadVar $⁚smoothed⁚0⁚arg1@0 + 0001 LoadVar src@0 + 0002 EvalModule module=0 n_inputs=2 + 0003 Ret +== main::smoothed [explicit] : initial+flow == + initial: + literals: [] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadVar $⁚smoothed⁚0⁚smth1@4 + 0001 AssignCurr smoothed@0 + 0002 Ret + flow: + literals: [] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadVar $⁚smoothed⁚0⁚smth1@4 + 0001 AssignCurr smoothed@0 + 0002 Ret + stock: +== main::src [explicit] : initial+flow == + initial: + literals: [3.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 AssignConstCurr src@0 #0 (=3.0) + 0001 Ret + flow: + literals: [3.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 AssignConstCurr src@0 #0 (=3.0) + 0001 Ret + stock: +== main::sub [explicit] : initial+flow+stock == + initial: + literals: [] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: + decl[0]: model=producer inputs=[input] var=sub@0 + static_views: [] + code: + 0000 LoadVar src@0 + 0001 EvalModule module=0 n_inputs=1 + 0002 Ret + flow: + literals: [] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: + decl[0]: model=producer inputs=[input] var=sub@0 + static_views: [] + code: + 0000 LoadVar src@0 + 0001 EvalModule module=0 n_inputs=1 + 0002 Ret + stock: + literals: [] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: + decl[0]: model=producer inputs=[input] var=sub@0 + static_views: [] + code: + 0000 LoadVar src@0 + 0001 EvalModule module=0 n_inputs=1 + 0002 Ret +== main::usesub [explicit] : flow == + initial: + flow: + literals: [2.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadVar sub@1 + 0001 LoadConstant #0 (=2.0) + 0002 BinOpAssignCurr Mul usesub@0 + 0003 Ret + stock: +########## model producer (module inputs: []) ########## +== layout == + n_slots: 2 + input: [0, 1) + output: [1, 2) +== producer::input [explicit] : flow == + initial: + flow: + literals: [0.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 AssignConstCurr input@0 #0 (=0.0) + 0001 Ret + stock: +== producer::output [explicit] : flow == + initial: + flow: + literals: [10.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadVar input@0 + 0001 LoadConstant #0 (=10.0) + 0002 BinOpAssignCurr Mul output@0 + 0003 Ret + stock: +########## model producer (module inputs: [input]) ########## +== layout == + n_slots: 2 + input: [0, 1) + output: [1, 2) +== producer::input [explicit] : flow == + initial: + flow: + literals: [] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadModuleInput input=0 + 0001 AssignCurr input@0 + 0002 Ret + stock: +== producer::output [explicit] : flow == + initial: + flow: + literals: [10.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadModuleInput input=0 + 0001 LoadConstant #0 (=10.0) + 0002 BinOpAssignCurr Mul output@0 + 0003 Ret + stock: +########## runtime ########## + step 0: + $⁚smoothed⁚0⁚arg1 = 2.0 + $⁚smoothed⁚0⁚smth1·delay_time = 2.0 + $⁚smoothed⁚0⁚smth1·flow = 0.0 + $⁚smoothed⁚0⁚smth1·initial_value = NaN + $⁚smoothed⁚0⁚smth1·input = 3.0 + $⁚smoothed⁚0⁚smth1·output = 3.0 + dt = 1.0 + final_time = 1.0 + initial_time = 0.0 + smoothed = 3.0 + src = 3.0 + sub·input = 3.0 + sub·output = 30.0 + time = 0.0 + usesub = 60.0 + step 1: + $⁚smoothed⁚0⁚arg1 = 2.0 + $⁚smoothed⁚0⁚smth1·delay_time = 2.0 + $⁚smoothed⁚0⁚smth1·flow = 0.0 + $⁚smoothed⁚0⁚smth1·initial_value = NaN + $⁚smoothed⁚0⁚smth1·input = 3.0 + $⁚smoothed⁚0⁚smth1·output = 3.0 + dt = 1.0 + final_time = 1.0 + initial_time = 0.0 + smoothed = 3.0 + src = 3.0 + sub·input = 3.0 + sub·output = 30.0 + time = 1.0 + usesub = 60.0 diff --git a/src/simlin-engine/src/db/fragment_char_golden/prev_init.txt b/src/simlin-engine/src/db/fragment_char_golden/prev_init.txt new file mode 100644 index 000000000..2d665995c --- /dev/null +++ b/src/simlin-engine/src/db/fragment_char_golden/prev_init.txt @@ -0,0 +1,192 @@ +########## model main (module inputs: []) ########## +== layout == + n_slots: 7 + $⁚init_expr⁚0⁚arg0: [5, 6) + $⁚prev_expr⁚0⁚arg0: [6, 7) + init_direct: [0, 1) + init_expr: [1, 2) + prev_direct: [2, 3) + prev_expr: [3, 4) + x: [4, 5) +== main::$⁚init_expr⁚0⁚arg0 [implicit-helper] : initial+flow == + initial: + literals: [1.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadVar x@0 + 0001 LoadConstant #0 (=1.0) + 0002 BinOpAssignCurr Add $⁚init_expr⁚0⁚arg0@0 + 0003 Ret + flow: + literals: [1.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadVar x@0 + 0001 LoadConstant #0 (=1.0) + 0002 BinOpAssignCurr Add $⁚init_expr⁚0⁚arg0@0 + 0003 Ret + stock: +== main::$⁚prev_expr⁚0⁚arg0 [implicit-helper] : flow == + initial: + flow: + literals: [1.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadVar x@0 + 0001 LoadConstant #0 (=1.0) + 0002 BinOpAssignCurr Add $⁚prev_expr⁚0⁚arg0@0 + 0003 Ret + stock: +== main::init_direct [explicit] : initial+flow == + initial: + literals: [] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadInitial x@0 + 0001 AssignCurr init_direct@0 + 0002 Ret + flow: + literals: [] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadInitial x@0 + 0001 AssignCurr init_direct@0 + 0002 Ret + stock: +== main::init_expr [explicit] : initial+flow == + initial: + literals: [] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadInitial $⁚init_expr⁚0⁚arg0@0 + 0001 AssignCurr init_expr@0 + 0002 Ret + flow: + literals: [] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadInitial $⁚init_expr⁚0⁚arg0@0 + 0001 AssignCurr init_expr@0 + 0002 Ret + stock: +== main::prev_direct [explicit] : flow == + initial: + flow: + literals: [0.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadConstant #0 (=0.0) + 0001 LoadPrev x@0 + 0002 AssignCurr prev_direct@0 + 0003 Ret + stock: +== main::prev_expr [explicit] : flow == + initial: + flow: + literals: [0.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadConstant #0 (=0.0) + 0001 LoadPrev $⁚prev_expr⁚0⁚arg0@0 + 0002 AssignCurr prev_expr@0 + 0003 Ret + stock: +== main::x [explicit] : initial+flow == + initial: + literals: [2.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadGlobalVar off=0 (time) + 0001 LoadConstant #0 (=2.0) + 0002 BinOpAssignCurr Mul x@0 + 0003 Ret + flow: + literals: [2.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadGlobalVar off=0 (time) + 0001 LoadConstant #0 (=2.0) + 0002 BinOpAssignCurr Mul x@0 + 0003 Ret + stock: +########## runtime ########## + step 0: + $⁚init_expr⁚0⁚arg0 = 1.0 + $⁚prev_expr⁚0⁚arg0 = 1.0 + dt = 1.0 + final_time = 2.0 + init_direct = 0.0 + init_expr = 1.0 + initial_time = 0.0 + prev_direct = 0.0 + prev_expr = 0.0 + time = 0.0 + x = 0.0 + step 1: + $⁚init_expr⁚0⁚arg0 = 3.0 + $⁚prev_expr⁚0⁚arg0 = 3.0 + dt = 1.0 + final_time = 2.0 + init_direct = 0.0 + init_expr = 1.0 + initial_time = 0.0 + prev_direct = 0.0 + prev_expr = 1.0 + time = 1.0 + x = 2.0 + step 2: + $⁚init_expr⁚0⁚arg0 = 5.0 + $⁚prev_expr⁚0⁚arg0 = 5.0 + dt = 1.0 + final_time = 2.0 + init_direct = 0.0 + init_expr = 1.0 + initial_time = 0.0 + prev_direct = 2.0 + prev_expr = 3.0 + time = 2.0 + x = 4.0 diff --git a/src/simlin-engine/src/db/fragment_char_golden/recurrence_scc.txt b/src/simlin-engine/src/db/fragment_char_golden/recurrence_scc.txt new file mode 100644 index 000000000..6be6cd819 --- /dev/null +++ b/src/simlin-engine/src/db/fragment_char_golden/recurrence_scc.txt @@ -0,0 +1,86 @@ +########## model main (module inputs: []) ########## +== layout == + n_slots: 6 + ce: [0, 3) + ecc: [3, 6) +== main::ce [explicit] : flow == + initial: + flow: + literals: [1.0, 1.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 AssignConstCurr ce@0 #0 (=1.0) + 0001 LoadVar ecc@0 + 0002 LoadConstant #1 (=1.0) + 0003 BinOpAssignCurr Add ce@1 + 0004 LoadVar ecc@1 + 0005 LoadConstant #1 (=1.0) + 0006 BinOpAssignCurr Add ce@2 + 0007 Ret + stock: +== main::ecc [explicit] : flow == + initial: + flow: + literals: [1.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadVar ce@0 + 0001 LoadConstant #0 (=1.0) + 0002 BinOpAssignCurr Add ecc@0 + 0003 LoadVar ce@1 + 0004 LoadConstant #0 (=1.0) + 0005 BinOpAssignCurr Add ecc@1 + 0006 LoadVar ce@2 + 0007 LoadConstant #0 (=1.0) + 0008 BinOpAssignCurr Add ecc@2 + 0009 Ret + stock: +########## resolved recurrence SCC ########## + phase: Dt + members: [ce, ecc] + element_order: [ce@0, ecc@0, ce@1, ecc@1, ce@2, ecc@2] + combined fragment: + literals: [1.0, 1.0, 1.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 AssignConstCurr ce@0 #0 (=1.0) + 0001 LoadVar ce@0 + 0002 LoadConstant #2 (=1.0) + 0003 BinOpAssignCurr Add ecc@0 + 0004 LoadVar ecc@0 + 0005 LoadConstant #1 (=1.0) + 0006 BinOpAssignCurr Add ce@1 + 0007 LoadVar ce@1 + 0008 LoadConstant #2 (=1.0) + 0009 BinOpAssignCurr Add ecc@1 + 0010 LoadVar ecc@1 + 0011 LoadConstant #1 (=1.0) + 0012 BinOpAssignCurr Add ce@2 + 0013 LoadVar ce@2 + 0014 LoadConstant #2 (=1.0) + 0015 BinOpAssignCurr Add ecc@2 + 0016 Ret +########## runtime ########## + step 0: + ce[t1] = 1.0 + ce[t2] = 3.0 + ce[t3] = 5.0 + dt = 1.0 + ecc[t1] = 2.0 + ecc[t2] = 4.0 + ecc[t3] = 6.0 + final_time = 1.0 + initial_time = 0.0 + time = 0.0 diff --git a/src/simlin-engine/src/db/fragment_char_golden/scalar_chain.txt b/src/simlin-engine/src/db/fragment_char_golden/scalar_chain.txt new file mode 100644 index 000000000..5539e82db --- /dev/null +++ b/src/simlin-engine/src/db/fragment_char_golden/scalar_chain.txt @@ -0,0 +1,125 @@ +########## model main (module inputs: []) ########## +== layout == + n_slots: 5 + derived: [0, 1) + inflow: [1, 2) + k: [2, 3) + level: [3, 4) + outflow: [4, 5) +== main::derived [explicit] : flow == + initial: + flow: + literals: [1.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadVar inflow@0 + 0001 LoadConstant #0 (=1.0) + 0002 BinOpAssignCurr Add derived@0 + 0003 Ret + stock: +== main::inflow [explicit] : flow == + initial: + flow: + literals: [3.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadVar k@0 + 0001 LoadConstant #0 (=3.0) + 0002 BinOpAssignCurr Mul inflow@0 + 0003 Ret + stock: +== main::k [explicit] : flow == + initial: + flow: + literals: [2.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 AssignConstCurr k@0 #0 (=2.0) + 0001 Ret + stock: +== main::level [explicit] : initial+stock == + initial: + literals: [10.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 AssignConstCurr level@0 #0 (=10.0) + 0001 Ret + flow: + stock: + literals: [] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadVar level@0 + 0001 LoadVar inflow@0 + 0002 LoadVar outflow@0 + 0003 Op2 Sub + 0004 LoadGlobalVar off=1 (dt) + 0005 Op2 Mul + 0006 BinOpAssignNext Add level@0 + 0007 Ret +== main::outflow [explicit] : flow == + initial: + flow: + literals: [0.1] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadVar level@0 + 0001 LoadConstant #0 (=0.1) + 0002 BinOpAssignCurr Mul outflow@0 + 0003 Ret + stock: +########## runtime ########## + step 0: + derived = 7.0 + dt = 1.0 + final_time = 2.0 + inflow = 6.0 + initial_time = 0.0 + k = 2.0 + level = 10.0 + outflow = 1.0 + time = 0.0 + step 1: + derived = 7.0 + dt = 1.0 + final_time = 2.0 + inflow = 6.0 + initial_time = 0.0 + k = 2.0 + level = 15.0 + outflow = 1.5 + time = 1.0 + step 2: + derived = 7.0 + dt = 1.0 + final_time = 2.0 + inflow = 6.0 + initial_time = 0.0 + k = 2.0 + level = 19.5 + outflow = 1.9500000000000002 + time = 2.0 diff --git a/src/simlin-engine/src/db/fragment_char_golden/subscripts.txt b/src/simlin-engine/src/db/fragment_char_golden/subscripts.txt new file mode 100644 index 000000000..0eb65b256 --- /dev/null +++ b/src/simlin-engine/src/db/fragment_char_golden/subscripts.txt @@ -0,0 +1,90 @@ +########## model main (module inputs: []) ########## +== layout == + n_slots: 6 + arr: [0, 3) + dynamic: [3, 4) + idx: [4, 5) + stat: [5, 6) +== main::arr [explicit] : flow == + initial: + flow: + literals: [1.0, 2.0, 3.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 AssignConstCurr arr@0 #0 (=1.0) + 0001 AssignConstCurr arr@1 #1 (=2.0) + 0002 AssignConstCurr arr@2 #2 (=3.0) + 0003 Ret + stock: +== main::dynamic [explicit] : flow == + initial: + flow: + literals: [] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadVar idx@0 + 0001 PushSubscriptIndex bounds=3 + 0002 LoadSubscript arr@0 + 0003 AssignCurr dynamic@0 + 0004 Ret + stock: +== main::idx [explicit] : flow == + initial: + flow: + literals: [1.0] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadConstant #0 (=1.0) + 0001 LoadGlobalVar off=0 (time) + 0002 BinOpAssignCurr Add idx@0 + 0003 Ret + stock: +== main::stat [explicit] : flow == + initial: + flow: + literals: [] + temp_sizes: [] + dim_lists: [] + graphical_functions: [] + module_decls: [] + static_views: [] + code: + 0000 LoadVar arr@1 + 0001 AssignCurr stat@0 + 0002 Ret + stock: +########## runtime ########## + step 0: + arr[east] = 1.0 + arr[north] = 3.0 + arr[west] = 2.0 + dt = 1.0 + dynamic = 1.0 + final_time = 1.0 + idx = 1.0 + initial_time = 0.0 + stat = 2.0 + time = 0.0 + step 1: + arr[east] = 1.0 + arr[north] = 3.0 + arr[west] = 2.0 + dt = 1.0 + dynamic = 2.0 + final_time = 1.0 + idx = 2.0 + initial_time = 0.0 + stat = 2.0 + time = 1.0 diff --git a/src/simlin-engine/src/db/fragment_char_tests.rs b/src/simlin-engine/src/db/fragment_char_tests.rs new file mode 100644 index 000000000..77e73cf82 --- /dev/null +++ b/src/simlin-engine/src/db/fragment_char_tests.rs @@ -0,0 +1,2801 @@ +// Copyright 2026 The Simlin Authors. All rights reserved. +// Use of this source code is governed by the Apache License, +// Version 2.0, that can be found in the LICENSE file. + +//! Characterization pins for the **per-variable fragment compiler** -- the +//! salsa-cached unit of production compilation (`db::compile_var_fragment` and +//! its implicit/LTM siblings). +//! +//! These were written while that compiler still reached its layout-independent +//! result the long way round -- private per-fragment offsets, a stand-in +//! one-variable `compiler::Module`, and a `symbolize_*` pass to undo both -- +//! and they are the gate GH #964's deletion of that round trip was measured +//! against. **All 12 goldens came through it byte-identical, with no +//! regeneration**, which is the strongest available evidence that the shorter +//! route produces the same value rather than a plausible one. The +//! integration corpus (`tests/integration/simulate.rs`) pins *numeric* results +//! of whole models, which is necessary but far too coarse here: it cannot see a +//! fragment change shape, a fragment stop being emitted (an `Option::None` +//! arm), a resource id get renumbered, or a fragment stop being deterministic. +//! +//! Three independent assertions per fixture, none of which subsumes another: +//! +//! 1. **A golden** of every fragment's rendered symbolic form -- opcode stream +//! with `SymVarRef { name, element_offset }` operands (since GH #964 that is +//! `compiler::VarRef`, the type lowering itself emits, not the output of a +//! conversion pass), literal pool, +//! graphical functions, module declarations, static views, temp sizes and +//! dim lists -- plus the model's `compute_layout` body layout and the VM's +//! result table. Regenerate with `UPDATE_FRAGMENT_GOLDEN=1`, but only after +//! a reviewer has adjudicated the change. +//! 2. **A declared phase map** ([`FixtureExpect::phases`]), a *required* +//! argument listing every rendered (variable, phase) pair and whether it +//! carries a fragment. This is the half a golden cannot provide: a golden +//! that pins an artifact is structurally blind to that artifact being +//! *stably absent* (the hard-won lesson recorded on +//! `ltm_char_tests::FragmentExpectation`), and `UPDATE_FRAGMENT_GOLDEN=1` +//! would happily re-capture a fixture whose fragments all vanished. The +//! phase map is hand-written and never regenerated, so it reds. +//! 3. **Hand-computed VM spot checks** ([`FixtureExpect::spot_checks`]), also +//! required and never regenerated. The rendered result table in the golden +//! shows a reviewer *what* moved; the spot checks are what stays meaningful +//! when a later commit legitimately changes bytecode shape. +//! +//! A fourth, universal assertion runs over every fragment of every fixture: +//! `temp_sizes` must be ordered by temp id. That is a determinism invariant, +//! not a formatting preference -- see `db::assemble::temp_sizes_by_id` -- and +//! `multi_temp_fragment_is_byte_identical_across_fresh_databases` sharpens it +//! from a coin flip into a reliable detector. +//! +//! The second half of the file measures INCREMENTALITY, which no golden can +//! see: which fragment-compiler bodies actually re-execute after an edit, using +//! the `#[cfg(test)]` execution records in `db::fragment_compile` rather than +//! memo pointer equality. Read `layout_only_edits_and_fragment_cache_reuse`'s +//! header comment for what that measurement found and for the two documented +//! reasons a module-instantiating add is still saturated. +//! +//! **That half is now the load-bearing one.** The value-equality assertions were +//! written when a fragment COULD have absorbed a layout dependency and still +//! looked right; after GH #964 a fragment cannot carry an offset of its own +//! model at all, and consulting one is a compile error, so the type subsumes +//! most of what those assertions were watching for. The execution counts are +//! what can still catch a regression -- a fragment newly depending on something +//! model-wide moves them and nothing else does. Do not loosen them. +//! +//! One property this suite established before that change is worth keeping on +//! the record, because the deletion rested on it: the goldens were byte- +//! identical under a reordering of the (now deleted) private dependency walk, +//! which was direct evidence that a fragment really was independent of the +//! offsets it was being handed. + +use std::collections::{BTreeMap, BTreeSet, HashMap}; + +use super::*; +use crate::compiler::symbolic::{ + PerVarBytecodes, SymStaticViewBase, SymVarRef, SymbolicModuleDecl, SymbolicOpcode, + SymbolicStaticView, +}; +use crate::datamodel; +use crate::test_common::TestProject; + +// --------------------------------------------------------------------------- +// Fixture declaration +// --------------------------------------------------------------------------- + +/// A compilation phase of one variable. The three phases are exactly the three +/// `Option` fields of `symbolic::CompiledVarFragment`. +#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] +enum Phase { + Initial, + Flow, + Stock, +} + +impl Phase { + const ALL: [Phase; 3] = [Phase::Initial, Phase::Flow, Phase::Stock]; + + fn label(self) -> &'static str { + match self { + Phase::Initial => "initial", + Phase::Flow => "flow", + Phase::Stock => "stock", + } + } +} + +/// Which emitter produced a fragment. Rendered into the golden so a variable +/// silently migrating between emitters (e.g. an explicit variable becoming an +/// implicit helper) is visible in the diff. +#[derive(Clone, Copy, PartialEq, Eq)] +enum FragmentKind { + Explicit, + Implicit, + Ltm, + LtmImplicit, +} + +impl FragmentKind { + fn label(self) -> &'static str { + match self { + FragmentKind::Explicit => "explicit", + FragmentKind::Implicit => "implicit-helper", + FragmentKind::Ltm => "ltm-synthetic", + FragmentKind::LtmImplicit => "ltm-implicit-helper", + } + } +} + +/// One rendered variable: its key (`model::variable`), which emitter produced +/// it, and its per-phase fragments. +struct RenderedVar { + key: String, + kind: FragmentKind, + initial: Option, + flow: Option, + stock: Option, +} + +impl RenderedVar { + fn phase(&self, phase: Phase) -> Option<&PerVarBytecodes> { + match phase { + Phase::Initial => self.initial.as_ref(), + Phase::Flow => self.flow.as_ref(), + Phase::Stock => self.stock.as_ref(), + } + } + + /// The canonical `initial+flow+stock` spelling of the phases that carry a + /// fragment, or `none`. This is the exact string a fixture declares. + fn phase_spelling(&self) -> String { + let present: Vec<&str> = Phase::ALL + .iter() + .filter(|p| self.phase(**p).is_some()) + .map(|p| p.label()) + .collect(); + if present.is_empty() { + "none".to_string() + } else { + present.join("+") + } + } +} + +/// Everything a fixture must state that the golden cannot state for it. +struct FixtureExpect { + /// The models to render, each with the module-input set to compile its + /// variables under. `&[]` is the input-agnostic set the diagnostic pass and + /// the element-graph probe use; a sub-model additionally deserves its real + /// instance input set, since that is a *different* salsa cache entry. + models: &'static [(&'static str, &'static [&'static str])], + /// EXHAUSTIVE `model::variable -> phases` map over every variable the + /// fixture renders, where `phases` is `none` or a `+`-joined subset of + /// `initial`, `flow`, `stock` in that order. + /// + /// Required, and never regenerated by `UPDATE_FRAGMENT_GOLDEN=1`. A + /// variable that stops being emitted, a phase that stops compiling, and a + /// variable that appears out of nowhere each fail here even if the golden + /// was re-captured. + phases: &'static [(&'static str, &'static str)], + /// Why the absences in `phases` are the right absences. A missing fragment + /// with no stated reason is indistinguishable from a bug that was captured + /// into a golden. + why: &'static str, + /// Hand-computed `(step, saved variable, value)` VM checks. Required and + /// non-empty: this is the runtime consequence, the assertion that survives + /// a legitimate bytecode-shape change. + spot_checks: &'static [(usize, &'static str, f64)], + /// Whether -- and in which loop-enumeration mode -- to enable LTM, so the + /// LTM synthetic vars and their implicit helpers are generated and + /// rendered. The two modes emit DIFFERENT synthetic variables from + /// different emitter arms, so both are worth a fixture. + ltm: FixtureLtm, + /// Assert the model resolves exactly one recurrence SCC and render its + /// combined fragment (the `combine_scc_fragment` interleave that + /// `assemble_module` injects) into the golden. + expect_one_resolved_scc: bool, +} + +impl FixtureExpect { + /// The common case: one `main` model, no module inputs, no LTM, no SCC. + const fn plain( + phases: &'static [(&'static str, &'static str)], + why: &'static str, + spot_checks: &'static [(usize, &'static str, f64)], + ) -> Self { + FixtureExpect { + models: &[("main", &[])], + phases, + why, + spot_checks, + ltm: FixtureLtm::Off, + expect_one_resolved_scc: false, + } + } +} + +/// How a fixture configures LTM. +#[derive(Clone, Copy, PartialEq, Eq)] +enum FixtureLtm { + Off, + /// Johnson enumeration of every elementary circuit: emits one link score + /// per loop edge plus a `$⁚ltm⁚loop_score⁚{id}` per circuit. + Exhaustive, + /// The strongest-path heuristic: emits a link score per causal edge + /// (including edges no circuit traverses) and no loop-score variables. + Discovery, +} + +// --------------------------------------------------------------------------- +// Rendering +// --------------------------------------------------------------------------- + +/// Shortest round-trip `f64` spelling (`1.0`, `0.5`, `NaN`, `inf`). Every +/// fixture's arithmetic is exact IEEE-754 `+ - * /` on small values, so this is +/// stable across platforms; deliberately NOT a rounded format, which would hide +/// a real numeric change. +fn f(v: f64) -> String { + format!("{v:?}") +} + +fn render_var_ref(v: &SymVarRef) -> String { + format!("{}@{}", v.name, v.element_offset) +} + +/// The four implicit globals live at fixed absolute slots and never go through +/// a variable layout, so `LoadGlobalVar` keeps a raw offset. Name it so the +/// golden reads as a program. +fn global_var_name(off: u16) -> &'static str { + match off as usize { + crate::vm::TIME_OFF => "time", + crate::vm::DT_OFF => "dt", + crate::vm::INITIAL_TIME_OFF => "initial_time", + crate::vm::FINAL_TIME_OFF => "final_time", + _ => "", + } +} + +/// Render one symbolic opcode as a single assembly-style line. +/// +/// The match is deliberately EXHAUSTIVE with no `_` arm: a new +/// `SymbolicOpcode` variant must be given a rendering here before it can +/// appear in a golden, so no reference-bearing opcode family can silently +/// enter the goldens as unprintable noise. +/// +/// Nine `SymbolicOpcode` variants carry a `SymVarRef` -- the operands the +/// GH #964 round trip exists to recover -- and the fixtures reach all nine. +/// There were eleven until C1b deleted the two that codegen could not emit: +/// `PushVarView` (never constructed in `compiler/codegen` at all -- only +/// `PushVarViewDirect` is) and a plain `AssignNext` (a stock update's last +/// operation is always the `Op2 Add` of `curr + net * dt`, so codegen now +/// emits the fused `BinOpAssignNext` directly and refuses any other shape). +/// Exhaustive coverage here is therefore a property of the suite, not a +/// coincidence: every reference-bearing opcode family the compiler can emit +/// appears in a golden. +fn render_opcode(op: &SymbolicOpcode, literals: &[f64]) -> String { + let lit = |id: u16| match literals.get(id as usize) { + Some(v) => format!("#{id} (={})", f(*v)), + None => format!("#{id} (=)"), + }; + match op { + SymbolicOpcode::Op2 { op } => format!("Op2 {op:?}"), + SymbolicOpcode::Not {} => "Not".to_string(), + SymbolicOpcode::LoadConstant { id } => format!("LoadConstant {}", lit(*id)), + SymbolicOpcode::LoadVar { var } => format!("LoadVar {}", render_var_ref(var)), + SymbolicOpcode::SymLoadPrev { var } => format!("LoadPrev {}", render_var_ref(var)), + SymbolicOpcode::SymLoadInitial { var } => format!("LoadInitial {}", render_var_ref(var)), + SymbolicOpcode::LoadGlobalVar { off } => { + format!("LoadGlobalVar off={off} ({})", global_var_name(*off)) + } + SymbolicOpcode::PushSubscriptIndex { bounds } => { + format!("PushSubscriptIndex bounds={bounds}") + } + SymbolicOpcode::LoadSubscript { var } => format!("LoadSubscript {}", render_var_ref(var)), + SymbolicOpcode::SetCond {} => "SetCond".to_string(), + SymbolicOpcode::If {} => "If".to_string(), + SymbolicOpcode::Ret => "Ret".to_string(), + SymbolicOpcode::LoadModuleInput { input } => format!("LoadModuleInput input={input}"), + SymbolicOpcode::EvalModule { id, n_inputs } => { + format!("EvalModule module={id} n_inputs={n_inputs}") + } + SymbolicOpcode::AssignCurr { var } => format!("AssignCurr {}", render_var_ref(var)), + SymbolicOpcode::Apply { func } => format!("Apply {func:?}"), + SymbolicOpcode::Lookup { + base_gf, + table_count, + mode, + } => format!("Lookup base_gf={base_gf} table_count={table_count} mode={mode:?}"), + SymbolicOpcode::AssignConstCurr { var, literal_id } => format!( + "AssignConstCurr {} {}", + render_var_ref(var), + lit(*literal_id) + ), + SymbolicOpcode::BinOpAssignCurr { op, var } => { + format!("BinOpAssignCurr {op:?} {}", render_var_ref(var)) + } + SymbolicOpcode::BinOpAssignNext { op, var } => { + format!("BinOpAssignNext {op:?} {}", render_var_ref(var)) + } + SymbolicOpcode::PushTempView { + temp_id, + dim_list_id, + } => format!("PushTempView temp={temp_id} dim_list={dim_list_id}"), + SymbolicOpcode::PushStaticView { view_id } => format!("PushStaticView view={view_id}"), + SymbolicOpcode::PushVarViewDirect { var, dim_list_id } => format!( + "PushVarViewDirect {} dim_list={dim_list_id}", + render_var_ref(var) + ), + SymbolicOpcode::ViewSubscriptConst { dim_idx, index } => { + format!("ViewSubscriptConst dim={dim_idx} index={index}") + } + SymbolicOpcode::ViewSubscriptDynamic { dim_idx } => { + format!("ViewSubscriptDynamic dim={dim_idx}") + } + SymbolicOpcode::ViewRange { + dim_idx, + start, + end, + } => format!("ViewRange dim={dim_idx} start={start} end={end}"), + SymbolicOpcode::ViewRangeDynamic { dim_idx } => format!("ViewRangeDynamic dim={dim_idx}"), + SymbolicOpcode::ViewStarRange { + dim_idx, + subdim_relation_id, + } => format!("ViewStarRange dim={dim_idx} subdim_relation={subdim_relation_id}"), + SymbolicOpcode::ViewWildcard { dim_idx } => format!("ViewWildcard dim={dim_idx}"), + SymbolicOpcode::ViewTranspose {} => "ViewTranspose".to_string(), + SymbolicOpcode::PopView {} => "PopView".to_string(), + SymbolicOpcode::DupView {} => "DupView".to_string(), + SymbolicOpcode::LoadTempConst { temp_id, index } => { + format!("LoadTempConst temp={temp_id} index={index}") + } + SymbolicOpcode::LoadTempDynamic { temp_id } => format!("LoadTempDynamic temp={temp_id}"), + SymbolicOpcode::BeginIter { + write_temp_id, + has_write_temp, + } => format!("BeginIter write_temp={write_temp_id} has_write_temp={has_write_temp}"), + SymbolicOpcode::LoadIterElement {} => "LoadIterElement".to_string(), + SymbolicOpcode::LoadIterTempElement { temp_id } => { + format!("LoadIterTempElement temp={temp_id}") + } + SymbolicOpcode::LoadIterViewTop {} => "LoadIterViewTop".to_string(), + SymbolicOpcode::LoadIterViewAt { offset } => format!("LoadIterViewAt offset={offset}"), + SymbolicOpcode::StoreIterElement {} => "StoreIterElement".to_string(), + SymbolicOpcode::NextIterOrJump { jump_back } => { + format!("NextIterOrJump jump_back={jump_back}") + } + SymbolicOpcode::EndIter {} => "EndIter".to_string(), + SymbolicOpcode::ArraySum {} => "ArraySum".to_string(), + SymbolicOpcode::ArrayMax {} => "ArrayMax".to_string(), + SymbolicOpcode::ArrayMin {} => "ArrayMin".to_string(), + SymbolicOpcode::ArrayMean {} => "ArrayMean".to_string(), + SymbolicOpcode::ArrayStddev {} => "ArrayStddev".to_string(), + SymbolicOpcode::ArraySize {} => "ArraySize".to_string(), + SymbolicOpcode::VectorSelect {} => "VectorSelect".to_string(), + SymbolicOpcode::VectorElmMap { + write_temp_id, + full_source_len, + } => format!("VectorElmMap write_temp={write_temp_id} full_source_len={full_source_len}"), + SymbolicOpcode::VectorSortOrder { write_temp_id } => { + format!("VectorSortOrder write_temp={write_temp_id}") + } + SymbolicOpcode::Rank { write_temp_id } => format!("Rank write_temp={write_temp_id}"), + SymbolicOpcode::LookupArray { + base_gf, + table_count, + mode, + write_temp_id, + } => format!( + "LookupArray base_gf={base_gf} table_count={table_count} mode={mode:?} \ + write_temp={write_temp_id}" + ), + SymbolicOpcode::AllocateAvailable { write_temp_id } => { + format!("AllocateAvailable write_temp={write_temp_id}") + } + SymbolicOpcode::AllocateByPriority { write_temp_id } => { + format!("AllocateByPriority write_temp={write_temp_id}") + } + SymbolicOpcode::BeginBroadcastIter { + n_sources, + dest_temp_id, + } => format!("BeginBroadcastIter n_sources={n_sources} dest_temp={dest_temp_id}"), + SymbolicOpcode::LoadBroadcastElement { source_idx } => { + format!("LoadBroadcastElement source={source_idx}") + } + SymbolicOpcode::StoreBroadcastElement {} => "StoreBroadcastElement".to_string(), + SymbolicOpcode::NextBroadcastOrJump { jump_back } => { + format!("NextBroadcastOrJump jump_back={jump_back}") + } + SymbolicOpcode::EndBroadcastIter {} => "EndBroadcastIter".to_string(), + } +} + +fn render_static_view(idx: usize, sv: &SymbolicStaticView) -> String { + let base = match &sv.base { + SymStaticViewBase::Var(v) => render_var_ref(v), + SymStaticViewBase::Temp(id) => format!("temp{id}"), + }; + let sparse: Vec = sv + .sparse + .iter() + .map(|m| format!("dim{}->{:?}", m.dim_index, m.parent_offsets.as_slice())) + .collect(); + format!( + " view[{idx}]: base={base} dims={:?} strides={:?} offset={} dim_ids={:?} sparse=[{}]", + sv.dims.as_slice(), + sv.strides.as_slice(), + sv.offset, + sv.dim_ids.as_slice(), + sparse.join(", ") + ) +} + +fn render_module_decl(idx: usize, md: &SymbolicModuleDecl) -> String { + let inputs: Vec<&str> = md.input_set.iter().map(|i| i.as_str()).collect(); + format!( + " decl[{idx}]: model={} inputs=[{}] var={}", + md.model_name.as_str(), + inputs.join(", "), + render_var_ref(&md.var) + ) +} + +/// Render one phase's `PerVarBytecodes` in full. Every field of the struct is +/// covered; a field added to `PerVarBytecodes` without a rendering here would +/// be invisible to the goldens, so keep this in sync with the struct. +fn render_fragment(bc: &PerVarBytecodes) -> String { + let mut out = String::new(); + let literals: Vec = bc.symbolic.literals.iter().map(|v| f(*v)).collect(); + out.push_str(&format!(" literals: [{}]\n", literals.join(", "))); + let temps: Vec = bc + .temp_sizes + .iter() + .map(|(id, size)| format!("temp{id}:{size}")) + .collect(); + out.push_str(&format!(" temp_sizes: [{}]\n", temps.join(", "))); + let dim_lists: Vec = bc.dim_lists.iter().map(|dl| format!("{dl:?}")).collect(); + out.push_str(&format!(" dim_lists: [{}]\n", dim_lists.join(", "))); + if bc.graphical_functions.is_empty() { + out.push_str(" graphical_functions: []\n"); + } else { + out.push_str(" graphical_functions:\n"); + for (i, gf) in bc.graphical_functions.iter().enumerate() { + let pts: Vec = gf + .iter() + .map(|(x, y)| format!("({}, {})", f(*x), f(*y))) + .collect(); + out.push_str(&format!(" gf[{i}]: {}\n", pts.join(" "))); + } + } + if bc.module_decls.is_empty() { + out.push_str(" module_decls: []\n"); + } else { + out.push_str(" module_decls:\n"); + for (i, md) in bc.module_decls.iter().enumerate() { + out.push_str(&render_module_decl(i, md)); + out.push('\n'); + } + } + if bc.static_views.is_empty() { + out.push_str(" static_views: []\n"); + } else { + out.push_str(" static_views:\n"); + for (i, sv) in bc.static_views.iter().enumerate() { + out.push_str(&render_static_view(i, sv)); + out.push('\n'); + } + } + out.push_str(" code:\n"); + for (pc, op) in bc.symbolic.code.iter().enumerate() { + out.push_str(&format!( + " {pc:04} {}\n", + render_opcode(op, &bc.symbolic.literals) + )); + } + out +} + +// --------------------------------------------------------------------------- +// Fragment collection (mirroring `assemble_module`'s enumeration) +// --------------------------------------------------------------------------- + +/// Collect every fragment `assemble_module` would compile for `model_name` +/// under `module_input_names`, in the same four passes assembly runs: explicit +/// source variables, implicit (SMOOTH/DELAY/TREND/PREVIOUS/INIT) helpers, and +/// -- when LTM is on -- LTM synthetic variables and their implicit helpers. +/// +/// Output is sorted by key so the golden is order-stable regardless of the +/// `HashMap` iteration order of any of the underlying maps. +/// +/// One deliberate divergence from assembly: `assemble_module` additionally +/// drops an LTM fragment whose symbolic references do not resolve in the +/// model's layout (`fragment_vars_in_layout`, the sub-model stdlib-instance +/// case). That filter is a property of the layout, not of the fragment +/// compiler, and it never fires on these single-model fixtures, so rendering +/// pre-filter keeps the goldens a pin on what the EMITTERS produce. +fn collect_model_fragments( + db: &SimlinDb, + project: SourceProject, + model_name: &str, + module_input_names: &[&str], +) -> Vec { + let model = *project + .models(db) + .get(model_name) + .unwrap_or_else(|| panic!("fixture declares model `{model_name}`, which does not exist")); + let owned_inputs: Vec = module_input_names.iter().map(|s| s.to_string()).collect(); + let inputs = ModuleInputSet::from_names(db, &owned_inputs); + let dep_graph = model_dependency_graph(db, model, project, inputs); + + let mut out: Vec = Vec::new(); + let push = |out: &mut Vec, kind: FragmentKind, result: &VarFragmentResult| { + out.push(RenderedVar { + key: format!("{model_name}::{}", result.fragment.ident), + kind, + initial: result.fragment.initial_bytecodes.clone(), + flow: result.fragment.flow_bytecodes.clone(), + stock: result.fragment.stock_bytecodes.clone(), + }); + }; + + let source_vars = model.variables(db); + let mut explicit_names: Vec<&String> = source_vars.keys().collect(); + explicit_names.sort(); + for name in explicit_names { + if let Some(result) = compile_var_fragment(db, source_vars[name], model, project, inputs) { + push(&mut out, FragmentKind::Explicit, result); + } + } + + let implicit_info = model_implicit_var_info(db, model, project); + let mut implicit_names: Vec<&String> = implicit_info.keys().collect(); + implicit_names.sort(); + for name in implicit_names { + if let Some(result) = compile_implicit_var_fragment( + db, + &implicit_info[name], + model, + project, + dep_graph, + &owned_inputs, + ) { + push(&mut out, FragmentKind::Implicit, &result); + } + } + + if project.ltm_enabled(db) { + let ltm_vars = model_ltm_variables(db, model, project); + let mut sorted: Vec<&LtmSyntheticVar> = ltm_vars.vars.iter().collect(); + sorted.sort_by(|a, b| a.name.cmp(&b.name)); + for ltm_var in sorted { + if let Some(result) = compile_ltm_synthetic_fragment(db, ltm_var, model, project) { + push(&mut out, FragmentKind::Ltm, &result); + } + } + let ltm_implicit = model_ltm_implicit_var_info(db, model, project); + let mut ltm_implicit_names: Vec<&String> = ltm_implicit.keys().collect(); + ltm_implicit_names.sort(); + for name in ltm_implicit_names { + if let Some(result) = compile_ltm_implicit_var_fragment( + db, + <m_implicit[name], + model, + project, + dep_graph, + &owned_inputs, + ) { + push(&mut out, FragmentKind::LtmImplicit, &result); + } + } + } + + out.sort_by(|a, b| a.key.cmp(&b.key)); + out +} + +/// `compute_layout`'s body layout for `model_name`, rendered as a compact +/// `name: [offset, offset+size)` table. +/// +/// Fragments are layout-INDEPENDENT by construction, so this cannot change +/// when a fragment does -- which is precisely why it is worth pinning +/// separately: a layout renumbering is a distinct failure class from a +/// fragment shape change, and only this section can see it. +fn render_layout(db: &SimlinDb, project: SourceProject, model_name: &str) -> String { + let model = *project.models(db).get(model_name).unwrap(); + let layout = compute_layout(db, model, project); + let mut names: Vec = model.variables(db).keys().cloned().collect(); + names.extend(model_implicit_var_info(db, model, project).keys().cloned()); + if project.ltm_enabled(db) { + names.extend( + model_ltm_variables(db, model, project) + .vars + .iter() + .map(|v| canonicalize(&v.name).into_owned()), + ); + names.extend( + model_ltm_implicit_var_info(db, model, project) + .keys() + .cloned(), + ); + } + names.sort(); + names.dedup(); + + let mut out = format!(" n_slots: {}\n", layout.n_slots); + for name in names { + match layout.get(&name) { + Some(entry) => out.push_str(&format!( + " {name}: [{}, {})\n", + entry.offset, + entry.offset + entry.size + )), + None => out.push_str(&format!(" {name}: \n")), + } + } + out +} + +// --------------------------------------------------------------------------- +// Runtime consequence +// --------------------------------------------------------------------------- + +/// Every saved variable's value at each of `steps`, sorted by name. +fn run_and_sample( + db: &SimlinDb, + project: SourceProject, + steps: &BTreeSet, +) -> BTreeMap> { + let compiled = compile_project_incremental(db, project, "main") + .expect("fixture must compile through the production incremental path"); + let mut vm = crate::vm::Vm::new(compiled).expect("fixture must build a VM"); + vm.run_to_end().expect("fixture must run to completion"); + let results = vm.into_results(); + + let mut sampled: BTreeMap> = BTreeMap::new(); + for &step in steps { + assert!( + step < results.step_count, + "fixture asks for step {step} but the run saved only {} steps", + results.step_count + ); + let row = &mut sampled.entry(step).or_default(); + for (name, &off) in results.offsets.iter() { + row.insert( + name.to_string(), + results.data[step * results.step_size + off], + ); + } + } + sampled +} + +fn render_runtime(sampled: &BTreeMap>) -> String { + let mut out = String::new(); + for (step, row) in sampled { + out.push_str(&format!(" step {step}:\n")); + for (name, value) in row { + out.push_str(&format!(" {name} = {}\n", f(*value))); + } + } + out +} + +// --------------------------------------------------------------------------- +// The characterization entry point +// --------------------------------------------------------------------------- + +/// Compare `actual` against `fragment_char_golden/{name}.txt`. Set +/// `UPDATE_FRAGMENT_GOLDEN=1` to (re)capture -- but only after a reviewer has +/// adjudicated the change; a re-captured golden that hides a vanished fragment +/// still fails on [`FixtureExpect::phases`]. A missing golden fails loudly. +#[track_caller] +fn assert_golden(name: &str, actual: &str) { + let dir = format!("{}/src/db/fragment_char_golden", env!("CARGO_MANIFEST_DIR")); + let path = format!("{dir}/{name}.txt"); + if std::env::var("UPDATE_FRAGMENT_GOLDEN").is_ok() { + std::fs::create_dir_all(&dir).expect("create golden dir"); + std::fs::write(&path, actual).expect("write golden"); + return; + } + let expected = std::fs::read_to_string(&path).unwrap_or_else(|e| { + panic!("missing golden {path}: {e}; run once with UPDATE_FRAGMENT_GOLDEN=1 to capture") + }); + if actual != expected { + eprintln!("\n===== GOLDEN MISMATCH ({name}): actual below ====="); + eprintln!("{actual}"); + eprintln!("===== end actual (expected in {path}) =====\n"); + } + assert_eq!(actual, &expected, "golden mismatch for {name}"); +} + +/// Build the fixture's db exactly as production does, plus the LTM flags when +/// the fixture asks for them. +fn fixture_db(project: &datamodel::Project, ltm: FixtureLtm) -> (SimlinDb, SourceProject) { + use salsa::Setter; + let mut db = SimlinDb::default(); + let source_project = sync_from_datamodel(&db, project).project; + match ltm { + FixtureLtm::Off => {} + FixtureLtm::Exhaustive => { + source_project.set_ltm_enabled(&mut db).to(true); + } + FixtureLtm::Discovery => { + source_project.set_ltm_discovery_mode(&mut db).to(true); + source_project.set_ltm_enabled(&mut db).to(true); + } + } + (db, source_project) +} + +/// `temp_sizes` must be strictly increasing in temp id. +/// +/// This is a determinism invariant, not a style rule: `PerVarBytecodes` is a +/// salsa-cached value with a derived `PartialEq` and `Vec` equality is +/// order-sensitive, so a `HashMap`-ordered `temp_sizes` makes two identical +/// compiles of the same fragment compare unequal whenever the per-process hash +/// seed differs -- defeating backdating and making the artifact irreproducible +/// (GH #595's class). Asserted on every fragment of every fixture rather than +/// in one place, because the ordering is established at THREE separate emission +/// sites (`db::assemble` and both copies in `db::ltm::compile`). +#[track_caller] +fn assert_temp_sizes_ordered(key: &str, phase: Phase, bc: &PerVarBytecodes) { + let ids: Vec = bc.temp_sizes.iter().map(|(id, _)| *id).collect(); + let mut sorted = ids.clone(); + sorted.sort_unstable(); + assert_eq!( + ids, + sorted, + "{key} ({}): temp_sizes must be ordered by temp id -- an unordered \ + (HashMap-iteration-order) vector makes this salsa-cached fragment \ + compare unequal to an identical recompile, defeating backdating and \ + making the compiled artifact run-to-run nondeterministic", + phase.label() + ); +} + +#[track_caller] +fn assert_fragment_fixture(golden: &str, project: datamodel::Project, expect: FixtureExpect) { + assert!( + !expect.spot_checks.is_empty(), + "fixture `{golden}` must declare at least one hand-computed VM spot \ + check: a fragment golden alone cannot tell a shape change from a \ + behavior change" + ); + + let (db, source_project) = fixture_db(&project, expect.ltm); + + let mut rendered = String::new(); + let mut actual_phases: Vec<(String, String)> = Vec::new(); + + for (model_name, module_inputs) in expect.models { + rendered.push_str(&format!( + "########## model {model_name} (module inputs: [{}]) ##########\n", + module_inputs.join(", ") + )); + rendered.push_str("== layout ==\n"); + rendered.push_str(&render_layout(&db, source_project, model_name)); + + for var in collect_model_fragments(&db, source_project, model_name, module_inputs) { + rendered.push_str(&format!( + "== {} [{}] : {} ==\n", + var.key, + var.kind.label(), + var.phase_spelling() + )); + for phase in Phase::ALL { + match var.phase(phase) { + None => rendered.push_str(&format!(" {}: \n", phase.label())), + Some(bc) => { + assert_temp_sizes_ordered(&var.key, phase, bc); + rendered.push_str(&format!(" {}:\n", phase.label())); + rendered.push_str(&render_fragment(bc)); + } + } + } + actual_phases.push((var.key.clone(), var.phase_spelling())); + } + } + + if expect.expect_one_resolved_scc { + rendered.push_str(&render_resolved_scc(&db, source_project)); + } + + // The declared phase map: exhaustive, hand-written, never regenerated. + // + // Checked BEFORE the model is run, deliberately. A dropped phase usually + // also makes the model refuse to compile (`NotSimulatable`) or run to a + // different answer, and if the run went first, every such regression would + // be reported as a downstream failure naming no phase. Reporting it here + // names the exact (variable, phase) pair that moved. + let expected_phases: Vec<(String, String)> = expect + .phases + .iter() + .map(|(k, v)| (k.to_string(), v.to_string())) + .collect(); + assert_eq!( + actual_phases, expected_phases, + "fixture `{golden}`'s (variable -> compiled phases) map does not match \ + its declaration.\n declared reason for the absences: {}\n A phase \ + that became `` is a silently-dropped fragment (the \ + variable keeps its layout slot and reads a constant 0). A phase that \ + appeared, or a variable that appeared/vanished, means the emitter's \ + gating changed. Neither can be fixed by re-running with \ + UPDATE_FRAGMENT_GOLDEN=1.", + expect.why + ); + + let steps: BTreeSet = expect.spot_checks.iter().map(|(s, _, _)| *s).collect(); + let sampled = run_and_sample(&db, source_project, &steps); + rendered.push_str("########## runtime ##########\n"); + rendered.push_str(&render_runtime(&sampled)); + + assert_golden(golden, &rendered); + + for (step, name, expected) in expect.spot_checks { + let row = sampled + .get(step) + .unwrap_or_else(|| panic!("fixture `{golden}`: no sampled row for step {step}")); + let actual = *row.get(*name).unwrap_or_else(|| { + panic!( + "fixture `{golden}`: spot check names `{name}`, which is not a saved \ + variable. Saved: {:?}", + row.keys().collect::>() + ) + }); + assert!( + (actual - expected).abs() < 1e-9, + "fixture `{golden}`: step {step} `{name}` = {actual}, hand-computed {expected}" + ); + } +} + +/// Assert the fixture resolves exactly one recurrence SCC and render its +/// combined fragment -- built through the EXACT production path +/// `assemble_module` uses (`var_phase_symbolic_fragment_prod` per member -> +/// `combine_scc_fragment`), so the golden pins the bytecode that is actually +/// injected into the runlist rather than a re-derivation. +fn render_resolved_scc(db: &SimlinDb, project: SourceProject) -> String { + let model = *project.models(db).get("main").unwrap(); + let dep_graph = model_dependency_graph(db, model, project, ModuleInputSet::empty(db)); + assert!( + !dep_graph.has_cycle, + "the SCC fixture's element-acyclic recurrence must survive the cycle gate" + ); + assert_eq!( + dep_graph.resolved_sccs.len(), + 1, + "the SCC fixture must resolve exactly one recurrence SCC" + ); + let scc = &dep_graph.resolved_sccs[0]; + + let mut member_fragments: HashMap, PerVarBytecodes> = HashMap::new(); + for member in &scc.members { + let frag = var_phase_symbolic_fragment_prod( + db, + model, + project, + member.as_str(), + scc.phase.clone(), + ) + .unwrap_or_else(|| { + panic!( + "SCC member `{}` must be element-sourceable", + member.as_str() + ) + }); + member_fragments.insert(member.clone(), frag); + } + let combined = combine_scc_fragment(scc, &member_fragments) + .expect("the resolved SCC must combine into one fragment"); + assert_temp_sizes_ordered("", Phase::Flow, &combined); + + let members: Vec<&str> = scc.members.iter().map(|m| m.as_str()).collect(); + let order: Vec = scc + .element_order + .iter() + .map(|(name, elem)| format!("{}@{elem}", name.as_str())) + .collect(); + let mut out = String::from("########## resolved recurrence SCC ##########\n"); + out.push_str(&format!(" phase: {:?}\n", scc.phase)); + out.push_str(&format!(" members: [{}]\n", members.join(", "))); + out.push_str(&format!(" element_order: [{}]\n", order.join(", "))); + out.push_str(" combined fragment:\n"); + out.push_str(&render_fragment(&combined)); + out +} + +// --------------------------------------------------------------------------- +// Fixture 1: scalar aux chain + a stock with an inflow and an outflow. +// +// The baseline shape: a constant, two flows, a stock, and a downstream aux. +// It is what pins the three-phase gating itself -- a stock compiles `initial` +// and `stock` but never `flow`, a non-stock compiles `flow` and (only when a +// stock's initial reaches it) `initial`. +// --------------------------------------------------------------------------- + +fn scalar_chain_model() -> datamodel::Project { + TestProject::new("frag_scalar_chain") + .with_sim_time(0.0, 2.0, 1.0) + .aux("k", "2", None) + .flow("inflow", "k * 3", None) + .flow("outflow", "level * 0.1", None) + .stock("level", "10", &["inflow"], &["outflow"], None) + .aux("derived", "inflow + 1", None) + .build_datamodel() +} + +#[test] +fn char_scalar_chain() { + assert_fragment_fixture( + "scalar_chain", + scalar_chain_model(), + FixtureExpect::plain( + &[ + ("main::derived", "flow"), + ("main::inflow", "flow"), + ("main::k", "flow"), + ("main::level", "initial+stock"), + ("main::outflow", "flow"), + ], + "A stock is excluded from the flows runlist (`!is_stock || \ + is_module_input`), so `level` carries `initial` (its `10` init) and \ + `stock` (its dt update) but no flow phase. No non-stock variable is \ + reachable from a stock's INITIAL equation here -- `level`'s init is \ + the literal `10` -- so none of them carries an initial phase.", + &[ + // k = 2, inflow = 6, outflow = 0.1 * level, derived = 7. + // t=0: level = 10 -> outflow 1.0, net +5 + // t=1: level = 15 -> outflow 1.5, net +4.5 + // t=2: level = 19.5 + (0, "level", 10.0), + (0, "outflow", 1.0), + (0, "derived", 7.0), + (1, "level", 15.0), + (1, "outflow", 1.5), + (2, "level", 19.5), + ], + ), + ); +} + +// --------------------------------------------------------------------------- +// Fixture 2: the three arrayed equation shapes. +// +// * `base[region]` -- `Equation::ApplyToAll` (one equation, all elements) +// * `perelem[region]` -- `Equation::Arrayed` (one equation PER element) +// * `exceptdef[region]` -- `Equation::Arrayed` with an EXCEPT default +// +// These three lower through visibly different paths (A2A iteration, per-element +// unrolling, default application), so the fragment shapes must stay distinct. +// --------------------------------------------------------------------------- + +fn arrayed_shapes_model() -> datamodel::Project { + TestProject::new("frag_arrayed_shapes") + .with_sim_time(0.0, 1.0, 1.0) + .named_dimension("region", &["east", "west"]) + .array_aux("base[region]", "10") + .array_with_ranges( + "perelem[region]", + vec![("east", "base[east] * 2"), ("west", "base[west] + 5")], + ) + .array_with_default_and_overrides("exceptdef[region]", "1", vec![("east", "7")]) + .build_datamodel() +} + +#[test] +fn char_arrayed_shapes() { + assert_fragment_fixture( + "arrayed_shapes", + arrayed_shapes_model(), + FixtureExpect::plain( + &[ + ("main::base", "flow"), + ("main::exceptdef", "flow"), + ("main::perelem", "flow"), + ], + "Three auxes, no stocks: nothing is in the initials or stocks \ + runlist, so every variable carries the flow phase alone.", + &[ + (0, "base[east]", 10.0), + (0, "base[west]", 10.0), + // per-element equations: east = 10 * 2, west = 10 + 5 + (0, "perelem[east]", 20.0), + (0, "perelem[west]", 15.0), + // EXCEPT default: `east` is overridden to 7, `west` takes the + // default `1`. + (0, "exceptdef[east]", 7.0), + (0, "exceptdef[west]", 1.0), + ], + ), + ); +} + +// --------------------------------------------------------------------------- +// Fixture 3: static vs dynamic subscripts. +// +// `stat = arr[west]` resolves its index at compile time (a `StaticSubscript` / +// constant view subscript); `dyn = arr[idx]` cannot, because `idx` is a +// variable, so it must emit a runtime index. Both reference `arr` through the +// mini-layout, so both are reference-bearing opcodes the round trip must +// round-trip. +// --------------------------------------------------------------------------- + +fn subscript_model() -> datamodel::Project { + TestProject::new("frag_subscripts") + .with_sim_time(0.0, 1.0, 1.0) + .named_dimension("region", &["east", "west", "north"]) + .array_with_ranges( + "arr[region]", + vec![("east", "1"), ("west", "2"), ("north", "3")], + ) + // `idx` is a variable, so `arr[idx]` is a genuine runtime subscript. + // Driving it off `time` also keeps it from being folded to a constant. + .scalar_aux("idx", "1 + time") + .scalar_aux("stat", "arr[west]") + .scalar_aux("dynamic", "arr[idx]") + .build_datamodel() +} + +#[test] +fn char_subscripts() { + assert_fragment_fixture( + "subscripts", + subscript_model(), + FixtureExpect::plain( + &[ + ("main::arr", "flow"), + ("main::dynamic", "flow"), + ("main::idx", "flow"), + ("main::stat", "flow"), + ], + "Four auxes, no stocks: flow phase only.", + &[ + (0, "stat", 2.0), + // idx = 1 + time; subscripts are 1-based, so t=0 selects + // `east` (1) and t=1 selects `west` (2). + (0, "idx", 1.0), + (0, "dynamic", 1.0), + (1, "idx", 2.0), + (1, "dynamic", 2.0), + ], + ), + ); +} + +// --------------------------------------------------------------------------- +// Fixture 3b: a RUNTIME array view, and an arrayed stock. +// +// The two reference-bearing shapes fixture 3 cannot reach. Everywhere else in +// this suite an array reference resolves at compile time into the static-view +// table (`PushStaticView`), which carries its base as a `SymbolicStaticView` +// rather than as an opcode operand. `VECTOR ELM MAP(arr[idx], ..)` cannot: +// `idx` is a variable, so the base view is pushed at RUNTIME by +// `PushVarViewDirect`, the only view opcode that carries a `SymVarRef` +// operand, and narrowed by `ViewSubscriptDynamic`. The arrayed stock adds +// per-element stock writes. +// +// (`SUM(mat[idx, *])` -- a dynamic index beside a wildcard -- would be the +// obvious way to reach the same opcodes and is NOT usable: the engine rejects +// it with `ArrayReferenceNeedsExplicitSubscripts`, loudly, before compiling.) +// +// Building this fixture surfaced a real bug, fixed in the same change: +// `codegen::full_source_len` had no arm for a DYNAMIC `Expr::Subscript`, so it +// reported a full source extent of 1 and every ELM MAP read outside `[0, 1)` +// returned the out-of-range `:NA:` (NaN) -- which was every read whenever +// `idx` selected anything but the source's first element. `mapped` was all-NaN +// at t=1 here. The golden's `full_source_len=3` operand and the `mapped` rows +// at both steps are the pin; the focused unit test is +// `array_tests::...::elm_map_dynamic_source_subscript_uses_full_variable_extent_vm`. +// The offsets are all zero only to keep this fixture about the VIEW opcodes. +// --------------------------------------------------------------------------- + +fn dynamic_view_model() -> datamodel::Project { + TestProject::new("frag_dynamic_view") + .with_sim_time(0.0, 1.0, 1.0) + .named_dimension("region", &["east", "west", "north"]) + .array_with_ranges( + "arr[region]", + vec![("east", "1"), ("west", "2"), ("north", "3")], + ) + // All-zero offsets: see the note above the fixture on why a non-zero + // offset cannot be used here. + .array_aux("off1[region]", "0") + .scalar_aux("idx", "1 + time") + .array_aux("mapped[region]", "VECTOR ELM MAP(arr[idx], off1[region])") + .array_stock("acc[region]", "0", &["grow"], &[], None) + .array_flow("grow[region]", "arr[region]", None) + .build_datamodel() +} + +#[test] +fn char_dynamic_view_and_arrayed_stock() { + assert_fragment_fixture( + "dynamic_view", + dynamic_view_model(), + FixtureExpect::plain( + &[ + ("main::acc", "initial+stock"), + ("main::arr", "flow"), + ("main::grow", "flow"), + ("main::idx", "flow"), + ("main::mapped", "flow"), + ("main::off1", "flow"), + ], + "`acc` is a stock: initial + stock, never flow. Everything else is \ + an aux or a flow, so flow only; no INIT reaches any of them, since \ + `acc`'s initial is the literal `0`.", + &[ + // VECTOR ELM MAP reads the source at `base + off1[i]`, where + // `base` is the flat position of `arr[idx]` and `idx = 1 + time` + // is 1-based. With all-zero offsets every element reads the base: + // arr = [1, 2, 3], so t=0 selects arr[east] and t=1 arr[west]. + (0, "mapped[east]", 1.0), + (0, "mapped[west]", 1.0), + (0, "mapped[north]", 1.0), + (1, "mapped[east]", 2.0), + (1, "mapped[north]", 2.0), + // The arrayed stock accumulates its arrayed inflow per element. + (0, "acc[north]", 0.0), + (1, "acc[east]", 1.0), + (1, "acc[west]", 2.0), + (1, "acc[north]", 3.0), + ], + ), + ); +} + +// --------------------------------------------------------------------------- +// Fixture 4: an array reducer plus two array-PRODUCING vector builtins. +// +// `total = SUM(arr[*])` is the reducer family (view push, reduce opcode, view +// pop). `order` and `ranked` are array-producing builtins, which the A2A +// hoisting path pre-computes once into an `AssignTemp` and then reads back +// per element -- so this fixture is what pins `temp_sizes`, the temp-carrying +// opcodes, and (via the universal ordering assertion) their determinism. +// --------------------------------------------------------------------------- + +fn array_ops_model() -> datamodel::Project { + TestProject::new("frag_array_ops") + .with_sim_time(0.0, 1.0, 1.0) + .named_dimension("region", &["east", "west", "north"]) + .array_with_ranges( + "arr[region]", + vec![("east", "30"), ("west", "10"), ("north", "20")], + ) + .scalar_aux("total", "SUM(arr[*])") + .array_aux("order[region]", "VECTOR SORT ORDER(arr[region], 1)") + .array_aux("ranked[region]", "RANK(arr[region], 1)") + // `sorted` routes an array-producing builtin's OUTPUT back through + // another one: ELM MAP reads `arr` at the offsets `order` computed. + .array_aux("sorted[region]", "VECTOR ELM MAP(arr[east], order[region])") + // Two array-producing builtins in ONE equation: the only shape that + // gives a single fragment more than one temp, and therefore the only + // one where `temp_sizes`' ordering is observable at all. + .array_aux( + "combo[region]", + "VECTOR SORT ORDER(arr[region], 1) + RANK(arr[region], 1)", + ) + .build_datamodel() +} + +#[test] +fn char_array_ops() { + assert_fragment_fixture( + "array_ops", + array_ops_model(), + FixtureExpect::plain( + &[ + ("main::arr", "flow"), + ("main::combo", "flow"), + ("main::order", "flow"), + ("main::ranked", "flow"), + ("main::sorted", "flow"), + ("main::total", "flow"), + ], + "Six auxes, no stocks: flow phase only.", + &[ + (0, "total", 60.0), + // VECTOR SORT ORDER ascending is the genuine-Vensim 0-BASED + // permutation: position i holds the source index of the i-th + // smallest element. arr = [30, 10, 20] -> [1, 2, 0]. + (0, "order[east]", 1.0), + (0, "order[west]", 2.0), + (0, "order[north]", 0.0), + // RANK is 1-BASED ordinal position: 30 is 3rd, 10 is 1st, + // 20 is 2nd. + (0, "ranked[east]", 3.0), + (0, "ranked[west]", 1.0), + (0, "ranked[north]", 2.0), + // VECTOR ELM MAP: result[i] = arr[base + order[i]] with base + // at `arr[east]` (flat 0), so `sorted` is `arr` ascending. + (0, "sorted[east]", 10.0), + (0, "sorted[west]", 20.0), + (0, "sorted[north]", 30.0), + // combo = order + ranked, element-wise. + (0, "combo[east]", 4.0), + (0, "combo[west]", 3.0), + (0, "combo[north]", 2.0), + ], + ), + ); +} + +// --------------------------------------------------------------------------- +// Fixture 5: graphical functions, scalar and per-element arrayed. +// +// * `curve` is the WITH LOOKUP shape -- a value-bearing aux whose own gf is +// applied to its input equation, lowering to a scalar `Lookup`. +// * `g[region]` carries one gf PER ELEMENT; `out` calls each element's own +// table (scalar `Lookup` at a per-element `base_gf`), and `gtotal` reduces +// over the whole arrayed-GF result, which routes through the array- +// producing `LookupArray` opcode into a temp (GH #580). +// +// `graphical_functions` is a fragment-local resource that assembly de-duplicates +// and renumbers (`GfDedup`, #582), so every fragment's gf block is pinned here. +// --------------------------------------------------------------------------- + +/// A two-point continuous gf over x in [0, 1] whose y-values are +/// `(base, base + slope)`. Evaluated at `time` with `time` running 0..=1 at +/// dt = 1 it yields exactly `base` then `base + slope`. +fn ramp_gf(base: f64, slope: f64) -> datamodel::GraphicalFunction { + datamodel::GraphicalFunction { + kind: datamodel::GraphicalFunctionKind::Continuous, + x_points: Some(vec![0.0, 1.0]), + y_points: vec![base, base + slope], + x_scale: datamodel::GraphicalFunctionScale { min: 0.0, max: 1.0 }, + y_scale: datamodel::GraphicalFunctionScale { + min: base.min(base + slope), + max: base.max(base + slope), + }, + } +} + +/// A per-element arrayed-GF holder: one `(element, "time", None, gf)` slot per +/// element, so each element's value is its OWN table evaluated at `time`. +fn arrayed_gf_holder( + ident: &str, + dim: &str, + elems: Vec<(&str, datamodel::GraphicalFunction)>, +) -> datamodel::Variable { + datamodel::Variable::Aux(datamodel::Aux { + ident: ident.to_string(), + equation: datamodel::Equation::Arrayed( + vec![dim.to_string()], + elems + .into_iter() + .map(|(name, gf)| (name.to_string(), "time".to_string(), None, Some(gf))) + .collect(), + None, + false, + ), + documentation: String::new(), + units: None, + gf: None, + ai_state: None, + uid: None, + compat: datamodel::Compat::default(), + }) +} + +fn graphical_function_model() -> datamodel::Project { + let mut tp = TestProject::new("frag_gf") + .with_sim_time(0.0, 1.0, 1.0) + .named_dimension("region", &["east", "west"]) + .scalar_aux("drive", "time") + .aux_with_gf("curve", "drive", ramp_gf(5.0, 10.0)); + tp.variables.push(arrayed_gf_holder( + "g", + "region", + vec![("east", ramp_gf(1.0, 1.0)), ("west", ramp_gf(2.0, 2.0))], + )); + tp.array_aux("out[region]", "LOOKUP(g[region], time)") + .scalar_aux("gtotal", "SUM(LOOKUP(g[*], time))") + .build_datamodel() +} + +#[test] +fn char_graphical_functions() { + assert_fragment_fixture( + "graphical_functions", + graphical_function_model(), + FixtureExpect::plain( + &[ + ("main::curve", "flow"), + ("main::drive", "flow"), + ("main::g", "flow"), + ("main::gtotal", "flow"), + ("main::out", "flow"), + ], + "Five auxes, no stocks: flow phase only.", + &[ + // curve's gf is (0,5) -> (1,15) evaluated at drive == time. + (0, "curve", 5.0), + (1, "curve", 15.0), + // per-element tables: east (0,1)->(1,2), west (0,2)->(1,4). + (0, "g[east]", 1.0), + (0, "g[west]", 2.0), + (1, "g[east]", 2.0), + (1, "g[west]", 4.0), + (0, "out[east]", 1.0), + (1, "out[west]", 4.0), + (0, "gtotal", 3.0), + (1, "gtotal", 6.0), + ], + ), + ); +} + +// --------------------------------------------------------------------------- +// Fixture 6: a standalone lookup-only table plus two consumers (#606). +// +// `table` is a *static table*, not a value-bearing variable: it is excluded +// from every runlist and from the saved output, so it must compile to NO +// fragment at all -- which is exactly the class of absence a golden alone is +// blind to and the declared phase map catches. Its data reaches the two +// consumers only through their `LOOKUP(table, x)` call sites, which resolve +// the table by ident rather than by a layout reference. +// --------------------------------------------------------------------------- + +fn lookup_only_model() -> datamodel::Project { + TestProject::new("frag_lookup_only") + .with_sim_time(0.0, 1.0, 1.0) + // An empty equation plus a gf IS the lookup-only form. + .aux_with_gf("table", "", ramp_gf(3.0, 4.0)) + .scalar_aux("at_time", "LOOKUP(table, time)") + .scalar_aux("at_half", "LOOKUP(table, 0.5)") + .build_datamodel() +} + +#[test] +fn char_lookup_only_table() { + assert_fragment_fixture( + "lookup_only_table", + lookup_only_model(), + FixtureExpect::plain( + &[ + ("main::at_half", "flow"), + ("main::at_time", "flow"), + ("main::table", "none"), + ], + "`table` is a standalone lookup-only holder (#606): a static table, \ + excluded from every runlist and from the saved output. \ + `compile_var_fragment` still returns a fragment for it -- so it \ + appears here -- but with all three phases empty, because the \ + runlist gates reject it in each. `none` is therefore the CONTRACT \ + for this variable, not a dropped fragment: if it ever gains a \ + phase it has started producing a series of its own, and if the \ + row disappears entirely the fragment compiler started returning \ + whole-variable `None` for it (a different, and unattributed, \ + failure path).", + &[ + // table is (0,3) -> (1,7); at_time samples it at `time`. + (0, "at_time", 3.0), + (1, "at_time", 7.0), + // ...and at_half samples the SAME table at a different, constant + // argument: two calls are independent. + (0, "at_half", 5.0), + ], + ), + ); +} + +// --------------------------------------------------------------------------- +// Fixture 7: an explicit sub-model instance with a wired input, plus a stdlib +// module call. +// +// This is the module-declaration family: `sub` compiles to a fragment carrying +// a `SymbolicModuleDecl` and an `EvalModule`, its input is fed by a +// `LoadModuleInput`, and the sub-model's own variables compile as a SEPARATE +// salsa cache entry per module-input set -- which is why the fixture renders +// `producer` twice, once input-agnostically (the diagnostic pass / element-graph +// probe wiring) and once at its real instance input set `{input}` (assembly's). +// +// `smoothed = SMTH1(src, 2)` additionally synthesizes the implicit +// SMOOTH helper variables, so this fixture is what pins +// `compile_implicit_var_fragment`'s output. +// --------------------------------------------------------------------------- + +fn module_model() -> datamodel::Project { + let aux = |ident: &str, equation: &str, can_be_module_input: bool| { + datamodel::Variable::Aux(datamodel::Aux { + ident: ident.to_string(), + equation: datamodel::Equation::Scalar(equation.to_string()), + documentation: String::new(), + units: None, + gf: None, + ai_state: None, + uid: None, + compat: datamodel::Compat { + can_be_module_input, + ..datamodel::Compat::default() + }, + }) + }; + datamodel::Project { + name: "frag_modules".to_string(), + sim_specs: datamodel::SimSpecs { + start: 0.0, + stop: 1.0, + dt: datamodel::Dt::Dt(1.0), + save_step: None, + sim_method: datamodel::SimMethod::Euler, + time_units: None, + }, + dimensions: vec![], + units: vec![], + models: vec![ + datamodel::Model { + name: "main".to_string(), + sim_specs: None, + variables: vec![ + aux("src", "3", false), + datamodel::Variable::Module(datamodel::Module { + ident: "sub".to_string(), + model_name: "producer".to_string(), + documentation: String::new(), + units: None, + references: vec![datamodel::ModuleReference { + src: "src".to_string(), + dst: "sub.input".to_string(), + }], + ai_state: None, + uid: None, + compat: datamodel::Compat::default(), + }), + aux("usesub", "sub.output * 2", false), + aux("smoothed", "SMTH1(src, 2)", false), + ], + views: vec![], + loop_metadata: vec![], + groups: vec![], + macro_spec: None, + }, + datamodel::Model { + name: "producer".to_string(), + sim_specs: None, + variables: vec![aux("input", "0", true), aux("output", "input * 10", false)], + views: vec![], + loop_metadata: vec![], + groups: vec![], + macro_spec: None, + }, + ], + source: None, + ai_information: None, + } +} + +#[test] +fn char_modules_and_stdlib_call() { + assert_fragment_fixture( + "modules", + module_model(), + FixtureExpect { + // `producer` twice, deliberately. A sub-model compiles a SEPARATE + // salsa cache entry per module-input set, and the two entries are + // materially different bytecode: at `&[]` the port variable + // compiles its own equation, at `{input}` it compiles + // `LoadModuleInput`. The `&[]` entry is the one the diagnostic + // pass and the element-graph SCC probe consume -- the surface + // stages 2 and 3 are most likely to perturb -- and the `{input}` + // entry is the one assembly consumes. + models: &[("main", &[]), ("producer", &[]), ("producer", &["input"])], + phases: &[ + ("main::$⁚smoothed⁚0⁚arg1", "initial+flow"), + ("main::$⁚smoothed⁚0⁚smth1", "initial+flow+stock"), + ("main::smoothed", "initial+flow"), + ("main::src", "initial+flow"), + ("main::sub", "initial+flow+stock"), + ("main::usesub", "flow"), + ("producer::input", "flow"), + ("producer::output", "flow"), + ("producer::input", "flow"), + ("producer::output", "flow"), + ], + why: "A module variable is the ONLY shape that compiles all three \ + phases: it is not a stock, so the `!is_stock` arm of the \ + flows gate admits it, and it IS a module, so the \ + `is_stock || is_module` stock gate admits it too -- hence \ + `initial+flow+stock` for both `sub` and the SMTH1 instance. \ + The SMOOTH's internal stock puts everything that reaches its \ + initial equation into the initials runlist as well: `src`, \ + the synthesized `arg1` delay-time helper, `smoothed` (which \ + reads the instance's output), and `sub`. `usesub` is not \ + among them because `producer` holds no stock. \ + `producer::input` carries a flow fragment at BOTH input sets, \ + and for the ordinary reason: it is an Aux, so `!is_stock` \ + already satisfies the flows gate. The `is_module_input` arm \ + of that gate (`fragment_compile.rs`, \ + `(!is_stock || is_module_input) && …`) only ever admits a \ + STOCK-typed input port, which this fixture has none of. What \ + the input set changes here is the compiled BODY, not the \ + gate: compare the two `producer::input` renders in the \ + golden -- `AssignConstCurr` of its own `0` equation at `&[]`, \ + `LoadModuleInput` at `{input}`.", + spot_checks: &[ + // src = 3 -> producer.input = 3 -> producer.output = 30 -> + // usesub = 60. SMTH1's default initial IS its input, and the + // input never moves, so `smoothed` sits at 3 forever. + (0, "src", 3.0), + (0, "sub\u{00B7}output", 30.0), + (0, "usesub", 60.0), + (0, "smoothed", 3.0), + (1, "smoothed", 3.0), + ], + ltm: FixtureLtm::Off, + expect_one_resolved_scc: false, + }, + ); +} + +// --------------------------------------------------------------------------- +// Fixture 8: PREVIOUS and INIT, in both of their lowered forms. +// +// A DIRECT scalar argument compiles to the `LoadPrev` / `LoadInitial` opcode; +// an expression argument is first rewritten through a synthesized scalar helper +// aux (`builtins_visitor.rs`), so the opcode reads the HELPER rather than the +// user variable. Both forms are reference-bearing, and the helper form is the +// only place an implicit helper is read by a `SymLoadPrev` / `SymLoadInitial`. +// --------------------------------------------------------------------------- + +fn prev_init_model() -> datamodel::Project { + TestProject::new("frag_prev_init") + .with_sim_time(0.0, 2.0, 1.0) + .scalar_aux("x", "time * 2") + .scalar_aux("prev_direct", "PREVIOUS(x)") + .scalar_aux("prev_expr", "PREVIOUS(x + 1)") + .scalar_aux("init_direct", "INIT(x)") + .scalar_aux("init_expr", "INIT(x + 1)") + .build_datamodel() +} + +#[test] +fn char_prev_and_init() { + assert_fragment_fixture( + "prev_init", + prev_init_model(), + FixtureExpect::plain( + &[ + ("main::$⁚init_expr⁚0⁚arg0", "initial+flow"), + ("main::$⁚prev_expr⁚0⁚arg0", "flow"), + ("main::init_direct", "initial+flow"), + ("main::init_expr", "initial+flow"), + ("main::prev_direct", "flow"), + ("main::prev_expr", "flow"), + ("main::x", "initial+flow"), + ], + "`INIT(...)` reads the frozen initial-values buffer, so everything \ + an INIT argument reaches must also be evaluated in the initials \ + phase: `x`, the `INIT(x + 1)` capture helper, and the two INIT \ + consumers. `PREVIOUS` reads the PRIOR step's committed values, \ + which the initials phase has not produced, so both PREVIOUS \ + consumers and the `PREVIOUS(x + 1)` capture helper are flow-only \ + -- the asymmetry between the two synthesized `arg0` helpers is \ + the load-bearing detail here.", + &[ + // x = 2 * time -> 0, 2, 4. + (0, "x", 0.0), + (2, "x", 4.0), + // PREVIOUS(x) is 0 at the initial step (the unary form + // desugars to PREVIOUS(x, 0)), then the prior step's x. + (0, "prev_direct", 0.0), + (1, "prev_direct", 0.0), + (2, "prev_direct", 2.0), + // PREVIOUS(x + 1) goes through a capture helper holding x + 1. + (1, "prev_expr", 1.0), + (2, "prev_expr", 3.0), + // INIT freezes at the initial step: x@t0 = 0, (x + 1)@t0 = 1. + (2, "init_direct", 0.0), + (2, "init_expr", 1.0), + ], + ), + ); +} + +// --------------------------------------------------------------------------- +// Fixture 9: a resolved recurrence SCC (the `ref.mdl` shape). +// +// `ce` and `ecc` form a whole-variable 2-cycle whose induced ELEMENT graph is +// acyclic, so `resolve_recurrence_sccs` resolves it and `assemble_module` +// injects ONE combined fragment built by interleaving the members' per-element +// segments. That combined fragment is a second, distinct consumer of the exact +// production per-variable fragment (through `var_phase_symbolic_fragment_prod`), +// so it is rendered into the golden alongside the members it is built from. +// --------------------------------------------------------------------------- + +fn recurrence_scc_model() -> datamodel::Project { + TestProject::new("frag_recurrence_scc") + .with_sim_time(0.0, 1.0, 1.0) + .named_dimension("t", &["t1", "t2", "t3"]) + .array_with_ranges( + "ce[t]", + vec![("t1", "1"), ("t2", "ecc[t1] + 1"), ("t3", "ecc[t2] + 1")], + ) + .array_with_ranges( + "ecc[t]", + vec![ + ("t1", "ce[t1] + 1"), + ("t2", "ce[t2] + 1"), + ("t3", "ce[t3] + 1"), + ], + ) + .build_datamodel() +} + +#[test] +fn char_resolved_recurrence_scc() { + assert_fragment_fixture( + "recurrence_scc", + recurrence_scc_model(), + FixtureExpect { + models: &[("main", &[])], + phases: &[("main::ce", "flow"), ("main::ecc", "flow")], + why: "Two arrayed auxes, no stocks: flow phase only. Both members \ + still compile their OWN per-variable fragment -- the combined \ + SCC fragment is built from them and injected at assembly, it \ + does not replace them here.", + spot_checks: &[ + // The element chain: ce[t1] = 1, ecc[e] = ce[e] + 1, + // ce[t2] = ecc[t1] + 1, ce[t3] = ecc[t2] + 1. + (0, "ce[t1]", 1.0), + (0, "ecc[t1]", 2.0), + (0, "ce[t2]", 3.0), + (0, "ecc[t2]", 4.0), + (0, "ce[t3]", 5.0), + (0, "ecc[t3]", 6.0), + ], + ltm: FixtureLtm::Off, + expect_one_resolved_scc: true, + }, + ); +} + +// --------------------------------------------------------------------------- +// Fixture 10: an LTM-enabled model, in BOTH loop-enumeration modes. +// +// The two `db/ltm/compile.rs` emission sites -- each an inline COPY of the +// shared compile+symbolize tail that `db::assemble` also carries -- are what +// this fixture reaches: `compile_ltm_synthetic_fragment` for the link/loop +// score variables, and `compile_ltm_implicit_var_fragment` for the PREVIOUS +// capture helpers those score equations synthesize. +// +// Both modes are rendered because they emit DIFFERENT synthetic variables from +// different arms: exhaustive enumeration produces one link score per LOOP edge +// plus a `$⁚ltm⁚loop_score⁚{id}` per circuit (the `compile_direct` arm), +// while discovery produces one link score per CAUSAL edge -- including +// `rate→growth`, which no circuit traverses -- and no loop scores. Pinning only +// one would leave the other arm's fragment shape unpinned. The model is +// deliberately the smallest one with a feedback loop, since the +// synthetic-variable count grows fast. +// --------------------------------------------------------------------------- + +fn ltm_loop_model() -> datamodel::Project { + TestProject::new("frag_ltm_loop") + .with_sim_time(0.0, 2.0, 1.0) + .aux("rate", "0.1", None) + .flow("growth", "level * rate", None) + .stock("level", "10", &["growth"], &[], None) + .build_datamodel() +} + +/// The core simulation is unchanged by LTM: level starts at 10 and +/// growth = 0.1 * level, Euler at dt = 1 -> 10, 11, 12.1. +const LTM_LOOP_SPOT_CHECKS: &[(usize, &str, f64)] = &[ + (0, "level", 10.0), + (0, "growth", 1.0), + (1, "level", 11.0), + (2, "level", 12.1), +]; + +#[test] +fn char_ltm_fragments_exhaustive() { + assert_fragment_fixture( + "ltm_loop_exhaustive", + ltm_loop_model(), + FixtureExpect { + models: &[("main", &[])], + phases: &[ + ( + "main::$⁚$⁚ltm⁚link_score⁚growth→level⁚0⁚arg0", + "initial+flow", + ), + ( + "main::$⁚$⁚ltm⁚link_score⁚growth→level⁚1⁚arg0", + "initial+flow", + ), + ( + "main::$⁚$⁚ltm⁚link_score⁚growth→level⁚2⁚arg0", + "initial+flow", + ), + ("main::$⁚ltm⁚link_score⁚growth→level", "flow"), + ("main::$⁚ltm⁚link_score⁚level→growth", "flow"), + ("main::$⁚ltm⁚loop_score⁚r1", "flow"), + ("main::growth", "flow"), + ("main::level", "initial+stock"), + ("main::rate", "flow"), + ], + why: "Exhaustive enumeration finds the single circuit \ + `level -> growth -> level` and emits a link score for each of \ + its TWO edges plus one `loop_score⁚r1` for the circuit -- so \ + `rate→growth`, a causal edge no circuit traverses, gets no \ + score here (contrast the discovery fixture). Every synthetic \ + is a scalar aux, hence flow-only. Only the `growth→level` \ + score synthesizes PREVIOUS capture helpers, and those land in \ + the initials runlist because a stock's initial equation \ + reaches them.", + spot_checks: LTM_LOOP_SPOT_CHECKS, + ltm: FixtureLtm::Exhaustive, + expect_one_resolved_scc: false, + }, + ); +} + +#[test] +fn char_ltm_fragments_discovery() { + assert_fragment_fixture( + "ltm_loop_discovery", + ltm_loop_model(), + FixtureExpect { + models: &[("main", &[])], + phases: &[ + ( + "main::$⁚$⁚ltm⁚link_score⁚growth→level⁚0⁚arg0", + "initial+flow", + ), + ( + "main::$⁚$⁚ltm⁚link_score⁚growth→level⁚1⁚arg0", + "initial+flow", + ), + ( + "main::$⁚$⁚ltm⁚link_score⁚growth→level⁚2⁚arg0", + "initial+flow", + ), + ("main::$⁚ltm⁚link_score⁚growth→level", "flow"), + ("main::$⁚ltm⁚link_score⁚level→growth", "flow"), + ("main::$⁚ltm⁚link_score⁚rate→growth", "flow"), + ("main::growth", "flow"), + ("main::level", "initial+stock"), + ("main::rate", "flow"), + ], + why: "Discovery mode scores every CAUSAL edge, so `rate→growth` \ + appears here even though no circuit traverses it, and no \ + loop-score variable is emitted at all (discovery ranks \ + strongest paths per step instead of enumerating circuits). \ + Every score is a scalar aux, hence flow-only. Only the \ + `growth→level` score -- the stock-update edge, whose \ + ceteris-paribus numerator re-integrates the stock -- \ + synthesizes PREVIOUS capture helpers, and those land in the \ + initials runlist because a stock's initial equation reaches \ + them.", + spot_checks: LTM_LOOP_SPOT_CHECKS, + ltm: FixtureLtm::Discovery, + expect_one_resolved_scc: false, + }, + ); +} + +// --------------------------------------------------------------------------- +// Salsa cache reuse under layout-only project edits +// +// GH #964's acceptance criteria include "layout-only project edits continue to +// reuse unchanged salsa-cached fragments". Stage 3 of that issue is exactly the +// change that could break it, so it is measured here -- with execution records +// rather than memo pointers, because a fragment is DESIGNED to be +// layout-independent and therefore compares equal across a layout edit whether +// or not the compile re-ran. Salsa backdates on that equality and keeps the +// memo address, so every pointer-based check passes either way. +// +// Each test has a control step -- re-syncing the IDENTICAL project must +// re-execute nothing -- so a count is attributable to the edit that follows it. +// +// The criterion held only as a VALUE property until C1b: the fragments were +// identical across a layout edit, but every one of them was recompiled to +// rediscover that. Two coarse salsa edges caused it, and both were inside the +// round trip GH #964 is deleting: +// +// 1. `compile_var_fragment` read the whole `ModelDepGraphResult` +// (`model_dependency_graph`) but used only three runlist-membership bits +// of it. Any variable added to the model changes the runlists, so the +// value changed and every dependent re-executed. Narrowed by the +// `var_runlist_membership` projection, which returns those three bits and +// backdates. +// 2. `lower_var_fragment` / `collect_var_dependencies` read the whole +// `SourceModel::variables` map field to resolve dependency names, so any +// change to the model's variable SET invalidated every fragment through +// that edge too. Narrowed by the `model_variable_by_name` firewall query, +// so a fragment depends on the dependencies it actually looks up. +// +// Fixing edge 1 alone was provably unmeasurable while edge 2 stood, which is +// why they landed together. A THIRD edge (`model_implicit_var_info`, narrowed +// by `model_implicit_var_by_name`) is exercised only by a variable that +// synthesizes an implicit helper, so it has its own test below rather than a +// row here. +// +// **These counts are the gate C1c is measured against, and they are TIGHT +// WITHIN A BOUND that `implicit_helper_add_is_tight_but_module_helper_add_is_not` +// states exactly.** Tight for a plain aux (this test) and for a +// PREVIOUS/INIT-helper-bearing variable; still saturated for a variable that +// instantiates a MODULE, for two reasons neither edge narrowing reaches. Read +// that test before relying on "the counts are tight" as a blanket claim: a +// stage-3 rewrite that reintroduces a model-wide dependency reds here on the +// count, not merely on the value assertions below, but only for the shapes +// these fixtures actually cover. +// --------------------------------------------------------------------------- + +/// A flat model: `probe` (the fragment under test) reads `k`; every other +/// variable is independent of both, so adding, deleting or renaming one is a +/// pure layout change from `probe`'s point of view. +fn cache_probe_project(extra: &[(&str, &str)], keep_other: bool) -> datamodel::Project { + let mut tp = TestProject::new("frag_cache_probe") + .with_sim_time(0.0, 1.0, 1.0) + .scalar_aux("k", "3") + .scalar_aux("probe", "k * 2"); + if keep_other { + tp = tp.scalar_aux("other", "1"); + } + for (name, eqn) in extra { + tp = tp.scalar_aux(name, eqn); + } + tp.build_datamodel() +} + +/// Re-sync `project` onto `prev` and re-assemble, returning the new sync state +/// and every fragment-compiler body entry the re-assembly caused. +fn resync_and_assemble( + db: &mut SimlinDb, + project: &datamodel::Project, + prev: Option<&PersistentSyncState>, +) -> (PersistentSyncState, Vec<(FragmentExecKind, String)>) { + let state = sync_from_datamodel_incremental(db, project, prev); + let source_project = state.to_sync_result().project; + reset_fragment_executions(); + assemble_simulation(db, source_project, "main".to_string()) + .expect("the cache-probe fixture must assemble"); + (state, fragment_executions()) +} + +/// One variable's whole `CompiledVarFragment` (all three phases), for the +/// value-equality half of the layout-edit tests. +fn probe_fragment( + db: &SimlinDb, + state: &PersistentSyncState, + model_name: &str, + var: &str, +) -> crate::compiler::symbolic::CompiledVarFragment { + let sync = state.to_sync_result(); + let model = sync.models[model_name].source; + let source_var = sync.models[model_name].variables[var].source; + compile_var_fragment( + db, + source_var, + model, + sync.project, + ModuleInputSet::empty(db), + ) + .as_ref() + .unwrap_or_else(|| panic!("`{var}` must compile")) + .fragment + .clone() +} + +fn explicit_execs(execs: &[(FragmentExecKind, String)]) -> Vec<&str> { + execs + .iter() + .filter(|(kind, _)| *kind == FragmentExecKind::Explicit) + .map(|(_, name)| name.as_str()) + .collect() +} + +#[test] +fn layout_only_edits_and_fragment_cache_reuse() { + let mut db = SimlinDb::default(); + + let base = cache_probe_project(&[], true); + let state1 = sync_from_datamodel_incremental(&mut db, &base, None); + assemble_simulation(&db, state1.to_sync_result().project, "main".to_string()) + .expect("priming assemble"); + let probe_before = probe_fragment(&db, &state1, "main", "probe"); + + // Control: an identical re-sync changes no input field, so nothing at all + // may re-execute. Without this, any count below could be measuring the + // re-sync rather than the edit. + let (state2, control) = resync_and_assemble(&mut db, &base, Some(&state1)); + assert_eq!( + control, + Vec::new(), + "control: re-syncing the identical project must re-execute no fragment \ + compiler at all" + ); + + // `probe`'s fragment must be VALUE-identical after EVERY edit, not merely + // at the end. This is the property the execution counts cannot express, + // and the one that matters most for stage 3. + // + // Both checks are kept, and the reason is NOT that one is weaker. Since + // C1b narrowed the two coarse edges the counts above are tight, and they + // are in fact the more sensitive of the two: a fragment cannot change + // value without its body re-running, so any layout dependency strong + // enough to move the value has already moved a count. What value equality + // adds is INDEPENDENCE from the counting apparatus -- it holds whatever + // C1c does to the query structure, the recorder's `FragmentExecKind`s, or + // which compiler owns which variable, and it holds whether or not the + // compile re-ran. A rewrite is free to change the counts for a legitimate + // reason; it is never free to change these values. + // + // Asserted after each edit INDIVIDUALLY and deliberately. Checking only + // the end state is not enough: the base and the post-rename project happen + // to hold the same NUMBER of variables here, so a fragment that leaked + // `n_slots` would come back to its original value and a single end-state + // comparison would pass. Only the post-add state (one more variable) has a + // different layout size. A mutation probe that leaked the layout into the + // literal pool is what surfaced that. + // + // `db::incremental_compile_tests::test_ac1_3_ac1_4_fragment_reuse_on_add_remove` + // already asserts this shape, but only on a two-scalar-variable fixture; + // `layout_only_edits_preserve_fragment_values_for_rich_shapes` adds the + // arrayed / module / SCC shapes, where a layout dependency would hide. + #[track_caller] + fn assert_probe_unchanged( + db: &SimlinDb, + state: &PersistentSyncState, + before: &crate::compiler::symbolic::CompiledVarFragment, + after_what: &str, + ) { + assert_eq!( + *before, + probe_fragment(db, state, "main", "probe"), + "`probe`'s fragment changed after {after_what} -- it is no longer \ + layout-independent, so assembly's single resolve step is no longer \ + the only place offsets are decided" + ); + } + + // Edit 1: ADD an unrelated variable. This is the only one of the three + // that changes the model's variable COUNT relative to the baseline. + let added = cache_probe_project(&[("added", "5")], true); + let (state3, add_execs) = resync_and_assemble(&mut db, &added, Some(&state2)); + assert_eq!( + explicit_execs(&add_execs), + vec!["added"], + "adding an unrelated variable must compile ONLY the new variable: no \ + existing fragment's inputs changed, so none may re-execute" + ); + assert_probe_unchanged(&db, &state3, &probe_before, "an unrelated ADD"); + + // Edit 2: DELETE an unrelated variable (`other`). Nothing referenced it, so + // no surviving fragment has a changed input -- not even a re-execution that + // would backdate. + let deleted = cache_probe_project(&[("added", "5")], false); + let (state4, del_execs) = resync_and_assemble(&mut db, &deleted, Some(&state3)); + assert_eq!( + explicit_execs(&del_execs), + Vec::<&str>::new(), + "deleting an unrelated variable must re-execute no fragment at all" + ); + assert_probe_unchanged(&db, &state4, &probe_before, "an unrelated DELETE"); + + // Edit 3: RENAME an unrelated variable (`added` -> `renamed`). A rename is + // a delete plus an add, so exactly the new name compiles. + let renamed = cache_probe_project(&[("renamed", "5")], false); + let (state5, rename_execs) = resync_and_assemble(&mut db, &renamed, Some(&state4)); + assert_eq!( + explicit_execs(&rename_execs), + vec!["renamed"], + "renaming an unrelated variable must compile only the renamed variable" + ); + assert_probe_unchanged(&db, &state5, &probe_before, "an unrelated RENAME"); +} + +/// The value half of the layout-edit contract on the shapes most likely to +/// acquire a layout dependency: an arrayed variable (per-element offsets), a +/// module instance (a `SymbolicModuleDecl` carrying a `SymVarRef`), and a +/// resolved recurrence SCC member (whose combined fragment is built from these +/// per-member ones). +/// +/// Kept separate from the counting test because it needs richer fixtures, and +/// because it is the assertion that survives stage 3 changing the counts. +#[test] +fn layout_only_edits_preserve_fragment_values_for_rich_shapes() { + // (fixture label, builder taking the extra unrelated variables, probe var) + let arrayed = |extra: &[(&str, &str)]| { + let mut tp = TestProject::new("layout_value_arrayed") + .with_sim_time(0.0, 1.0, 1.0) + .named_dimension("region", &["east", "west", "north"]) + .array_with_ranges( + "arr[region]", + vec![("east", "1"), ("west", "2"), ("north", "3")], + ) + .array_aux("probe[region]", "arr[region] * 2"); + for (name, eqn) in extra { + tp = tp.scalar_aux(name, eqn); + } + tp.build_datamodel() + }; + assert_fragment_value_survives_layout_edits( + &arrayed(&[]), + &arrayed(&[("added", "5")]), + "probe", + ); + + let scc = |extra: &[(&str, &str)]| { + let mut tp = TestProject::new("layout_value_scc") + .with_sim_time(0.0, 1.0, 1.0) + .named_dimension("t", &["t1", "t2", "t3"]) + .array_with_ranges( + "ce[t]", + vec![("t1", "1"), ("t2", "ecc[t1] + 1"), ("t3", "ecc[t2] + 1")], + ) + .array_with_ranges( + "ecc[t]", + vec![ + ("t1", "ce[t1] + 1"), + ("t2", "ce[t2] + 1"), + ("t3", "ce[t3] + 1"), + ], + ); + for (name, eqn) in extra { + tp = tp.scalar_aux(name, eqn); + } + tp.build_datamodel() + }; + assert_fragment_value_survives_layout_edits(&scc(&[]), &scc(&[("added", "5")]), "ce"); + + // The module instance: `sub`'s fragment carries a `SymbolicModuleDecl` + // whose `var` is a `SymVarRef`, plus an `EvalModule`. Both are the kind of + // operand a layout dependency would show up in. + assert_fragment_value_survives_layout_edits( + &module_layout_project(false), + &module_layout_project(true), + "sub", + ); +} + +/// `main` holds `src`, a `sub` instance of `producer`, and (when `with_extra`) +/// one unrelated aux -- a pure layout edit from `sub`'s point of view. +fn module_layout_project(with_extra: bool) -> datamodel::Project { + let aux = |ident: &str, equation: &str, can_be_module_input: bool| { + datamodel::Variable::Aux(datamodel::Aux { + ident: ident.to_string(), + equation: datamodel::Equation::Scalar(equation.to_string()), + documentation: String::new(), + units: None, + gf: None, + ai_state: None, + uid: None, + compat: datamodel::Compat { + can_be_module_input, + ..datamodel::Compat::default() + }, + }) + }; + let mut main_vars = vec![ + aux("src", "3", false), + datamodel::Variable::Module(datamodel::Module { + ident: "sub".to_string(), + model_name: "producer".to_string(), + documentation: String::new(), + units: None, + references: vec![datamodel::ModuleReference { + src: "src".to_string(), + dst: "sub.input".to_string(), + }], + ai_state: None, + uid: None, + compat: datamodel::Compat::default(), + }), + ]; + if with_extra { + main_vars.push(aux("added", "5", false)); + } + datamodel::Project { + name: "layout_value_module".to_string(), + sim_specs: datamodel::SimSpecs { + start: 0.0, + stop: 1.0, + dt: datamodel::Dt::Dt(1.0), + save_step: None, + sim_method: datamodel::SimMethod::Euler, + time_units: None, + }, + dimensions: vec![], + units: vec![], + models: vec![ + datamodel::Model { + name: "main".to_string(), + sim_specs: None, + variables: main_vars, + views: vec![], + loop_metadata: vec![], + groups: vec![], + macro_spec: None, + }, + datamodel::Model { + name: "producer".to_string(), + sim_specs: None, + variables: vec![aux("input", "0", true), aux("output", "input * 10", false)], + views: vec![], + loop_metadata: vec![], + groups: vec![], + macro_spec: None, + }, + ], + source: None, + ai_information: None, + } +} + +/// Compile `var`'s fragment before and after a layout-only edit (adding an +/// unrelated variable) on the same incremental database, and assert the value +/// is unchanged. +#[track_caller] +fn assert_fragment_value_survives_layout_edits( + base: &datamodel::Project, + edited: &datamodel::Project, + var: &str, +) { + let mut db = SimlinDb::default(); + let state1 = sync_from_datamodel_incremental(&mut db, base, None); + let before = probe_fragment(&db, &state1, "main", var); + + let state2 = sync_from_datamodel_incremental(&mut db, edited, Some(&state1)); + let after = probe_fragment(&db, &state2, "main", var); + assert_eq!( + before, after, + "`{var}`'s fragment changed when an unrelated variable was added -- it \ + is not layout-independent" + ); + + // ...and back again: deleting the unrelated variable must restore the + // identical fragment, not merely a differently-shaped one. + let state3 = sync_from_datamodel_incremental(&mut db, base, Some(&state2)); + let restored = probe_fragment(&db, &state3, "main", var); + assert_eq!( + before, restored, + "`{var}`'s fragment changed when the unrelated variable was deleted again" + ); +} + +/// `main` reads `sub·output` across a module boundary. `producer_extra` adds a +/// variable to the SUB-model that sorts before `output` (so `output`'s offset +/// inside `producer` moves); `unrelated_extra` adds one to a model nothing +/// instantiates. +fn submodel_layout_project(producer_extra: bool, unrelated_extra: bool) -> datamodel::Project { + let aux = |ident: &str, equation: &str, can_be_module_input: bool| { + datamodel::Variable::Aux(datamodel::Aux { + ident: ident.to_string(), + equation: datamodel::Equation::Scalar(equation.to_string()), + documentation: String::new(), + units: None, + gf: None, + ai_state: None, + uid: None, + compat: datamodel::Compat { + can_be_module_input, + ..datamodel::Compat::default() + }, + }) + }; + + let mut producer_vars = vec![aux("input", "0", true), aux("output", "input * 10", false)]; + if producer_extra { + // Sorts before both, so `compute_layout`'s alphabetical body layout + // pushes `output` from slot 1 to slot 2. + producer_vars.insert(0, aux("aaa", "1", false)); + } + + let mut unrelated_vars = vec![aux("u1", "1", false)]; + if unrelated_extra { + unrelated_vars.push(aux("u2", "2", false)); + } + + datamodel::Project { + name: "frag_submodel_layout".to_string(), + sim_specs: datamodel::SimSpecs { + start: 0.0, + stop: 1.0, + dt: datamodel::Dt::Dt(1.0), + save_step: None, + sim_method: datamodel::SimMethod::Euler, + time_units: None, + }, + dimensions: vec![], + units: vec![], + models: vec![ + datamodel::Model { + name: "main".to_string(), + sim_specs: None, + variables: vec![ + aux("src", "3", false), + datamodel::Variable::Module(datamodel::Module { + ident: "sub".to_string(), + model_name: "producer".to_string(), + documentation: String::new(), + units: None, + references: vec![datamodel::ModuleReference { + src: "src".to_string(), + dst: "sub.input".to_string(), + }], + ai_state: None, + uid: None, + compat: datamodel::Compat::default(), + }), + aux("usesub", "sub.output * 2", false), + ], + views: vec![], + loop_metadata: vec![], + groups: vec![], + macro_spec: None, + }, + datamodel::Model { + name: "producer".to_string(), + sim_specs: None, + variables: producer_vars, + views: vec![], + loop_metadata: vec![], + groups: vec![], + macro_spec: None, + }, + datamodel::Model { + name: "unrelated".to_string(), + sim_specs: None, + variables: unrelated_vars, + views: vec![], + loop_metadata: vec![], + groups: vec![], + macro_spec: None, + }, + ], + source: None, + ai_information: None, + } +} + +/// Every `element_offset` a fragment's opcodes carry for references to +/// `module_var`, across all three phases, in opcode order. +fn cross_module_element_offsets( + fragment: &crate::compiler::symbolic::CompiledVarFragment, + module_var: &str, +) -> Vec { + let phases = [ + fragment.initial_bytecodes.as_ref(), + fragment.flow_bytecodes.as_ref(), + fragment.stock_bytecodes.as_ref(), + ]; + let mut offsets = Vec::new(); + for bc in phases.into_iter().flatten() { + for op in &bc.symbolic.code { + let var = match op { + SymbolicOpcode::LoadVar { var } + | SymbolicOpcode::SymLoadPrev { var } + | SymbolicOpcode::SymLoadInitial { var } + | SymbolicOpcode::LoadSubscript { var } + | SymbolicOpcode::AssignCurr { var } + | SymbolicOpcode::AssignConstCurr { var, .. } + | SymbolicOpcode::BinOpAssignCurr { var, .. } + | SymbolicOpcode::BinOpAssignNext { var, .. } + | SymbolicOpcode::PushVarViewDirect { var, .. } => var, + _ => continue, + }; + if var.name.as_str() == module_var { + offsets.push(var.element_offset); + } + } + } + offsets +} + +/// The ONE layout channel that still reaches into a fragment -- and the only +/// place the value-equality assertions above still have teeth of their own. +/// +/// Since GH #964 a fragment carries no offset of its own model at all, and +/// consulting one is a compile error (`VariableMetadata::offset` is `None` for +/// the model being compiled), so the TYPE now subsumes most of what those +/// assertions were watching for. One channel survives, in the other direction: +/// a cross-module reference `sub·output` lowers to +/// `VarRef { name: sub, element_offset: }`, +/// because the parent's layout has a single entry spanning the whole instance +/// and none for `sub·output`. `Context::submodel_offset_within` reads that +/// offset out of `compute_layout(producer)`, which is already fixed before any +/// parent fragment compiles. +/// +/// So the parent's cached fragment is a function of the SUB-model's layout, +/// and both directions are load-bearing: +/// +/// * growing `producer` ahead of `output` MUST change the parent's fragment, +/// and MUST re-execute it. A cache hit here would serve a stale +/// `element_offset` and the parent would read the wrong slot of the instance +/// -- a silent wrong number, not a failure to compile. +/// * growing a model nothing instantiates must NOT change it. +/// +/// Neither direction was pinned before. The behavior is pre-existing and +/// unchanged by GH #964; what changed is that it is now the only layout input +/// a fragment has, which is what makes it worth its own test. +#[test] +fn parent_fragment_tracks_the_sub_models_layout_and_nothing_else() { + let base = submodel_layout_project(false, false); + let grown_submodel = submodel_layout_project(true, false); + let grown_unrelated = submodel_layout_project(false, true); + + let mut db = SimlinDb::default(); + let state1 = sync_from_datamodel_incremental(&mut db, &base, None); + assemble_simulation(&db, state1.to_sync_result().project, "main".to_string()) + .expect("the sub-model layout fixture must assemble"); + let before = probe_fragment(&db, &state1, "main", "usesub"); + + // Precondition: the fixture really does carry a cross-module reference, so + // a shape change that stopped emitting one fails here rather than making + // the assertions below vacuously true. + assert_eq!( + cross_module_element_offsets(&before, "sub"), + vec![1], + "`usesub` must read `sub` at `output`'s offset inside `producer` \ + (input@0, output@1)" + ); + + // Direction 1: the sub-model's layout shifts under the parent. + let (state2, execs) = resync_and_assemble(&mut db, &grown_submodel, Some(&state1)); + let after_submodel = probe_fragment(&db, &state2, "main", "usesub"); + assert_eq!( + cross_module_element_offsets(&after_submodel, "sub"), + vec![2], + "adding `aaa` to `producer` moves `output` to slot 2, so the parent's \ + cross-module reference must follow it" + ); + assert_ne!( + before, after_submodel, + "the parent's fragment must change when the sub-model's layout shifts \ + beneath it" + ); + // Exactly three fragments re-execute, and the set is the point: `usesub` + // MUST be in it (a cache hit would serve the stale element offset), `sub` + // must be too (its `SymbolicModuleDecl` and `EvalModule` describe an + // instance whose size changed), and `src` -- an ordinary aux in `main` -- + // must not, or the sub-model edit has become a whole-project invalidation. + // `aaa` is the new sub-model variable itself. + assert_eq!( + explicit_execs(&execs), + vec!["aaa", "sub", "usesub"], + "growing the sub-model must re-execute the new variable, the module \ + variable, and the parent's cross-module reader -- and nothing else" + ); + + // ...and back: shrinking the sub-model again restores the identical + // fragment, so the dependence is on the layout rather than on the edit. + let state3 = sync_from_datamodel_incremental(&mut db, &base, Some(&state2)); + assert_eq!( + before, + probe_fragment(&db, &state3, "main", "usesub"), + "removing the sub-model variable must restore the identical parent \ + fragment" + ); + + // Direction 2: a model nothing instantiates grows. `main` and `producer` + // are untouched, so the parent's fragment must be bit-identical AND must + // not recompile. + // + // The execution set is measured, not assumed: exactly `["sub"]`, stable + // across repeated runs. `sub` re-executes because growing the project's + // model set re-keys the interned module-ident context, which is the + // saturation `implicit_helper_add_is_tight_but_module_helper_add_is_not` + // pins with both causes named -- so this test does not re-litigate it. What + // it does pin is that `usesub`, the cross-module READER this test is about, + // is not in the set: its fragment is a function of `producer`'s layout, and + // `producer` did not move. + // + // Value equality alone could not make that claim. It cannot distinguish + // "correctly cached" from "recompiled and happened to agree", which is + // precisely the distinction direction 1 turns on. + let (state4, unrelated_execs) = resync_and_assemble(&mut db, &grown_unrelated, Some(&state3)); + assert_eq!( + explicit_execs(&unrelated_execs), + vec!["sub"], + "growing a model nothing instantiates must not recompile the parent's \ + cross-module reader" + ); + assert_eq!( + before, + probe_fragment(&db, &state4, "main", "usesub"), + "adding a variable to a model nothing instantiates must not change the \ + parent's fragment" + ); +} + +#[test] +fn equation_only_edit_recompiles_only_the_edited_fragment() { + // The contrast case for `layout_only_edits_and_fragment_cache_reuse`: an + // equation edit that does not change the edited variable's dependency set + // leaves every other variable's inputs bit-identical, so only the edited + // fragment recompiles. This is the incrementality the per-variable + // fragment cache actually delivers today, and it is what makes the + // layout-edit result above a *narrower* claim than "the cache does not + // work". + // + // `db::fragment_cache_tests::test_compile_var_fragment_caching` asserts the + // same thing with memo pointer equality, which cannot distinguish "reused" + // from "recompiled to an equal value"; this is the version with evidence. + let mut db = SimlinDb::default(); + + let base = cache_probe_project(&[("independent", "9")], true); + let state1 = sync_from_datamodel_incremental(&mut db, &base, None); + assemble_simulation(&db, state1.to_sync_result().project, "main".to_string()) + .expect("priming assemble"); + + let (state2, control) = resync_and_assemble(&mut db, &base, Some(&state1)); + assert_eq!( + control, + Vec::new(), + "control: re-syncing the identical project must re-execute nothing" + ); + + // Same variable set, same dependency set for every variable -- only + // `independent`'s constant moves, and nothing reads `independent`. + let edited = cache_probe_project(&[("independent", "11")], true); + let (state3, execs) = resync_and_assemble(&mut db, &edited, Some(&state2)); + assert_eq!( + explicit_execs(&execs), + vec!["independent"], + "an equation-only edit to a variable nothing reads must recompile ONLY \ + that variable's fragment" + ); + + // ...but the blast radius is one hop wide, not zero: `lower_var_fragment` + // builds its dependency-granular mini `ModelStage0` by PARSING each + // dependency, so a consumer's fragment depends on its dependencies' + // equation text and not merely on their shape. Editing `k`'s constant + // therefore recompiles `probe` as well. + // + // That one hop is intrinsic to the mini-stage design (track-C invariant 2: + // pointing the mini stage at a whole-project cached stage would make the + // radius the whole project instead), so it is pinned as the CURRENT + // contract, not flagged as a defect. If a later stage widens it beyond one + // hop, this reds. + let mut k_edited = cache_probe_project(&[("independent", "11")], true); + assert_eq!( + k_edited.models[0].variables[0].get_ident(), + "k", + "this edit rewrites slot 0 in place; it must be `k`, or the test is \ + silently measuring a different variable" + ); + k_edited.models[0].variables[0] = datamodel::Variable::Aux(datamodel::Aux { + ident: "k".to_string(), + equation: datamodel::Equation::Scalar("4".to_string()), + documentation: String::new(), + units: None, + gf: None, + ai_state: None, + uid: None, + compat: datamodel::Compat::default(), + }); + let (_state4, k_execs) = resync_and_assemble(&mut db, &k_edited, Some(&state3)); + assert_eq!( + explicit_execs(&k_execs), + vec!["k", "probe"], + "editing `k` recompiles `k` AND its one consumer `probe`, whose mini \ + ModelStage0 re-parses `k`; it must not reach any further" + ); +} + +/// The cache granularity of the OTHER two fragment compilers, measured rather +/// than assumed. +/// +/// `compile_implicit_var_fragment` is not a salsa query at all -- it is a plain +/// function called from the tracked `assemble_module` -- so every implicit +/// (SMOOTH/DELAY/TREND/PREVIOUS/INIT) helper recompiles whenever assembly +/// re-runs, which an equation edit to ANY variable in the model causes. +/// `compile_ltm_var_fragment` IS tracked, per `(from, to)` link. +/// +/// Pinned because stage 3 of GH #964 routes all three emitters through one +/// implementation: if that implementation is a salsa query, these numbers +/// should drop, and if it is a plain function, the explicit path could +/// silently acquire the implicit path's granularity instead. +#[test] +fn implicit_and_ltm_fragment_cache_granularity() { + use salsa::Setter; + + let project_with = |smoothed_input: &str, unrelated: &str| { + TestProject::new("frag_cache_implicit") + .with_sim_time(0.0, 1.0, 1.0) + .scalar_aux("src", smoothed_input) + .scalar_aux("smoothed", "SMTH1(src, 2)") + .scalar_aux("unrelated", unrelated) + .build_datamodel() + }; + + let mut db = SimlinDb::default(); + let base = project_with("3", "1"); + let state1 = sync_from_datamodel_incremental(&mut db, &base, None); + assemble_simulation(&db, state1.to_sync_result().project, "main".to_string()) + .expect("priming assemble"); + + let (state2, control) = resync_and_assemble(&mut db, &base, Some(&state1)); + assert_eq!( + control, + Vec::new(), + "control: re-syncing the identical project must re-execute nothing" + ); + + // Edit a variable the SMTH1 helper does not read. + let edited = project_with("3", "2"); + let (_state3, execs) = resync_and_assemble(&mut db, &edited, Some(&state2)); + assert_eq!( + explicit_execs(&execs), + vec!["unrelated"], + "the explicit path is per-variable: only the edited fragment recompiles" + ); + let implicit: Vec<&str> = execs + .iter() + .filter(|(kind, _)| *kind == FragmentExecKind::Implicit) + .map(|(_, name)| name.as_str()) + .collect(); + assert_eq!( + implicit, + vec!["smoothed#0", "smoothed#1"], + "every implicit helper of the model recompiles on an edit to a variable \ + none of them reads: `compile_implicit_var_fragment` has no cache entry \ + of its own, so its granularity is `assemble_module`'s" + ); + + // The LTM link fragments, on the same shape of edit. + let mut ltm_db = SimlinDb::default(); + let ltm_base = ltm_loop_model(); + let ltm_state1 = sync_from_datamodel_incremental(&mut ltm_db, <m_base, None); + let ltm_project = ltm_state1.to_sync_result().project; + ltm_project.set_ltm_enabled(&mut ltm_db).to(true); + assemble_simulation(<m_db, ltm_project, "main".to_string()).expect("priming LTM assemble"); + + let (ltm_state2, ltm_control) = resync_and_assemble(&mut ltm_db, <m_base, Some(<m_state1)); + assert_eq!( + ltm_control, + Vec::new(), + "control: re-syncing the identical LTM project must re-execute nothing" + ); + + // Step A: change a CONSTANT. A link score's equation is derived from the + // TARGET's equation structure, not from any source's value, so no link + // score's text moves and no LTM fragment recompiles. + let rate_edited = TestProject::new("frag_ltm_loop") + .with_sim_time(0.0, 2.0, 1.0) + .aux("rate", "0.2", None) + .flow("growth", "level * rate", None) + .stock("level", "10", &["growth"], &[], None) + .build_datamodel(); + let (ltm_state3, rate_execs) = + resync_and_assemble(&mut ltm_db, &rate_edited, Some(<m_state2)); + assert_eq!( + ltm_execs_of(&rate_execs), + Vec::<&str>::new(), + "editing a constant moves no link-score equation, so no LTM link \ + fragment recompiles" + ); + + // Step B: change an equation ON the circuit. At least the link score whose + // TARGET moved must recompile; `rate\u{2192}growth` is not emitted in + // exhaustive mode at all, so the whole set is a subset of the circuit's two + // edges. + // + // Only the LOWER BOUND is asserted, because the exact set is + // NONDETERMINISTIC. Over 40 in-process repetitions of exactly this + // sequence, `["growth\u{2192}level", "level\u{2192}growth"]` came up 33 + // times and `["level\u{2192}growth"]` 7 times. + // + // What that nondeterminism is, established by measurement rather than + // inferred: `compile_ltm_var_fragment`'s returned VALUE is equal before and + // after the edit in every repetition, and `link_score_equation_text_shaped` + // backdates correctly in every repetition -- only whether the body re-runs + // varies. Salsa backdates the equal value, so no consumer observes a + // difference and no artifact changes (both `ltm_loop_*` goldens hold across + // repeated runs). It is wasted recompilation of one link fragment, not a + // wrong answer. It also reproduces when the query is demanded DIRECTLY, + // so it is not assembly's map-walk order. + // + // What it is NOT: it is not the `dep_idents` ordering (measured across that + // fix with no effect) and not any value on the shaded query's chain. The + // residual is which recorded dependency reports "maybe changed" given an + // unchanged dependency set, which is a salsa dependency-verification + // question rather than a fragment-compiler one. Asserting the full set + // here would land a flaky test. + let growth_edited = TestProject::new("frag_ltm_loop") + .with_sim_time(0.0, 2.0, 1.0) + .aux("rate", "0.2", None) + .flow("growth", "level * rate * 1", None) + .stock("level", "10", &["growth"], &[], None) + .build_datamodel(); + let (_ltm_state4, growth_execs) = + resync_and_assemble(&mut ltm_db, &growth_edited, Some(<m_state3)); + let recompiled = ltm_execs_of(&growth_execs); + assert!( + recompiled.contains(&"level\u{2192}growth"), + "editing `growth`'s equation must recompile the link score whose TARGET \ + it is; got {recompiled:?}" + ); + assert!( + recompiled + .iter() + .all(|edge| *edge == "level\u{2192}growth" || *edge == "growth\u{2192}level"), + "only the circuit's own two edges have link scores in exhaustive mode, \ + so nothing outside them may recompile; got {recompiled:?}" + ); +} + +/// The THIRD coarse edge (`model_implicit_var_info`), pinned in BOTH +/// directions: narrowed for a plain implicit helper, still saturated for one +/// that instantiates a module. +/// +/// `layout_only_edits_and_fragment_cache_reuse` above adds a plain aux, which +/// synthesizes no implicit variable at all and so never touches this edge. A +/// variable whose equation calls `PREVIOUS` or `INIT` DOES synthesize one, and +/// before the `model_implicit_var_by_name` projection every fragment in the +/// model re-executed for it -- `collect_var_dependencies` read the whole +/// implicit-var map to answer a per-name question. +/// +/// The `SMTH1` half is the honest limit, asserted as the CURRENT CONTRACT so +/// it cannot be quietly over-claimed. Narrowing this edge does not make a +/// module-instantiating helper tight, because two further whole-model +/// dependencies survive on that path and neither is a projection away: +/// +/// 1. `project.models(db)` changes -- the implicit `stdlib⁚smth1` template is +/// spliced into the project's model map, so the map every fragment reads +/// is a different value. +/// 2. `model_module_ident_context` is an INTERNED handle keyed on the model's +/// module-ident set. Adding a module instance grows that set, which mints +/// a NEW interned id, which becomes a new cache key for every variable's +/// `parse_source_variable_with_module_context`. A changed key cannot +/// backdate -- there is no prior memo to compare against -- so this is a +/// cache-key problem, not a value-equality problem, and no projection over +/// the existing query can fix it. +/// +/// If someone fixes (2), this test reds on the `SMTH1` assertion rather than +/// leaving a stale sentence in a doc comment. That is the point of pinning a +/// limitation instead of describing it. +#[test] +fn implicit_helper_add_is_tight_but_module_helper_add_is_not() { + // `probe` reads `k`; the added variable is independent of both, so this is + // a layout-only edit from `probe`'s point of view in every case. + let project_with = |extra: Option<(&str, &str)>| { + let mut tp = TestProject::new("frag_cache_implicit_edge") + .with_sim_time(0.0, 2.0, 1.0) + .scalar_aux("k", "3") + .scalar_aux("probe", "k * 2"); + if let Some((name, eqn)) = extra { + tp = tp.scalar_aux(name, eqn); + } + tp.build_datamodel() + }; + + // ── A PREVIOUS helper: narrowed by `model_implicit_var_by_name`. + let mut db = SimlinDb::default(); + let base = project_with(None); + let state1 = sync_from_datamodel_incremental(&mut db, &base, None); + assemble_simulation(&db, state1.to_sync_result().project, "main".to_string()) + .expect("priming assemble"); + + let (state2, control) = resync_and_assemble(&mut db, &base, Some(&state1)); + assert_eq!( + control, + Vec::new(), + "control: re-syncing the identical project must re-execute nothing" + ); + + // The argument must be an EXPRESSION, not a bare variable. A direct scalar + // `PREVIOUS(k, 0)` compiles straight to the `LoadPrev` opcode and + // synthesizes NO implicit variable at all, so it never touches this edge -- + // it passes whether or not the projection exists, which makes it a + // vacuous fixture. (A mutation probe reintroducing the whole-map read is + // how that was caught: it stayed green on the bare-variable form and reds + // on this one.) `PREVIOUS(k * 2, 0)` is rewritten through a synthesized + // scalar helper aux, which is the shape that changes + // `model_implicit_var_info` without instantiating any module. + let with_prev = project_with(Some(("lagged", "PREVIOUS(k * 2, 0)"))); + let (_state3, prev_execs) = resync_and_assemble(&mut db, &with_prev, Some(&state2)); + assert_eq!( + explicit_execs(&prev_execs), + vec!["lagged"], + "adding a PREVIOUS-helper-bearing variable must compile only the new \ + variable: the implicit-var map changed, but no existing fragment asks \ + it about a name whose answer moved" + ); + + // ── An SMTH1 helper: still saturated, for the two reasons above. + let mut db2 = SimlinDb::default(); + let base2 = project_with(None); + let s1 = sync_from_datamodel_incremental(&mut db2, &base2, None); + assemble_simulation(&db2, s1.to_sync_result().project, "main".to_string()) + .expect("priming assemble"); + + let (s2, control2) = resync_and_assemble(&mut db2, &base2, Some(&s1)); + assert_eq!( + control2, + Vec::new(), + "control: re-syncing the identical project must re-execute nothing" + ); + + let with_smth = project_with(Some(("smoothed", "SMTH1(k, 2)"))); + let (_s3, smth_execs) = resync_and_assemble(&mut db2, &with_smth, Some(&s2)); + // `delay_time`/`flow`/`initial_value`/`input`/`output` are the spliced + // `stdlib⁚smth1` template's own variables, compiling for the first time -- + // those are legitimately new work. `k` and `probe` are not: they are the + // saturation this test pins. + assert_eq!( + explicit_execs(&smth_execs), + vec![ + "delay_time", + "flow", + "initial_value", + "input", + "k", + "output", + "probe", + "smoothed" + ], + "CURRENT CONTRACT, not an aspiration: adding a MODULE-instantiating \ + helper still re-executes every PRE-EXISTING fragment (`k`, `probe`), \ + because `project.models` gains the spliced `stdlib⁚smth1` template \ + and the interned `ModuleIdentContext` key every parse is memoized \ + under changes. If this assertion reds because `k` and `probe` \ + dropped out, someone fixed one of those two and this pin should be \ + tightened to match" + ); +} + +fn ltm_execs_of(execs: &[(FragmentExecKind, String)]) -> Vec<&str> { + execs + .iter() + .filter(|(kind, _)| *kind == FragmentExecKind::Ltm) + .map(|(_, name)| name.as_str()) + .collect() +} + +/// A fragment carrying more than one temp must compile to a byte-identical +/// `PerVarBytecodes` every time, on independent databases. +/// +/// The single-fixture `assert_temp_sizes_ordered` check inside +/// `assert_fragment_fixture` is a 50/50 detector on a two-temp fragment: a +/// two-entry `HashMap` yields its keys in the right order about half the time, +/// and each `HashMap` gets a fresh hash key from the thread-local counter. This +/// test repeats the compile on independent databases so an unordered +/// `temp_sizes` is caught with probability `1 - 2^-N` instead. +/// +/// What is at stake is not the rendering: `PerVarBytecodes` is a salsa-cached +/// value with a derived `PartialEq`, so an order flip makes an identical +/// fragment compare unequal, salsa stops backdating, and the compiled artifact +/// stops being reproducible run to run (GH #595's class). +#[test] +fn multi_temp_fragment_is_byte_identical_across_fresh_databases() { + const REPEATS: usize = 12; + + let dm = array_ops_model(); + let compile_combo = || { + let db = SimlinDb::default(); + let source_project = sync_from_datamodel(&db, &dm).project; + let model = *source_project.models(&db).get("main").unwrap(); + let combo = source_project.models(&db)["main"].variables(&db)["combo"]; + compile_var_fragment( + &db, + combo, + model, + source_project, + ModuleInputSet::empty(&db), + ) + .as_ref() + .expect("`combo` must compile") + .fragment + .flow_bytecodes + .clone() + .expect("`combo` must have a flow fragment") + }; + + let first = compile_combo(); + assert_eq!( + first.temp_sizes.len(), + 2, + "the fixture must keep MORE THAN ONE temp in a single fragment, or this \ + test cannot observe an ordering at all" + ); + for i in 1..REPEATS { + let again = compile_combo(); + assert_eq!( + first, again, + "compile #{i} of the same variable on a fresh database produced a \ + different fragment; a salsa-cached value that is not a function of \ + its inputs defeats backdating and makes the artifact irreproducible" + ); + } +} diff --git a/src/simlin-engine/src/db/fragment_compile.rs b/src/simlin-engine/src/db/fragment_compile.rs index 0d837d6b4..5a18e47a3 100644 --- a/src/simlin-engine/src/db/fragment_compile.rs +++ b/src/simlin-engine/src/db/fragment_compile.rs @@ -5,13 +5,13 @@ //! Per-variable bytecode emission: the *emission half* of per-variable //! compilation. The *lowering half* (parse + lower to `Vec`) lives in //! the sibling `db/var_fragment.rs` (`lower_var_fragment`); this module -//! consumes that and emits + symbolizes the per-phase bytecode. +//! consumes that and emits the per-phase symbolic bytecode. //! //! `compile_var_fragment` is the salsa-tracked per-variable fragment //! compiler for explicit variables. `compile_implicit_var_fragment` / //! `compile_implicit_var_phase_bytecodes` do the same for the implicit //! (SMOOTH/DELAY/TREND) helpers, sharing the `lower_implicit_var` -//! parent->implicit->parse->lower prefix. The compile+symbolize tail and the +//! parent->implicit->parse->lower prefix. The emission tail and the //! production element-graph source (`compile_phase_to_per_var_bytecodes`, //! `var_phase_symbolic_fragment_prod`) live in the sibling `db/assemble.rs`. @@ -22,6 +22,85 @@ use salsa::Accumulator; use super::*; use crate::common::{Canonical, Ident}; +// Test-only per-thread record of which fragment-compiler bodies ran. +// +// Pointer equality of a memo does NOT prove a query body did not run: salsa +// backdates a re-executed query whose value compares equal and keeps the memo +// address (the trap `db::stages` documents at length). For the fragment +// compilers that matters even more than it did there, because a fragment is +// *designed* to be layout-independent -- so a layout-only edit produces an +// EQUAL fragment whether or not the expensive compile re-ran, and every +// pointer-based test passes either way. Recording each body entry, with the +// name it ran for, is the only evidence that separates "reused the memo" from +// "recompiled it and found it equal". +// +// Names, not just counts: the acceptance criterion under test is per-variable +// ("an *unchanged* fragment is reused"), so an aggregate count cannot say +// whether the one re-execution was the edited variable or an unrelated one. +// +// Thread-local rather than a global atomic, for the same reasons as +// `db::stages`: no lock, and a parallel test run cannot charge one test's +// work to another. The same caveat applies -- the record happens INSIDE the +// body, on whatever thread salsa ran it, so anyone introducing query +// parallelism here must move this to a shared atomic in the same change. +// `reset_fragment_executions()` at the start of a measured region is what +// isolates a test under `--test-threads=1`, where every test shares a thread. +#[cfg(test)] +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub(crate) enum FragmentExecKind { + /// `compile_var_fragment` -- salsa-tracked, one cache entry per + /// `(variable, model, project, module inputs)`. + Explicit, + /// `compile_implicit_var_fragment` -- NOT salsa-tracked. It is a plain + /// function called from the tracked `assemble_module`, so its results are + /// cached only at whole-module granularity. + Implicit, + /// `compile_ltm_var_fragment` -- salsa-tracked, keyed by `(from, to)` link. + Ltm, +} + +#[cfg(test)] +thread_local! { + /// `None` = not recording. Recording is armed by `reset_fragment_executions` + /// and disarmed by `fragment_executions`, so that the ~5k tests that never + /// measure anything pay nothing and cannot accumulate an unbounded log on a + /// libtest worker thread they happen to share with a measuring test. + static FRAGMENT_EXECUTIONS: std::cell::RefCell>> = + const { std::cell::RefCell::new(None) }; +} + +/// Start (or restart) recording on this thread, discarding anything already +/// recorded (test-only). Call it after the fixture is built and primed, so +/// setup work is not charged to the measured region. +#[cfg(test)] +pub(crate) fn reset_fragment_executions() { + FRAGMENT_EXECUTIONS.with(|c| *c.borrow_mut() = Some(Vec::new())); +} + +/// Stop recording and return every body entry since the last reset, sorted so +/// the result is comparable regardless of the order assembly happened to walk +/// its maps in (test-only). Panics if recording was never armed, since an empty +/// answer from an unarmed recorder would read as "nothing recompiled". +#[cfg(test)] +pub(crate) fn fragment_executions() -> Vec<(FragmentExecKind, String)> { + let mut execs = FRAGMENT_EXECUTIONS + .with(|c| c.borrow_mut().take()) + .expect("fragment_executions() without a preceding reset_fragment_executions()"); + execs.sort(); + execs +} + +/// Record one fragment-compiler body entry, if this thread is recording +/// (test-only). +#[cfg(test)] +pub(crate) fn note_fragment_execution(kind: FragmentExecKind, name: &str) { + FRAGMENT_EXECUTIONS.with(|c| { + if let Some(log) = c.borrow_mut().as_mut() { + log.push((kind, name.to_string())); + } + }); +} + #[salsa::tracked(returns(ref))] pub fn compile_var_fragment<'db>( db: &'db dyn Db, @@ -33,12 +112,15 @@ pub fn compile_var_fragment<'db>( use crate::compiler::symbolic::{CompiledVarFragment, PerVarBytecodes}; use crate::db::var_fragment::{LoweredVarFragment, lower_var_fragment}; + #[cfg(test)] + note_fragment_execution(FragmentExecKind::Explicit, var.ident(db)); + let var_ident = var.ident(db).clone(); let var_ident_canonical: Ident = Ident::new(&var_ident); // The interned input set stores the sorted canonical names; the plain - // lowering helpers (`lower_var_fragment`/`build_caller_module_refs`) still - // take `&[String]`, so read it back as a slice. + // lowering helper (`lower_var_fragment`) still takes `&[String]`, so read + // it back as a slice. let module_input_names = module_inputs.names(db); // Caller-owned, lowering-independent context (built only from @@ -67,7 +149,7 @@ pub fn compile_var_fragment<'db>( &inputs, ); - let (unit_diags, per_phase_lowered, tables, offsets, rmap, mini_offset) = match lowered { + let (unit_diags, per_phase_lowered, tables, var_sizes) = match lowered { LoweredVarFragment::Fatal { unit_diags, fatal_diags, @@ -87,17 +169,8 @@ pub fn compile_var_fragment<'db>( unit_diags, per_phase_lowered, tables, - offsets, - rmap, - mini_offset, - } => ( - unit_diags, - per_phase_lowered, - tables, - offsets, - rmap, - mini_offset, - ), + var_sizes, + } => (unit_diags, per_phase_lowered, tables, var_sizes), }; // Malformed-unit diagnostics are non-fatal: record them and continue. @@ -105,48 +178,59 @@ pub fn compile_var_fragment<'db>( CompilationDiagnostic(diag).accumulate(db); } - // Determine which runlists this variable belongs to - let dep_graph = model_dependency_graph(db, model, project, module_inputs); + // Which runlists this variable belongs to, read through the three-bit + // projection rather than the whole `ModelDepGraphResult`: the projection + // backdates when this variable's membership is unchanged, so an unrelated + // variable being added, deleted or renamed no longer invalidates every + // fragment in the model (GH #964). + let membership = var_runlist_membership(db, var, model, project, module_inputs); let is_stock = var.kind(db) == SourceVariableKind::Stock; let is_module = var.kind(db) == SourceVariableKind::Module; let is_module_input = inputs.contains(&var_ident_canonical); - let module_refs = crate::db::var_fragment::build_caller_module_refs( - db, - var, - model, - project, - module_input_names, + // The phase-invariant emission context, built once per variable and + // borrowed by every phase (up to three). All three fragment emitters -- + // this one, the implicit-helper one, and both LTM ones -- go through the + // same `compile_phase_to_per_var_bytecodes`. + let base_ctx = fragment_emit_ctx( + &model_name_ident, + &inputs, + &var_sizes, + &tables, + converted_dims, + dim_context, ); - - // Compile for each phase and symbolize. The closure now delegates to - // the factored `compile_phase_to_per_var_bytecodes` so the SCC - // element-graph builder reuses the EXACT production compile+symbolize - // path (no re-derivation); the per-variable production behavior is - // byte-identical to the former inline closure (same minimal `Module`, - // same temp extraction, same symbolization, same `None` arms). let compile_phase = |exprs: &[crate::compiler::Expr]| -> Option { - compile_phase_to_per_var_bytecodes( - exprs, - &offsets, - &rmap, - &tables, - &module_refs, - mini_offset, - converted_dims, - dim_context, - &model_name_ident, - &var_ident_canonical, - &inputs, - ) + compile_phase_to_per_var_bytecodes(&base_ctx, exprs) }; - // Runlists use canonical names, so compare with the canonical form. - let var_ident_str = var_ident_canonical.as_str().to_string(); - // Accumulate a diagnostic when per-variable compilation (Var::new) // fails. Without this, errors like DoesNotExist (unknown dependency) // are silently dropped and never appear in collect_all_diagnostics. + // + // KNOWN LOSS, deliberately not fixed here: `err.details` is dropped. + // `EquationError` is `{start, end, code}` with no message field, so every + // message a `Var::new` failure writes -- including the one naming the + // stock in `compiler::check_stock_updates_are_emittable` -- reaches the + // user only as its `ErrorCode`. The variable name still rides on the + // `Diagnostic`, so the report stays attributable; it is the *reason* that + // is lost, not the *location*. + // + // The obvious fix is wrong. Switching to `DiagnosticError::Model(err)` + // does carry `details`, but `errors.rs` treats the two variants + // differently on purpose: the `Equation` arm produces + // `FormattedErrorKind::Variable`, names the variable in the summary, and + // -- via `format_diagnostic_with_datamodel` -> `format_equation_error` -- + // enriches the message with a source snippet from the equation text. The + // `Model` arm produces `FormattedErrorKind::Model`, drops the variable + // from the summary, and gets no snippet. Trading a per-variable + // diagnostic for a model-level one to gain a message is a net regression + // of the user-facing surface (pinned by + // `db::diagnostic_tests::test_compile_var_fragment_per_phase_var_new_failure`). + // + // The right fix is to give `EquationError` a details field, which is 48 + // construction sites across 20 files and the type the FFI error surface + // is built on -- its own change, not a rider on this one. let accumulate_var_compile_error = |err: &crate::Error| { CompilationDiagnostic(Diagnostic { model: model.name(db).clone(), @@ -162,7 +246,7 @@ pub fn compile_var_fragment<'db>( }; // Initial phase: stocks and their deps get compiled with is_initial=true - let initial_bytecodes = if dep_graph.runlist_initials.contains(&var_ident_str) { + let initial_bytecodes = if membership.initials { match &per_phase_lowered.initial { Ok(var_result) => compile_phase(&var_result.ast), Err(err) => { @@ -179,8 +263,7 @@ pub fn compile_var_fragment<'db>( // AssignCurr in the flows phase to propagate the parent-provided value // each timestep (matching the monolithic path's `instantiation.contains(id) // || !var.is_stock()` filter). - let in_flows_runlist = - (!is_stock || is_module_input) && dep_graph.runlist_flows.contains(&var_ident_str); + let in_flows_runlist = (!is_stock || is_module_input) && membership.flows; let flow_bytecodes = if in_flows_runlist { match &per_phase_lowered.noninitial { Ok(var_result) => compile_phase(&var_result.ast), @@ -200,8 +283,6 @@ pub fn compile_var_fragment<'db>( let flow_invariance = if in_flows_runlist { crate::db::assemble::compute_flow_invariance_support( &per_phase_lowered.noninitial, - &offsets, - &model_name_ident, &var_ident_canonical, ) } else { @@ -209,18 +290,17 @@ pub fn compile_var_fragment<'db>( }; // Stock phase: stocks and modules get compiled with is_initial=false - let stock_bytecodes = - if (is_stock || is_module) && dep_graph.runlist_stocks.contains(&var_ident_str) { - match &per_phase_lowered.noninitial { - Ok(var_result) => compile_phase(&var_result.ast), - Err(err) => { - accumulate_var_compile_error(err); - None - } + let stock_bytecodes = if (is_stock || is_module) && membership.stocks { + match &per_phase_lowered.noninitial { + Ok(var_result) => compile_phase(&var_result.ast), + Err(err) => { + accumulate_var_compile_error(err); + None } - } else { - None - }; + } + } else { + None + }; Some(VarFragmentResult { fragment: CompiledVarFragment { @@ -376,6 +456,21 @@ pub(crate) fn compile_implicit_var_fragment( ) -> Option { use crate::compiler::symbolic::CompiledVarFragment; + // Recorded at body entry (before the name is even resolved), keyed by the + // parent variable and the implicit index -- the identity this compiler is + // called with. Recording after `lower_implicit_var` would silently omit + // every entry that failed to lower, which is exactly the work a caching + // claim needs to account for. + #[cfg(test)] + note_fragment_execution( + FragmentExecKind::Implicit, + &format!( + "{}#{}", + meta.parent_source_var.ident(db), + meta.index_in_parent + ), + ); + // The implicit var's canonical name (the runlist-gate key). Resolve it // through the shared prefix so this and the per-phase compile agree on // the name by construction. `None` here is the same loud-safe signal @@ -439,7 +534,7 @@ pub(crate) fn compile_implicit_var_fragment( }) } -/// Build the mini-layout context for one implicit variable and compile a +/// Build the minimal symbol table for one implicit variable and compile a /// single phase (`is_initial`) to symbolic `PerVarBytecodes`. NOT a tracked /// function -- the parent variable's parse result already provides salsa /// caching. @@ -453,18 +548,17 @@ pub(crate) fn compile_implicit_var_fragment( /// the element-graph accessor's bytecode is byte-identical to the /// production fragment by construction (DRY -- "single shared relation, /// never re-derive"). The shared `parent → implicit → parse → lower` -/// prefix is `lower_implicit_var`; the shared compile+symbolize tail is +/// prefix is `lower_implicit_var`; the shared emission tail is /// `compile_phase_to_per_var_bytecodes` (the exact function the real-var /// arm of `var_phase_symbolic_fragment_prod` and `compile_var_fragment` -/// use). The mini-layout/metadata/dep-collection glue between them is -/// intrinsic to the implicit-var shape (the `meta.is_module` branch, the -/// body-relative mini-layout, the dep-stub/sub-model collection) and is not -/// separately extractable without restructuring this function. +/// use). The metadata/dep-collection glue between them is intrinsic to the +/// implicit-var shape (the `meta.is_module` branch, the dep-stub/sub-model +/// collection) and is not separately extractable without restructuring this +/// function. /// /// Loud-safe `None` (never panics): the shared prefix failed (absent /// implicit index / equation errors), a graphical-function table failed to -/// build, the phase's `Var::new` errored, or `Module::compile()` / -/// symbolization failed -- exactly the original closure's `None` arms. +/// build, the phase's `Var::new` errored, or codegen failed. #[allow(clippy::too_many_arguments)] pub(crate) fn compile_implicit_var_phase_bytecodes( db: &dyn Db, @@ -474,8 +568,6 @@ pub(crate) fn compile_implicit_var_phase_bytecodes( module_input_names: &[String], is_initial: bool, ) -> Option { - use crate::compiler::symbolic::{ReverseOffsetMap, VariableLayout}; - let module_ident_context = model_module_ident_context(db, model, project, module_input_names.to_vec()); @@ -510,15 +602,11 @@ pub(crate) fn compile_implicit_var_phase_bytecodes( // Arena for sub-model stub variables allocated by build_submodel_metadata let arena = bumpalo::Bump::new(); - // The mini-layout is always body-relative (offset 0). The implicit - // globals (time/dt/initial_time/final_time) are NOT inserted: they - // lower to `LoadGlobalVar` at fixed absolute slots, never through this - // metadata/rmap, so the symbolic fragment is role-independent. The root - // +IMPLICIT_VAR_COUNT shift is applied later in `assemble_module` via - // `VariableLayout::root_shifted`. + // The minimal symbol table for this helper: itself plus its dependencies, + // carrying no offsets. See `lower_var_fragment` for why a fragment must not + // know where its own model's variables live. let mut mini_metadata: HashMap, crate::compiler::VariableMetadata<'_>> = HashMap::new(); - let mut mini_offset = 0; let project_models = project.models(db); let self_size = if meta.is_module { @@ -535,19 +623,18 @@ pub(crate) fn compile_implicit_var_phase_bytecodes( // An *arrayed* implicit helper (the GH #541 bare-arrayed-PREVIOUS case) // occupies one slot per element; `meta.size` is its element count (1 // for the scalar helpers that are every other implicit var). The - // mini-layout self-size MUST match the `compute_layout` allocation, or - // the helper's per-element writes spill into the next variable's slots. + // recorded size MUST match the `compute_layout` allocation, or the + // helper's per-element writes spill into the next variable's slots. meta.size }; mini_metadata.insert( var_ident_canonical.clone(), crate::compiler::VariableMetadata { - offset: mini_offset, + offset: None, size: self_size, var: &lowered, }, ); - mini_offset += self_size; // Implicit vars' deps are always explicit vars in the same model (or other implicit vars) // Keep dependency context conservative for implicit vars as well: both @@ -598,7 +685,6 @@ pub(crate) fn compile_implicit_var_phase_bytecodes( let implicit_info = model_implicit_var_info(db, model, project); let all_names: Vec<&String> = all_dep_names.iter().chain(extra_dep_names.iter()).collect(); let mut dep_variables: Vec<(Ident, crate::variable::Variable, usize)> = Vec::new(); - let mut extra_module_refs: HashMap, crate::vm::ModuleKey> = HashMap::new(); let mut extra_submodels: HashMap = HashMap::new(); for dep_name in &all_names { @@ -647,15 +733,11 @@ pub(crate) fn compile_implicit_var_phase_bytecodes( ident: module_ident.clone(), model_name: Ident::new(mod_model_name), units: None, - inputs: module_inputs.clone(), + inputs: module_inputs, errors: vec![], unit_errors: vec![], }; - dep_variables.push((module_ident.clone(), mod_var, sub_size)); - - let input_set: BTreeSet> = - module_inputs.iter().map(|mi| mi.dst.clone()).collect(); - extra_module_refs.insert(module_ident, (Ident::new(mod_model_name), input_set)); + dep_variables.push((module_ident, mod_var, sub_size)); if let Some(sub_model) = project_models.get(sub_canonical.as_ref()) { extra_submodels.insert(mod_model_name.to_string(), *sub_model); @@ -696,15 +778,11 @@ pub(crate) fn compile_implicit_var_phase_bytecodes( ident: module_ident.clone(), model_name: Ident::new(im_model_name), units: None, - inputs: module_inputs.clone(), + inputs: module_inputs, errors: vec![], unit_errors: vec![], }; - dep_variables.push((module_ident.clone(), mod_var, sub_size)); - - let input_set: BTreeSet> = - module_inputs.iter().map(|mi| mi.dst.clone()).collect(); - extra_module_refs.insert(module_ident, (Ident::new(im_model_name), input_set)); + dep_variables.push((module_ident, mod_var, sub_size)); if let Some(sub_model) = project_models.get(sub_canonical.as_ref()) { extra_submodels.insert(im_model_name.to_string(), *sub_model); @@ -788,12 +866,11 @@ pub(crate) fn compile_implicit_var_phase_bytecodes( mini_metadata.insert( dep_ident.clone(), crate::compiler::VariableMetadata { - offset: mini_offset, + offset: None, size: *dep_size, var: dep_var, }, ); - mini_offset += dep_size; } } @@ -807,11 +884,6 @@ pub(crate) fn compile_implicit_var_phase_bytecodes( build_submodel_metadata(&arena, db, *sub_model, project, &mut all_metadata); } - let mini_layout = - crate::compiler::symbolic::layout_from_metadata(&all_metadata, &model_name_ident) - .unwrap_or_else(|_| VariableLayout::new(HashMap::new(), 0)); - let rmap = ReverseOffsetMap::from_layout(&mini_layout); - let mut tables: HashMap, Vec> = HashMap::new(); { let gf_tables = lowered.tables(); @@ -851,47 +923,33 @@ pub(crate) fn compile_implicit_var_phase_bytecodes( } let inputs = canonical_module_input_set(module_input_names); - let (module_models, mut module_refs) = if meta.is_module { - let mm = model_module_map(db, model, project).clone(); - - // Build module_refs from the implicit var's datamodel::Module references, - // stripping the module ident prefix from dst (matching compile_var_fragment - // and enumerate_module_instances_inner). - let mut refs: HashMap, crate::vm::ModuleKey> = HashMap::new(); + let module_models = if meta.is_module { + // A module-typed implicit helper compiles an `EvalModule`, so its + // sub-model's variables must be in the metadata map for the + // reference offsets to resolve. if let datamodel::Variable::Module(dm_module) = implicit_dm_var { - let input_prefix = format!("{implicit_name}\u{00B7}"); - let input_set: BTreeSet> = dm_module - .references - .iter() - .filter_map(|mr| { - let dst_canonical = canonicalize(&mr.dst); - let bare = dst_canonical.strip_prefix(&input_prefix)?; - Some(Ident::new(bare)) - }) - .collect(); - refs.insert( - var_ident_canonical.clone(), - (Ident::new(&dm_module.model_name), input_set), - ); - - // Populate sub-model metadata let sub_canonical = canonicalize(&dm_module.model_name); if let Some(sub_model) = project_models.get(sub_canonical.as_ref()) { build_submodel_metadata(&arena, db, *sub_model, project, &mut all_metadata); } } - (mm, refs) + model_module_map(db, model, project).clone() } else { - (HashMap::new(), HashMap::new()) + HashMap::new() }; - module_refs.extend(extra_module_refs); + + // Reference extents in the per-variable form BOTH halves borrow: lowering's + // ELM MAP fold below, and the emission context further down. + let var_sizes: PerVarSizes = + crate::compiler::whole_variable_extents(&all_metadata, &model_name_ident); let core = crate::compiler::ContextCore { dimensions: converted_dims, dimensions_ctx: dim_context, model_name: &model_name_ident, metadata: &all_metadata, + var_sizes: &var_sizes, module_models: &module_models, inputs: &inputs, }; @@ -902,35 +960,13 @@ pub(crate) fn compile_implicit_var_phase_bytecodes( ) .ok()?; - // Offsets in the per-variable form `compile_phase_to_per_var_bytecodes` - // expects, built from the mini-layout `all_metadata` exactly as the - // former inline `compile_phase` closure built them (so the shared - // compile+symbolize tail is byte-identical to the prior per-implicit - // behavior -- this replaces the verbatim-duplicate closure with the - // single shared relation). - let offsets: PerVarOffsetMap = all_metadata - .iter() - .map(|(k, v)| { - ( - k.clone(), - v.iter() - .map(|(vk, vm)| (vk.clone(), (vm.offset, vm.size))) - .collect(), - ) - }) - .collect(); - - compile_phase_to_per_var_bytecodes( - &var.ast, - &offsets, - &rmap, + let base_ctx = fragment_emit_ctx( + &model_name_ident, + &inputs, + &var_sizes, &tables, - &module_refs, - mini_offset, converted_dims, dim_context, - &model_name_ident, - &var_ident_canonical, - &inputs, - ) + ); + compile_phase_to_per_var_bytecodes(&base_ctx, &var.ast) } diff --git a/src/simlin-engine/src/db/fragment_determinism_tests.rs b/src/simlin-engine/src/db/fragment_determinism_tests.rs new file mode 100644 index 000000000..5796bd03a --- /dev/null +++ b/src/simlin-engine/src/db/fragment_determinism_tests.rs @@ -0,0 +1,289 @@ +// Copyright 2026 The Simlin Authors. All rights reserved. +// Use of this source code is governed by the Apache License, +// Version 2.0, that can be found in the LICENSE file. + +//! Compilation must be a function of its inputs. +//! +//! Every test here compiles the SAME model on independent, freshly-built +//! databases and asserts the result is byte-identical. That is not a style +//! preference: a compiled fragment is a salsa-cached value with a *derived* +//! `PartialEq`, so a field whose order comes from `HashMap` iteration makes two +//! identical compiles compare unequal. Salsa then stops backdating (every +//! downstream consumer re-executes) and the compiled artifact stops being +//! reproducible run to run. Neither symptom shows up in a numeric result, which +//! is how every defect pinned here survived. +//! +//! Repetition rather than a single comparison, because these are probabilistic +//! detectors: Rust's `RandomState` draws a fresh key per `HashMap`, so a +//! two-entry map comes out in the "right" order about half the time. `REPEATS` +//! runs make a miss `2^-(REPEATS-1)`. +//! +//! Deliberately independent of `db::fragment_char_tests`: these pin the +//! determinism *fixes*, the characterization suite pins *behavior*, and the two +//! must be reviewable (and revertable) apart from each other. + +use crate::compiler::symbolic::PerVarBytecodes; +use crate::datamodel; +use crate::db::{ + ModuleInputSet, SimlinDb, assemble_simulation, compile_var_fragment, sync_from_datamodel, +}; +use crate::test_common::TestProject; + +/// Independent compiles per assertion. Twelve makes a missed ordering defect a +/// 1-in-2048 event while costing a few milliseconds. +const REPEATS: usize = 12; + +/// Compile one variable's flow-phase fragment on a brand-new database. +fn compile_flow_fragment(dm: &datamodel::Project, var: &str) -> PerVarBytecodes { + let db = SimlinDb::default(); + let source_project = sync_from_datamodel(&db, dm).project; + let model = *source_project + .models(&db) + .get("main") + .expect("fixture must have a `main` model"); + let source_var = source_project.models(&db)["main"].variables(&db)[var]; + compile_var_fragment( + &db, + source_var, + model, + source_project, + ModuleInputSet::empty(&db), + ) + .as_ref() + .unwrap_or_else(|| panic!("`{var}` must compile")) + .fragment + .flow_bytecodes + .clone() + .unwrap_or_else(|| panic!("`{var}` must have a flow fragment")) +} + +#[track_caller] +fn assert_fragment_stable(dm: &datamodel::Project, var: &str, what: &str) { + let first = compile_flow_fragment(dm, var); + for i in 1..REPEATS { + let again = compile_flow_fragment(dm, var); + assert_eq!( + first, again, + "compile #{i} of `{var}` on a fresh database produced a different \ + fragment; {what} is not a function of the query's inputs, which \ + defeats salsa backdating and makes the compiled artifact \ + irreproducible" + ); + } +} + +/// A fragment carrying more than one temp must be byte-identical every time. +/// +/// `temp_sizes` was built straight out of a `HashMap`, so its `(temp_id, size)` +/// vector came out in hash order. `FragmentMerger::absorb` folds those entries +/// order-independently, so no bytecode and no result ever changed -- only the +/// derived `PartialEq` on the cached value. Fixed by +/// `db::assemble::temp_sizes_by_id`. +/// +/// The fixture needs TWO array-producing builtins in ONE equation; a single +/// temp cannot express an ordering. +#[test] +fn multi_temp_fragment_is_stable_across_fresh_databases() { + let dm = TestProject::new("determinism_multi_temp") + .with_sim_time(0.0, 1.0, 1.0) + .named_dimension("region", &["east", "west", "north"]) + .array_with_ranges( + "arr[region]", + vec![("east", "30"), ("west", "10"), ("north", "20")], + ) + .array_aux( + "combo[region]", + "VECTOR SORT ORDER(arr[region], 1) + RANK(arr[region], 1)", + ) + .build_datamodel(); + + let probe = compile_flow_fragment(&dm, "combo"); + assert_eq!( + probe.temp_sizes.len(), + 2, + "the fixture must put MORE THAN ONE temp in a single fragment, or this \ + test cannot observe an ordering at all" + ); + assert_fragment_stable(&dm, "combo", "`temp_sizes`"); +} + +/// A fragment referencing more than one table-bearing variable must be +/// byte-identical every time. +/// +/// `Compiler::new` laid out `graphical_functions` -- and therefore every +/// `base_gf` operand its `Lookup`/`LookupArray` opcodes carry -- by iterating +/// `module.tables`, a `HashMap`. Two tables in one fragment meant two different +/// (each self-consistent, each numerically correct) bytecodes across runs. It +/// reaches shipped models: `test/metasd/theil-statistics/Theil_2011.mdl` +/// compiles a fragment holding `["dummy_data", "dummy_simulation"]`. +/// +/// Two lookup-only tables plus one consumer reading both is the minimal shape. +#[test] +fn multi_table_fragment_is_stable_across_fresh_databases() { + let dm = TestProject::new("determinism_multi_table") + .with_sim_time(0.0, 1.0, 1.0) + .aux_with_gf("first_table", "", two_point_gf(1.0, 2.0)) + .aux_with_gf("second_table", "", two_point_gf(10.0, 20.0)) + .scalar_aux( + "consumer", + "LOOKUP(first_table, time) + LOOKUP(second_table, time)", + ) + .build_datamodel(); + + let probe = compile_flow_fragment(&dm, "consumer"); + assert_eq!( + probe.graphical_functions.len(), + 2, + "the fixture must put MORE THAN ONE graphical function in a single \ + fragment, or this test cannot observe an ordering at all" + ); + assert_fragment_stable(&dm, "consumer", "the `graphical_functions` layout"); +} + +/// The whole assembled simulation must be byte-identical across fresh +/// databases, not merely each fragment. +/// +/// The per-fragment tests above cannot see a defect introduced by assembly +/// itself (fragment merge order, GF dedup, resource renumbering), and the +/// `graphical_functions` ordering defect was originally caught here: the root +/// `CompiledModule` differed on 18 of 23 repeats. `CompiledSimulation` has no +/// `PartialEq`, so this compares the debug rendering of the root module's +/// bytecode and GF table -- coarser than a field-by-field compare, but it is +/// the layer at which the artifact is actually consumed. +#[test] +fn assembled_simulation_is_stable_across_fresh_databases() { + let dm = TestProject::new("determinism_assembled") + .with_sim_time(0.0, 2.0, 1.0) + .aux_with_gf("first_table", "", two_point_gf(1.0, 2.0)) + .aux_with_gf("second_table", "", two_point_gf(10.0, 20.0)) + .scalar_aux( + "consumer", + "LOOKUP(first_table, time) + LOOKUP(second_table, time)", + ) + .named_dimension("region", &["east", "west", "north"]) + .array_with_ranges( + "arr[region]", + vec![("east", "30"), ("west", "10"), ("north", "20")], + ) + .array_aux( + "combo[region]", + "VECTOR SORT ORDER(arr[region], 1) + RANK(arr[region], 1)", + ) + .stock("level", "10", &["inflow"], &[], None) + .flow("inflow", "consumer", None) + .build_datamodel(); + + let render = || { + let db = SimlinDb::default(); + let source_project = sync_from_datamodel(&db, &dm).project; + let sim = assemble_simulation(&db, source_project, "main".to_string()) + .expect("fixture must assemble"); + let root = &sim.modules[&sim.root]; + format!( + "gf={:?}\nflows={:?}\nstocks={:?}", + root.context.graphical_functions, root.compiled_flows.code, root.compiled_stocks.code + ) + }; + + let first = render(); + for i in 1..REPEATS { + assert_eq!( + first, + render(), + "assembly #{i} on a fresh database produced a different root \ + CompiledModule; the compiled artifact must be a function of the \ + project alone" + ); + } +} + +/// The LTM emission path has its OWN copy of the compile+symbolize tail, so +/// `temp_sizes_by_id` being called there is a separate fact from it being +/// called in `db::assemble`, and must be pinned separately: reverting either +/// call site alone leaves the other's test green. +/// +/// An LTM link score is woven from its TARGET's equation, so a target built +/// from two array-producing builtins yields a synthetic link-score fragment +/// carrying several temps -- six for this fixture. +/// +/// This covers `db::ltm::compile::compile_ltm_equation_fragment` (the LTM +/// synthetic-variable tail). The other LTM copy, in +/// `compile_ltm_implicit_var_fragment`, is NOT covered and cannot be: its +/// fragments are the `PREVIOUS` capture auxes the score generators synthesize, +/// which are pinned to a single element (`…⁚arg0⁚east`) and are therefore +/// always scalar. An array-producing builtin needs a whole-array operand, and +/// the one shape that would supply one -- `PREVIOUS` of a wildcard slice -- +/// has no `LoadPrev`-of-array-view codegen path and degrades to a warned zero +/// (the GH #517 case `db::ltm_char_tests::char_agg_nested_reducer` pins). So +/// that call site has no reachable temps to order. +#[test] +fn ltm_fragment_with_temps_is_stable_across_fresh_databases() { + use salsa::Setter; + + let dm = TestProject::new("determinism_ltm_temps") + .with_sim_time(0.0, 2.0, 1.0) + .named_dimension("region", &["east", "west"]) + .array_aux("rate[region]", "0.1") + .array_flow( + "growth[region]", + "(VECTOR SORT ORDER(level[region], 1) + RANK(level[region], 1)) * rate[region] * 0.01", + None, + ) + .array_stock("level[region]", "10", &["growth"], &[], None) + .build_datamodel(); + + let compile_score = || { + let mut db = SimlinDb::default(); + let source_project = sync_from_datamodel(&db, &dm).project; + source_project.set_ltm_enabled(&mut db).to(true); + let model = *source_project.models(&db).get("main").unwrap(); + // Reached through the selector `assemble_module` uses, not the + // `(from, to)`-keyed salsa query: that query re-derives the score as a + // scalar `Bare` shape, which for an ARRAYED target is the wrong + // equation and compiles to nothing. `compile_ltm_synthetic_fragment` is + // the production entry that routes an arrayed score to + // `compile_ltm_equation_fragment` verbatim. + let ltm_vars = crate::db::model_ltm_variables(&db, model, source_project); + let score = ltm_vars + .vars + .iter() + .find(|v| v.name.contains("link_score") && v.name.contains("level\u{2192}growth")) + .expect("the level->growth link score must be generated"); + crate::db::compile_ltm_synthetic_fragment(&db, score, model, source_project) + .expect("the level->growth link score must compile") + .fragment + .flow_bytecodes + .clone() + .expect("the link score must have a flow fragment") + }; + + let first = compile_score(); + assert!( + first.temp_sizes.len() >= 2, + "the fixture must give the LTM link-score fragment MORE THAN ONE temp, \ + or this test cannot observe an ordering at all; got {:?}", + first.temp_sizes + ); + for i in 1..REPEATS { + assert_eq!( + first, + compile_score(), + "LTM compile #{i} on a fresh database produced a different \ + link-score fragment" + ); + } +} + +/// A two-point continuous graphical function over x in [0, 1]. +fn two_point_gf(y0: f64, y1: f64) -> datamodel::GraphicalFunction { + datamodel::GraphicalFunction { + kind: datamodel::GraphicalFunctionKind::Continuous, + x_points: Some(vec![0.0, 1.0]), + y_points: vec![y0, y1], + x_scale: datamodel::GraphicalFunctionScale { min: 0.0, max: 1.0 }, + y_scale: datamodel::GraphicalFunctionScale { + min: y0.min(y1), + max: y0.max(y1), + }, + } +} diff --git a/src/simlin-engine/src/db/invariance.rs b/src/simlin-engine/src/db/invariance.rs index 02a776691..7b4106ada 100644 --- a/src/simlin-engine/src/db/invariance.rs +++ b/src/simlin-engine/src/db/invariance.rs @@ -16,17 +16,20 @@ //! run with an all-`Invariant` offset callback, so it flags only variant //! *builtins* (TIME / PULSE / RAMP / STEP / PREVIOUS / EvalModule / //! ModuleInput); -//! * `dep_names` -- the variables whose slots the FLOW exprs actually read -//! (`collect_expr_offsets` reverse-mapped through the mini-layout; `INIT()` -//! arguments are skipped because the init buffer is frozen). +//! * `dep_names` -- the variables the FLOW exprs actually read +//! (`collect_expr_refs`, reading the owning variable's name straight off +//! each reference; `INIT()` arguments are skipped because the init buffer +//! is frozen). //! //! This query is then just the fixpoint over the dependency graph: //! `invariant(v) iff locally_pure(v) && dep_names(v) ⊆ invariant-set`, so the //! whole burden of catching variant *dependencies* (stocks, dynamic auxes, -//! module outputs) rides on `dep_names` -- which is why -//! `compute_flow_invariance_support` is loud about unresolvable offsets and the -//! end-to-end bit-constancy oracle (`tests/integration/simulate.rs` `oracle_*`) -//! pins the production mechanism. +//! module outputs) rides on `dep_names`. That set used to be recovered by +//! reverse-mapping slot offsets through a private per-fragment layout, with a +//! `debug_assert!` guarding the silent-drop case; references now carry the name +//! and there is no lookup left to fail. The end-to-end bit-constancy oracle +//! (`tests/integration/simulate.rs` `oracle_*`) still pins the production +//! mechanism. //! //! The flow runlist (`ModelDepGraphResult.runlist_flows`) is a topological //! order: every non-stock/non-module dt dependency precedes its reader. So a @@ -163,7 +166,7 @@ pub(crate) fn model_flows_invariant<'db>( mod tests { use super::*; use crate::compiler::Expr; - use crate::compiler::invariance::{OffsetClass, exprs_are_invariant}; + use crate::compiler::invariance::{RefClass, exprs_are_invariant}; use crate::datamodel; use crate::db::{ModuleInputSet, SimlinDb, sync_from_datamodel}; use crate::test_common::TestProject; @@ -182,22 +185,23 @@ mod tests { /// Compute the monolithic-path run-invariant flow var-name set by running /// the SAME shared classifier over the test-only `Module`'s model-global - /// lowered exprs. The offset callback resolves a model-global offset to its - /// owning variable via the `Module`'s metadata, then classifies the owner by - /// kind (stock/module -> variant) and by the accumulated invariant set, - /// mirroring the salsa callback. Restricted to scalar variables (no array - /// temps) so `Module::get_flow_exprs` captures each variable's full flow - /// statement list. + /// lowered exprs. The callback reads the owning variable's name straight off + /// the reference, then classifies it by kind (stock/module -> variant) and + /// by the accumulated invariant set, mirroring the salsa callback. + /// Restricted to scalar variables (no array temps) so + /// `Module::get_flow_exprs` captures each variable's full flow statement + /// list. + /// + /// Before references carried names this callback had to reverse-map a + /// model-global offset through the `Module`'s own offset map -- a second, + /// hand-written copy of the resolution the salsa side did differently. The + /// two callbacks are now the same lookup, which is why this differential + /// test is cheaper AND stronger than it was. fn monolithic_invariant_set(tp: &TestProject, runlist_order: &[String]) -> BTreeSet { use crate::common::{Canonical, Ident}; let module = tp.build_module().expect("build monolithic module"); let model_ident = module.ident.clone(); - let model_offsets = module - .offsets - .get(&model_ident) - .expect("model offsets") - .clone(); // Stocks and modules in this model (by canonical name): a referenced // owner of these kinds is variant. We derive stock/module membership @@ -235,28 +239,22 @@ mod tests { continue; } - let classify_offset = |off: usize| -> OffsetClass { - let owner = model_offsets - .iter() - .find(|(_, (base, size))| off >= *base && off < *base + *size) - .map(|(name, _)| name.as_str().to_string()); - let Some(owner) = owner else { - return OffsetClass::Variant; - }; - if owner == *var_name { - return OffsetClass::Invariant; + let classify_ref = |var: &crate::compiler::VarRef| -> RefClass { + let owner = var.name.as_str(); + if owner == var_name.as_str() { + return RefClass::Invariant; } - if stock_or_module.contains(&owner) { - return OffsetClass::Variant; + if stock_or_module.contains(owner) { + return RefClass::Variant; } - if invariant.contains(&owner) { - OffsetClass::Invariant + if invariant.contains(owner) { + RefClass::Invariant } else { - OffsetClass::Variant + RefClass::Variant } }; - if exprs_are_invariant(&exprs, &classify_offset) { + if exprs_are_invariant(&exprs, &classify_ref) { invariant.insert(var_name.clone()); } } diff --git a/src/simlin-engine/src/db/ltm/compile.rs b/src/simlin-engine/src/db/ltm/compile.rs index 37115c3ee..45c933ff5 100644 --- a/src/simlin-engine/src/db/ltm/compile.rs +++ b/src/simlin-engine/src/db/ltm/compile.rs @@ -13,20 +13,21 @@ //! compile-failure diagnostic pass (`model_ltm_fragment_diagnostics`), and //! the implicit-variable fragment compiler (`compile_ltm_implicit_var_fragment`). -use std::collections::{BTreeSet, HashMap, HashSet}; +use std::collections::{BTreeSet, HashMap}; use crate::canonicalize; use crate::common::{Canonical, Ident}; use crate::datamodel; use crate::db::{ - Db, LtmLinkId, LtmSyntheticVar, ModelDepGraphResult, ModuleInputSet, RefShape, SourceModel, - SourceProject, SourceVariableKind, VarFragmentResult, build_module_inputs, build_stub_variable, - build_submodel_metadata, canonical_module_input_set, compute_layout, - extract_tables_from_source_var, model_dependency_graph, model_implicit_var_info, - model_module_ident_context, model_module_map, parse_source_variable_with_module_context, - project_converted_dimensions, project_datamodel_dims, project_dimensions_context, - project_units_context, reconstruct_single_variable, variable_dimensions, variable_size, + Db, LtmLinkId, LtmSyntheticVar, ModelDepGraphResult, ModuleInputSet, PerVarSizes, RefShape, + SourceModel, SourceProject, SourceVariableKind, VarFragmentResult, build_module_inputs, + build_stub_variable, build_submodel_metadata, canonical_module_input_set, + compile_phase_to_per_var_bytecodes, compute_layout, extract_tables_from_source_var, + fragment_emit_ctx, model_dependency_graph, model_implicit_var_info, model_module_ident_context, + model_module_map, parse_source_variable_with_module_context, project_converted_dimensions, + project_datamodel_dims, project_dimensions_context, project_units_context, + reconstruct_single_variable, variable_dimensions, variable_size, }; use super::parse::{ltm_equation_dimensions, parse_ltm_equation, scalarize_ltm_equation}; @@ -86,6 +87,12 @@ pub fn compile_ltm_var_fragment( model: SourceModel, project: SourceProject, ) -> Option { + #[cfg(test)] + crate::db::note_fragment_execution( + crate::db::FragmentExecKind::Ltm, + &format!("{}\u{2192}{}", link_id.link_from(db), link_id.link_to(db)), + ); + let ShapedLinkScore::Scored(lsv) = link_score_equation_text_shaped(db, link_id, RefShape::Bare, model, project) else { @@ -444,7 +451,40 @@ struct LoweredLtmVariable { /// the dt AST). Identifier sets are lowering-scope-independent, so /// this is valid for the returned `variable` whether or not the /// scoped re-lower ran. - dep_idents: HashSet>, + /// + /// ORDERED, not a `HashSet` -- **deliberately, even though no consumer is + /// order-sensitive today.** Keep it that way; the reasoning is not + /// "something breaks if you don't", so a reader who checks only that will + /// wrongly conclude it is free to change. + /// + /// Both consumers walk this set to build per-dependency stub variables, + /// which land in a `HashMap`-keyed symbol table and a `HashMap` of tables. + /// Iteration order therefore does not reach the emitted fragment: the + /// stubs are keyed by ident, the insertion loops are first-inserted-wins + /// over distinct idents, and `Compiler::new` sorts the table idents it + /// lays graphical functions out from. The walk USED to assign consecutive + /// private per-fragment offsets in iteration order -- so a `HashSet` made + /// them a function of the per-process hash seed -- and symbolization then + /// inverted those offsets, which is why even then nothing observable + /// changed. GH #964 deleted both the offsets and the symbolization, so the + /// ordering is now inert rather than invisible. + /// + /// The rule it upholds is unchanged and is the reason to keep it: a + /// query's intermediate state must be a function of its inputs, not of the + /// hash seed. That is the same rule `db::assemble::temp_sizes_by_id` + /// restores, and the class of defect it prevents (GH #595) does not + /// announce itself -- it surfaces as salsa backdating quietly failing or a + /// compiled artifact that is not reproducible run to run, neither of which + /// a test would attribute back to here. It costs nothing, and the non-LTM + /// twins (`db::var_fragment::collect_var_dependencies`, + /// `db::compile_implicit_var_phase_bytecodes`) use a `BTreeSet` here too; + /// these two hand-copied LTM walks were the outliers. + /// + /// This does NOT explain (or fix) the separately-reported + /// nondeterministic *invalidation* of `compile_ltm_var_fragment`: that + /// was measured before and after this change with no effect, and salsa + /// verifies a dependency SET, which an ordering cannot alter. + dep_idents: BTreeSet>, /// `classify_dependencies(..).referenced_tables` of the same AST. referenced_tables: BTreeSet, } @@ -669,8 +709,8 @@ fn lower_ltm_variable( .ast() .map(|ast| crate::variable::classify_dependencies(ast, &[], None)); let (dep_idents, referenced_tables) = match classification { - Some(c) => (c.all, c.referenced_tables), - None => (HashSet::new(), BTreeSet::new()), + Some(c) => (c.all.into_iter().collect(), c.referenced_tables), + None => (BTreeSet::new(), BTreeSet::new()), }; // Structural gate: without a Pass-1 temp-decomposition site in the @@ -808,9 +848,7 @@ pub(crate) fn compile_ltm_equation_fragment( model: SourceModel, project: SourceProject, ) -> Option { - use crate::compiler::symbolic::{ - CompiledVarFragment, PerVarBytecodes, ReverseOffsetMap, VariableLayout, - }; + use crate::compiler::symbolic::{CompiledVarFragment, PerVarBytecodes}; // Project-global dims (datamodel form, used to resolve the equation's // dimension names) plus the canonicalized context + converted dims, all @@ -864,9 +902,6 @@ pub(crate) fn compile_ltm_equation_fragment( let mut mini_metadata: HashMap, crate::compiler::VariableMetadata<'_>> = HashMap::new(); - // Mini-layout starts after the 4 implicit time vars (time, dt, initial_time, final_time) - let mut mini_offset = crate::vm::IMPLICIT_VAR_COUNT; - // Add implicit time/dt/initial_time/final_time variables { use std::sync::LazyLock; @@ -929,7 +964,7 @@ pub(crate) fn compile_ltm_equation_fragment( mini_metadata.insert( Ident::new("time"), crate::compiler::VariableMetadata { - offset: 0, + offset: None, size: 1, var: &IMPLICIT_TIME, }, @@ -937,7 +972,7 @@ pub(crate) fn compile_ltm_equation_fragment( mini_metadata.insert( Ident::new("dt"), crate::compiler::VariableMetadata { - offset: 1, + offset: None, size: 1, var: &IMPLICIT_DT, }, @@ -945,7 +980,7 @@ pub(crate) fn compile_ltm_equation_fragment( mini_metadata.insert( Ident::new("initial_time"), crate::compiler::VariableMetadata { - offset: 2, + offset: None, size: 1, var: &IMPLICIT_INITIAL_TIME, }, @@ -953,7 +988,7 @@ pub(crate) fn compile_ltm_equation_fragment( mini_metadata.insert( Ident::new("final_time"), crate::compiler::VariableMetadata { - offset: 3, + offset: None, size: 1, var: &IMPLICIT_FINAL_TIME, }, @@ -978,12 +1013,11 @@ pub(crate) fn compile_ltm_equation_fragment( mini_metadata.insert( var_ident_canonical.clone(), crate::compiler::VariableMetadata { - offset: mini_offset, + offset: None, size: var_size, var: &lowered, }, ); - mini_offset += var_size; // `dep_idents`/`referenced_tables` came back from `lower_ltm_variable` // (classified once during lowering). Lookup-table references are not @@ -1469,12 +1503,11 @@ pub(crate) fn compile_ltm_equation_fragment( mini_metadata.insert( dep_ident.clone(), crate::compiler::VariableMetadata { - offset: mini_offset, + offset: None, size: *dep_size, var: dep_var, }, ); - mini_offset += dep_size; } } @@ -1484,12 +1517,11 @@ pub(crate) fn compile_ltm_equation_fragment( mini_metadata.insert( im_ident.clone(), crate::compiler::VariableMetadata { - offset: mini_offset, + offset: None, size: *im_size, var: im_var, }, ); - mini_offset += im_size; } } @@ -1505,11 +1537,6 @@ pub(crate) fn compile_ltm_equation_fragment( build_submodel_metadata(&arena, db, *sub_model, project, &mut all_metadata); } - let mini_layout = - crate::compiler::symbolic::layout_from_metadata(&all_metadata, &model_name_ident) - .unwrap_or_else(|_| VariableLayout::new(HashMap::new(), 0)); - let rmap = ReverseOffsetMap::from_layout(&mini_layout); - let inputs = BTreeSet::new(); // Merge LTM implicit module references from LTM equation parsing into the @@ -1534,11 +1561,20 @@ pub(crate) fn compile_ltm_equation_fragment( &merged_module_models }; + // The LTM synthetic var's fragment goes through the SAME emission entry + // point the explicit and implicit paths use. This site used to carry a + // hand-copied duplicate of that function's body, differing only in which + // module-ref map it stuffed into the stand-in `Module` -- a field codegen + // never read. + let var_sizes: PerVarSizes = + crate::compiler::whole_variable_extents(&all_metadata, &model_name_ident); + let core = crate::compiler::ContextCore { dimensions: converted_dims, dimensions_ctx: dim_context, model_name: &model_name_ident, metadata: &all_metadata, + var_sizes: &var_sizes, module_models, inputs: &inputs, }; @@ -1550,103 +1586,16 @@ pub(crate) fn compile_ltm_equation_fragment( ) }; + let base_ctx = fragment_emit_ctx( + &model_name_ident, + &inputs, + &var_sizes, + &tables, + converted_dims, + dim_context, + ); let compile_phase = |exprs: &[crate::compiler::Expr]| -> Option { - if exprs.is_empty() { - return None; - } - - let module_inputs_set: HashSet> = inputs.iter().cloned().collect(); - let module = crate::compiler::Module { - ident: model_name_ident.clone(), - inputs: module_inputs_set, - n_slots: mini_offset, - n_temps: 0, - temp_sizes: vec![], - runlist_initials: vec![], - runlist_initials_by_var: vec![], - runlist_flows: exprs.to_vec(), - runlist_stocks: vec![], - offsets: all_metadata - .iter() - .map(|(k, v)| { - ( - k.clone(), - v.iter() - .map(|(vk, vm)| (vk.clone(), (vm.offset, vm.size))) - .collect(), - ) - }) - .collect(), - runlist_order: vec![var_ident_canonical.clone()], - tables: tables.clone(), - // Owned Module fields: the slice/context come from the cached - // queries by reference, so materialize owned copies here. The - // interned-backed `Dimension`s clone cheaply -- the expensive - // rebuild (re-canonicalizing every element) is what the cache - // removes; only the relationship-cache memo is rebuilt cold. - dimensions: converted_dims.to_vec(), - dimensions_ctx: (*dim_context).clone(), - module_refs: implicit_module_refs.clone(), - }; - - let mut temp_sizes_map: HashMap = HashMap::new(); - for expr in exprs { - crate::compiler::extract_temp_sizes_pub(expr, &mut temp_sizes_map); - } - let n_temps = temp_sizes_map.len(); - let mut temp_sizes: Vec = vec![0; n_temps]; - for (id, size) in &temp_sizes_map { - if (*id as usize) < temp_sizes.len() { - temp_sizes[*id as usize] = *size; - } - } - - let module = crate::compiler::Module { - n_temps, - temp_sizes: temp_sizes.clone(), - ..module - }; - - match module.compile() { - Ok(compiled) => { - let sym_bc = - crate::compiler::symbolic::symbolize_bytecode(&compiled.compiled_flows, &rmap) - .ok()?; - - let ctx = &*compiled.context; - let sym_views: Vec<_> = ctx - .static_views - .iter() - .map(|sv| crate::compiler::symbolic::symbolize_static_view(sv, &rmap)) - .collect::, _>>() - .ok()?; - let sym_mods: Vec<_> = ctx - .modules - .iter() - .map(|md| crate::compiler::symbolic::symbolize_module_decl(md, &rmap)) - .collect::, _>>() - .ok()?; - - let temp_sizes_vec: Vec<(u32, usize)> = - temp_sizes_map.iter().map(|(&k, &v)| (k, v)).collect(); - - let dim_lists: Vec> = ctx - .dim_lists - .iter() - .map(|(n, arr)| arr[..(*n as usize)].to_vec()) - .collect(); - - Some(PerVarBytecodes { - symbolic: sym_bc, - graphical_functions: ctx.graphical_functions.clone(), - module_decls: sym_mods, - static_views: sym_views, - temp_sizes: temp_sizes_vec, - dim_lists, - }) - } - Err(_) => None, - } + compile_phase_to_per_var_bytecodes(&base_ctx, exprs) }; // LTM vars are always flow-phase only (scalar auxes, not stocks) @@ -1993,9 +1942,7 @@ pub(crate) fn compile_ltm_implicit_var_fragment( _dep_graph: &ModelDepGraphResult, module_input_names: &[String], ) -> Option { - use crate::compiler::symbolic::{ - CompiledVarFragment, PerVarBytecodes, ReverseOffsetMap, VariableLayout, - }; + use crate::compiler::symbolic::{CompiledVarFragment, PerVarBytecodes}; // The implicit variable rides on the meta (captured at LTM-equation parse // time by `model_ltm_implicit_var_info`), so no parent re-parse is needed. @@ -2048,7 +1995,7 @@ pub(crate) fn compile_ltm_implicit_var_fragment( // Dependency classification handed back by `lower_ltm_variable` for the // non-module path, reused by the dep-collection pass below (the module // path constructs its deps from the dm_module references instead). - let mut ltm_lowered_deps: Option<(HashSet>, BTreeSet)> = None; + let mut ltm_lowered_deps: Option<(BTreeSet>, BTreeSet)> = None; // Module-type implicit vars need direct Module construction let lowered = if meta.is_module { @@ -2104,8 +2051,6 @@ pub(crate) fn compile_ltm_implicit_var_fragment( let mut mini_metadata: HashMap, crate::compiler::VariableMetadata<'_>> = HashMap::new(); - // LTM implicit vars are in the root model context - let mut mini_offset = crate::vm::IMPLICIT_VAR_COUNT; // Add implicit time/dt/initial_time/final_time { @@ -2169,7 +2114,7 @@ pub(crate) fn compile_ltm_implicit_var_fragment( mini_metadata.insert( Ident::new("time"), crate::compiler::VariableMetadata { - offset: 0, + offset: None, size: 1, var: &IMPLICIT_TIME, }, @@ -2177,7 +2122,7 @@ pub(crate) fn compile_ltm_implicit_var_fragment( mini_metadata.insert( Ident::new("dt"), crate::compiler::VariableMetadata { - offset: 1, + offset: None, size: 1, var: &IMPLICIT_DT, }, @@ -2185,7 +2130,7 @@ pub(crate) fn compile_ltm_implicit_var_fragment( mini_metadata.insert( Ident::new("initial_time"), crate::compiler::VariableMetadata { - offset: 2, + offset: None, size: 1, var: &IMPLICIT_INITIAL_TIME, }, @@ -2193,7 +2138,7 @@ pub(crate) fn compile_ltm_implicit_var_fragment( mini_metadata.insert( Ident::new("final_time"), crate::compiler::VariableMetadata { - offset: 3, + offset: None, size: 1, var: &IMPLICIT_FINAL_TIME, }, @@ -2206,12 +2151,11 @@ pub(crate) fn compile_ltm_implicit_var_fragment( mini_metadata.insert( var_ident_canonical.clone(), crate::compiler::VariableMetadata { - offset: mini_offset, + offset: None, size: self_size, var: &lowered, }, ); - mini_offset += self_size; // Collect dependencies from the implicit var itself let source_vars = model.variables(db); @@ -2333,7 +2277,7 @@ pub(crate) fn compile_ltm_implicit_var_fragment( let (dep_idents, referenced_tables) = if lowered.ast().is_some() { ltm_lowered_deps.take().unwrap_or_default() } else { - (HashSet::new(), BTreeSet::new()) + (BTreeSet::new(), BTreeSet::new()) }; let implicit_info = model_implicit_var_info(db, model, project); @@ -2474,12 +2418,11 @@ pub(crate) fn compile_ltm_implicit_var_fragment( mini_metadata.insert( dep_ident.clone(), crate::compiler::VariableMetadata { - offset: mini_offset, + offset: None, size: *dep_size, var: dep_var, }, ); - mini_offset += dep_size; } } @@ -2508,11 +2451,6 @@ pub(crate) fn compile_ltm_implicit_var_fragment( } } - let mini_layout = - crate::compiler::symbolic::layout_from_metadata(&all_metadata, &model_name_ident) - .unwrap_or_else(|_| VariableLayout::new(HashMap::new(), 0)); - let rmap = ReverseOffsetMap::from_layout(&mini_layout); - let tables = fragment_tables; let inputs = canonical_module_input_set(module_input_names); @@ -2547,11 +2485,19 @@ pub(crate) fn compile_ltm_implicit_var_fragment( &merged_module_models }; + // Same single emission entry point as every other fragment site. This was + // the second hand-copied duplicate of that body; it differed from the + // first only in the (unread) module-ref map, and it rebuilt the offsets + // projection once per phase rather than once per variable. + let var_sizes: PerVarSizes = + crate::compiler::whole_variable_extents(&all_metadata, &model_name_ident); + let core = crate::compiler::ContextCore { dimensions: converted_dims, dimensions_ctx: dim_context, model_name: &model_name_ident, metadata: &all_metadata, + var_sizes: &var_sizes, module_models, inputs: &inputs, }; @@ -2563,103 +2509,16 @@ pub(crate) fn compile_ltm_implicit_var_fragment( ) }; + let base_ctx = fragment_emit_ctx( + &model_name_ident, + &inputs, + &var_sizes, + &tables, + converted_dims, + dim_context, + ); let compile_phase = |exprs: &[crate::compiler::Expr]| -> Option { - if exprs.is_empty() { - return None; - } - - let module_inputs_set: HashSet> = inputs.iter().cloned().collect(); - let module = crate::compiler::Module { - ident: model_name_ident.clone(), - inputs: module_inputs_set, - n_slots: mini_offset, - n_temps: 0, - temp_sizes: vec![], - runlist_initials: vec![], - runlist_initials_by_var: vec![], - runlist_flows: exprs.to_vec(), - runlist_stocks: vec![], - offsets: all_metadata - .iter() - .map(|(k, v)| { - ( - k.clone(), - v.iter() - .map(|(vk, vm)| (vk.clone(), (vm.offset, vm.size))) - .collect(), - ) - }) - .collect(), - runlist_order: vec![var_ident_canonical.clone()], - tables: tables.clone(), - // Owned Module fields: the slice/context come from the cached - // queries by reference, so materialize owned copies here. The - // interned-backed `Dimension`s clone cheaply -- the expensive - // rebuild (re-canonicalizing every element) is what the cache - // removes; only the relationship-cache memo is rebuilt cold. - dimensions: converted_dims.to_vec(), - dimensions_ctx: (*dim_context).clone(), - module_refs: module_refs.clone(), - }; - - let mut temp_sizes_map: HashMap = HashMap::new(); - for expr in exprs { - crate::compiler::extract_temp_sizes_pub(expr, &mut temp_sizes_map); - } - let n_temps = temp_sizes_map.len(); - let mut temp_sizes: Vec = vec![0; n_temps]; - for (id, size) in &temp_sizes_map { - if (*id as usize) < temp_sizes.len() { - temp_sizes[*id as usize] = *size; - } - } - - let module = crate::compiler::Module { - n_temps, - temp_sizes: temp_sizes.clone(), - ..module - }; - - match module.compile() { - Ok(compiled) => { - let sym_bc = - crate::compiler::symbolic::symbolize_bytecode(&compiled.compiled_flows, &rmap) - .ok()?; - - let ctx = &*compiled.context; - let sym_views: Vec<_> = ctx - .static_views - .iter() - .map(|sv| crate::compiler::symbolic::symbolize_static_view(sv, &rmap)) - .collect::, _>>() - .ok()?; - let sym_mods: Vec<_> = ctx - .modules - .iter() - .map(|md| crate::compiler::symbolic::symbolize_module_decl(md, &rmap)) - .collect::, _>>() - .ok()?; - - let temp_sizes_vec: Vec<(u32, usize)> = - temp_sizes_map.iter().map(|(&k, &v)| (k, v)).collect(); - - let dim_lists: Vec> = ctx - .dim_lists - .iter() - .map(|(n, arr)| arr[..(*n as usize)].to_vec()) - .collect(); - - Some(PerVarBytecodes { - symbolic: sym_bc, - graphical_functions: ctx.graphical_functions.clone(), - module_decls: sym_mods, - static_views: sym_views, - temp_sizes: temp_sizes_vec, - dim_lists, - }) - } - Err(_) => None, - } + compile_phase_to_per_var_bytecodes(&base_ctx, exprs) }; // LTM implicit vars participate in whichever phases their lowered diff --git a/src/simlin-engine/src/db/ltm_tests.rs b/src/simlin-engine/src/db/ltm_tests.rs index c6af3e3bd..7d4eb0859 100644 --- a/src/simlin-engine/src/db/ltm_tests.rs +++ b/src/simlin-engine/src/db/ltm_tests.rs @@ -166,7 +166,7 @@ fn test_a2a_ltm_equation_fragment_compiles() { .iter() .filter_map(|op| match op { SymbolicOpcode::BinOpAssignCurr { var, .. } - if var.name.contains("test_a2a_link_score") => + if var.name.as_str().contains("test_a2a_link_score") => { Some(var.element_offset) } @@ -288,7 +288,7 @@ fn test_a2a_ltm_previous_per_element() { .code .iter() .filter_map(|op| match op { - SymbolicOpcode::SymLoadPrev { var } if var.name == "population" => { + SymbolicOpcode::SymLoadPrev { var } if var.name.as_str() == "population" => { Some(var.element_offset) } _ => None, @@ -314,7 +314,7 @@ fn test_a2a_ltm_previous_per_element() { .iter() .filter_map(|op| match op { SymbolicOpcode::BinOpAssignCurr { var, .. } - if var.name.contains("test_prev_per_elem") => + if var.name.as_str().contains("test_prev_per_elem") => { Some(var.element_offset) } diff --git a/src/simlin-engine/src/db/query.rs b/src/simlin-engine/src/db/query.rs index 5c228084f..9fa20d672 100644 --- a/src/simlin-engine/src/db/query.rs +++ b/src/simlin-engine/src/db/query.rs @@ -538,6 +538,66 @@ pub fn model_implicit_var_info( result } +/// Resolve one *implicit* (SMOOTH/DELAY/TREND/PREVIOUS/INIT) helper of `model` +/// by its canonical name -- the firewall over `model_implicit_var_info`, the +/// third coarse edge on the per-variable fragment path. +/// +/// `model_implicit_var_info` is derived from every variable's parse, so adding +/// a variable that synthesizes ANY implicit helper changes the whole map and +/// re-executes every fragment that reads it. `collect_var_dependencies` only +/// ever asks it "is this one name an implicit helper, and what shape is it", +/// so this projection returns that and backdates on it. +/// +/// **This narrowing is partial, and the limitation is pinned, not merely +/// noted.** It makes an added `PREVIOUS`/`INIT` helper tight. It does NOT make +/// an added *module-instantiating* helper (`SMTH1`, `DELAY`, a user sub-model) +/// tight, because two further whole-model dependencies survive on that path +/// and neither is a projection away: `project.models(db)` changes when the +/// implicit `stdlib⁚smth1` model is spliced into the project, and +/// `model_module_ident_context` is an INTERNED handle whose value changes when +/// the module-ident set grows -- which gives every variable's parse a new +/// cache key, and a new key cannot backdate at all. Both outcomes are asserted +/// by `implicit_helper_add_is_tight_but_module_helper_add_is_not`, so the day +/// someone fixes the module-ident cache key that test reds rather than +/// silently over-delivering. +#[salsa::tracked] +pub(crate) fn model_implicit_var_by_name( + db: &dyn Db, + model: SourceModel, + project: SourceProject, + name: String, +) -> Option { + model_implicit_var_info(db, model, project) + .get(&name) + .cloned() +} + +/// Resolve one variable of `model` by its canonical name -- a salsa +/// *firewall* over the `SourceModel::variables` map field. +/// +/// Reading `model.variables(db)` directly makes the reader depend on the whole +/// map, so adding, deleting or renaming ANY variable invalidates it. That is +/// what made every per-variable fragment in a model recompile whenever the +/// model's variable SET changed, even though the fragments were bit-identical +/// (GH #964's "layout-only project edits continue to reuse unchanged +/// salsa-cached fragments" criterion). This query still reads the whole map, +/// so it re-executes on any such edit -- but its VALUE is one handle, so salsa +/// backdates it whenever that one variable is untouched and no reader re-runs. +/// A fragment therefore depends on exactly the dependencies it looks up. +/// +/// The handle is stable across unrelated edits because `sync` threads the +/// prior `SourceVariable` inputs through `PersistentSyncState`; a genuinely +/// re-created variable (delete + re-add, or a rename) gets a new handle, and +/// its readers -- only those that name it -- re-execute. +#[salsa::tracked] +pub(crate) fn model_variable_by_name( + db: &dyn Db, + model: SourceModel, + name: String, +) -> Option { + model.variables(db).get(&name).copied() +} + #[salsa::tracked(returns(ref))] pub fn model_module_map( db: &dyn Db, diff --git a/src/simlin-engine/src/db/var_fragment.rs b/src/simlin-engine/src/db/var_fragment.rs index 0e67589e6..e6fd7dd49 100644 --- a/src/simlin-engine/src/db/var_fragment.rs +++ b/src/simlin-engine/src/db/var_fragment.rs @@ -16,12 +16,21 @@ //! `crate::db::compile_var_fragment` (in `db/fragment_compile.rs`), which //! consumes the owned, lowering-independent values this returns. //! -//! The split exists for two reasons. First, the lowered `Vec` is -//! the natural reuse surface for read-only structural probes that need -//! the engine's *own* per-variable production lowering without re-running -//! it with a reconstructed context. Second, `Vec` does not -//! implement `salsa::Update`, so the lowering step must be a plain -//! function while the caller remains the salsa-tracked query. +//! The split exists because the lowered `Vec` is the natural reuse +//! surface for read-only structural probes that need the engine's *own* +//! per-variable production lowering without re-running it with a +//! reconstructed context. +//! +//! It used to have a second reason -- `Vec` does not implement +//! `salsa::Update`, so lowering had to be a plain function while the caller +//! stayed the tracked query. That reason was really the *prohibition* on +//! `Expr` deriving `Update` (an `Expr` was keyed to one model-global slot +//! layout), and it is gone: since GH #964 an `Expr` references variables by +//! name and carries no offsets, so it is as layout-independent as the +//! symbolic bytecode it becomes. Nothing derives `Update` for it yet, but +//! making this step a tracked query of its own is now a free choice rather +//! than an unsound one. See the rustdoc on +//! `model.rs::stage_types_and_error_implement_salsa_update`. //! //! Because a plain function cannot accumulate salsa diagnostics, the //! diagnostics this step would emit are returned **as data** @@ -43,11 +52,12 @@ //! The coupled cluster `{lowered, all_metadata, arena, ContextCore, //! Context, Var::new}` is internal to `lower_var_fragment` and drops //! together at return; only owned, lifetime-free values -//! (`per_phase_lowered`, `tables`, `offsets`, `rmap`, `mini_offset`) -//! cross back to the caller. The metadata map borrows the lowered -//! variable (its self-entry is `&lowered`) and the sub-model stub arena, -//! so it must not outlive them; the caller never sees it -- it consumes -//! only the owned `offsets` projection (variable -> (offset, size)). +//! (`per_phase_lowered`, `tables`, `var_sizes`) cross back to the caller. +//! The metadata map borrows the lowered variable (its self-entry is +//! `&lowered`) and the sub-model stub arena, so it must not outlive them; +//! the caller never sees it -- it consumes only the owned `var_sizes` +//! projection (reference -> the extent of the variable it addresses in +//! whole). //! //! This is a submodule of `db` (a child of `db.rs`, like `dep_graph` / //! `ltm_ir` / `macro_registry`) kept in its own file purely to keep `db.rs` @@ -63,17 +73,11 @@ use crate::db::{ Db, Diagnostic, DiagnosticError, DiagnosticSeverity, ModuleInputSet, SourceModel, SourceProject, SourceVariable, SourceVariableKind, build_module_inputs, build_stub_variable, build_submodel_metadata, compute_layout, extract_tables_from_source_var, - model_implicit_var_info, model_module_ident_context, parse_source_variable_with_module_context, - variable_dimensions, variable_direct_dependencies, variable_size, + model_implicit_var_by_name, model_module_ident_context, model_variable_by_name, + parse_source_variable_with_module_context, variable_dimensions, variable_direct_dependencies, + variable_size, }; -/// Per-model variable -> (offset, size) projection of the minimal -/// metadata map. This is the owned, borrow-free view of the symbol -/// layout the caller's `compile_phase` consumes (it never reads the -/// borrowed variable, only the offset/size pair), mirroring the -/// `VariableOffsetMap` shape used inside the compiler. -type VarOffsets = HashMap, HashMap, (usize, usize)>>; - /// Result of `crate::compiler::Var::new` for each compilation phase. /// /// `initial` is `Var::new(.., is_initial = true)`; `noninitial` is @@ -111,34 +115,24 @@ pub(crate) enum LoweredVarFragment { unit_diags: Vec, per_phase_lowered: PerPhaseLowered, tables: HashMap, Vec>, - offsets: VarOffsets, - rmap: crate::compiler::symbolic::ReverseOffsetMap, - mini_offset: usize, + var_sizes: crate::db::assemble::PerVarSizes, }, } -/// Dependency-collection outputs for a single variable. +/// Dependency-collection outputs for a single variable: the stub +/// `dep_variables` / `implicit_module_vars` woven into the minimal metadata +/// map, and the sub-model lists feeding sub-model metadata. /// -/// The dependency walk produces both lowering-coupled values (the stub -/// `dep_variables` / `implicit_module_vars` woven into the minimal -/// metadata map, and the sub-model lists feeding sub-model metadata) and -/// lowering-*independent* values (`extra_module_refs` / -/// `implicit_module_refs`, which the caller's `compile_phase` needs). -/// The walk logic is defined exactly once (`collect_var_dependencies`). -/// It is invoked from two call sites per variable -- `lower_var_fragment` -/// and the caller's module-ref reconstruction (`build_caller_module_refs`) -/// -- but `collect_var_dependencies` is pure over salsa-tracked inputs, -/// so the second invocation is a memoized cache hit with no -/// recomputation. `unknown_dependency` is `Some` when the walk hit a -/// reference that is neither a source nor an implicit variable (the fatal -/// unknown-dependency site); the walk stops there (a fatal unknown -/// dependency short-circuits the rest of the walk). +/// The walk logic is defined exactly once (`collect_var_dependencies`) and +/// invoked once per variable, from `lower_var_fragment`. +/// `unknown_dependency` is `Some` when the walk hit a reference that is +/// neither a source nor an implicit variable (the fatal unknown-dependency +/// site); the walk stops there (a fatal unknown dependency short-circuits +/// the rest of the walk). pub(crate) struct VarDepCollection { pub(crate) dep_variables: Vec<(Ident, crate::variable::Variable, usize)>, - pub(crate) extra_module_refs: HashMap, crate::vm::ModuleKey>, pub(crate) extra_submodels: Vec<(String, SourceModel)>, pub(crate) implicit_module_vars: Vec<(Ident, crate::variable::Variable, usize)>, - pub(crate) implicit_module_refs: HashMap, crate::vm::ModuleKey>, pub(crate) implicit_submodels: Vec<(String, SourceModel)>, pub(crate) unknown_dependency: Option, } @@ -261,7 +255,6 @@ fn collect_var_dependencies( // For each dep, build a dimension-only Variable for context. // We need these to live long enough for the metadata references. - let source_vars = model.variables(db); let mut dep_variables: Vec<(Ident, crate::variable::Variable, usize)> = Vec::new(); // Also add inflows/outflows for stocks (needed by stock update expressions) @@ -281,10 +274,8 @@ fn collect_var_dependencies( .chain(extra_dep_names.iter()) .collect(); - // Track module deps that need module_refs and sub-model metadata - let mut extra_module_refs: HashMap, crate::vm::ModuleKey> = HashMap::new(); + // Track module deps that need sub-model metadata let mut extra_submodels: Vec<(String, SourceModel)> = Vec::new(); - let implicit_var_info = model_implicit_var_info(db, model, project); for dep_name in &all_names { // Skip self and implicit vars @@ -312,8 +303,12 @@ fn collect_var_dependencies( continue; } - // Look up the module variable in source_vars or implicit vars - if let Some(mod_source_var) = source_vars.get(module_var_name) { + // Look up the module variable by name (one salsa firewall query + // per dependency, NOT a read of the whole variables map) or among + // the implicit vars. + if let Some(mod_source_var) = + model_variable_by_name(db, model, module_var_name.to_string()) + { if mod_source_var.kind(db) == SourceVariableKind::Module { let mod_model_name = mod_source_var.model_name(db); let sub_canonical = canonicalize(mod_model_name); @@ -351,22 +346,18 @@ fn collect_var_dependencies( ident: module_ident.clone(), model_name: Ident::new(mod_model_name), units: None, - inputs: module_inputs.clone(), + inputs: module_inputs, errors: vec![], unit_errors: vec![], }; - dep_variables.push((module_ident.clone(), mod_var, sub_size)); - - // Build module_refs entry - let input_set: BTreeSet> = - module_inputs.iter().map(|mi| mi.dst.clone()).collect(); - extra_module_refs.insert(module_ident, (Ident::new(mod_model_name), input_set)); + dep_variables.push((module_ident, mod_var, sub_size)); if let Some(sub_model) = project_models.get(sub_canonical.as_ref()) { extra_submodels.push((mod_model_name.to_string(), *sub_model)); } } - } else if let Some(meta) = implicit_var_info.get(module_var_name) + } else if let Some(meta) = + model_implicit_var_by_name(db, model, project, module_var_name.to_string()) && meta.is_module { // Implicit module already handled in the implicit_module_vars section below @@ -379,14 +370,17 @@ fn collect_var_dependencies( continue; } - if let Some(dep_source_var) = source_vars.get(effective_name) { - let dep_dims = variable_dimensions(db, *dep_source_var, project); - let dep_size = variable_size(db, *dep_source_var, project); + if let Some(dep_source_var) = model_variable_by_name(db, model, effective_name.to_string()) + { + let dep_dims = variable_dimensions(db, dep_source_var, project); + let dep_size = variable_size(db, dep_source_var, project); - let dep_var = build_stub_variable(db, dep_source_var, &dep_ident, dep_dims); + let dep_var = build_stub_variable(db, &dep_source_var, &dep_ident, dep_dims); dep_variables.push((dep_ident, dep_var, dep_size)); - } else if let Some(meta) = implicit_var_info.get(effective_name) { + } else if let Some(meta) = + model_implicit_var_by_name(db, model, project, effective_name.to_string()) + { if !meta.is_module { // An *arrayed* implicit helper (GH #541's bare-arrayed-PREVIOUS // case) needs its array shape in the stub so a consumer that @@ -395,7 +389,7 @@ fn collect_var_dependencies( // subscript rejected. Mirror `build_stub_variable`'s dummy // `ApplyToAll(dims, 0)` AST. Scalar helpers (the common case) // keep `ast: None`. - let dep_dims = implicit_dep_dimensions(db, project, meta); + let dep_dims = implicit_dep_dimensions(db, project, &meta); let dummy_ast = if dep_dims.is_empty() { None } else { @@ -444,10 +438,8 @@ fn collect_var_dependencies( .unwrap_or_default(); return VarDepCollection { dep_variables, - extra_module_refs, extra_submodels, implicit_module_vars: Vec::new(), - implicit_module_refs: HashMap::new(), implicit_submodels: Vec::new(), unknown_dependency: Some(Diagnostic { model: model.name(db).clone(), @@ -469,7 +461,6 @@ fn collect_var_dependencies( // module in mini_metadata to resolve the sub-model offset. let mut implicit_module_vars: Vec<(Ident, crate::variable::Variable, usize)> = Vec::new(); - let mut implicit_module_refs: HashMap, crate::vm::ModuleKey> = HashMap::new(); let mut implicit_submodels: Vec<(String, SourceModel)> = Vec::new(); for implicit_dm_var in &parsed.implicit_vars { @@ -494,21 +485,7 @@ fn collect_var_dependencies( errors: vec![], unit_errors: vec![], }; - implicit_module_vars.push((im_ident.clone(), im_var, sub_size)); - - // Build module_refs entry for the implicit module, stripping - // the module ident prefix from dst (same as resolve_module_input) - let im_input_prefix = format!("{im_name}\u{00B7}"); - let input_set: BTreeSet> = dm_module - .references - .iter() - .filter_map(|mr| { - let dst_canonical = canonicalize(&mr.dst); - let bare = dst_canonical.strip_prefix(&im_input_prefix)?; - Some(Ident::new(bare)) - }) - .collect(); - implicit_module_refs.insert(im_ident, (Ident::new(&dm_module.model_name), input_set)); + implicit_module_vars.push((im_ident, im_var, sub_size)); if let Some(sub_model) = project_models.get(sub_canonical.as_ref()) { implicit_submodels.push((dm_module.model_name.clone(), *sub_model)); @@ -518,64 +495,13 @@ fn collect_var_dependencies( VarDepCollection { dep_variables, - extra_module_refs, extra_submodels, implicit_module_vars, - implicit_module_refs, implicit_submodels, unknown_dependency: None, } } -/// Reconstruct the `module_refs` map the caller's `compile_phase` needs. -/// -/// `module_refs` is built only from project/variable data (the variable's -/// own module references plus the module/implicit-module references found -/// during the dependency walk) -- it does not depend on the lowered -/// equation, the metadata arena, or the symbol-table context, so it is -/// rebuilt on the caller side from the same single dependency walk -/// (`collect_var_dependencies`) rather than threaded back across the -/// lowering boundary. This is the exact `module_refs` assembly that -/// previously followed the dependency loop inline. -pub(crate) fn build_caller_module_refs( - db: &dyn Db, - var: SourceVariable, - model: SourceModel, - project: SourceProject, - module_input_names: &[String], -) -> HashMap, crate::vm::ModuleKey> { - let var_ident = var.ident(db).clone(); - let var_ident_canonical: Ident = Ident::new(&var_ident); - let is_module = var.kind(db) == SourceVariableKind::Module; - - let deps = collect_var_dependencies(db, var, model, project, module_input_names); - - // We need module_refs for module variables (explicit or implicit) - let mut module_refs: HashMap, crate::vm::ModuleKey> = if is_module { - let ref_prefix = format!("{var_ident}\u{00B7}"); - let input_set: BTreeSet> = var - .module_refs(db) - .iter() - .filter_map(|mr| { - let dst_canonical = canonicalize(&mr.dst); - let bare = dst_canonical.strip_prefix(&ref_prefix)?; - Some(Ident::new(bare)) - }) - .collect(); - let mut refs = HashMap::new(); - refs.insert( - var_ident_canonical.clone(), - (Ident::new(var.model_name(db)), input_set), - ); - refs - } else { - HashMap::new() - }; - module_refs.extend(deps.implicit_module_refs); - module_refs.extend(deps.extra_module_refs); - module_refs -} - /// Whether `flow_ident` names a flow that is DRIVEN by a special-stock /// expansion pass: an outflow of a stock carrying the `` or `` /// marker. Such a flow's value comes from the native pass (which writes its slot @@ -596,8 +522,19 @@ pub(crate) fn build_caller_module_refs( /// marker: dropping the ``/`` block re-emits the flow's /// `EmptyEquation` diagnostic (pinned by /// `test_conveyor_marker_removal_reinstates_empty_equation`). -fn flow_is_special_stock_driven(db: &dyn Db, model: SourceModel, flow_ident: &str) -> bool { - let flow_canon = canonicalize(flow_ident); +/// +/// Salsa-tracked for the same firewall reason as `model_variable_by_name`: +/// answering the question needs a scan of every stock, so an untracked helper +/// would give its caller a dependency on the whole variables map. Tracked, the +/// `bool` backdates and only a genuine change of the flow's driven-ness +/// invalidates the fragment. +#[salsa::tracked] +pub(crate) fn flow_is_special_stock_driven( + db: &dyn Db, + model: SourceModel, + flow_ident: String, +) -> bool { + let flow_canon = canonicalize(&flow_ident); model.variables(db).values().any(|sv| { sv.kind(db) == SourceVariableKind::Stock && sv @@ -633,8 +570,6 @@ pub(crate) fn lower_var_fragment( module_models: &HashMap, HashMap, Ident>>, inputs: &BTreeSet>, ) -> LoweredVarFragment { - use crate::compiler::symbolic::{ReverseOffsetMap, VariableLayout}; - let var_ident = var.ident(db).clone(); let module_ident_context = model_module_ident_context(db, model, project, module_input_names.to_vec()); @@ -679,7 +614,7 @@ pub(crate) fn lower_var_fragment( && errors .iter() .any(|e| e.code == crate::common::ErrorCode::EmptyEquation) - && flow_is_special_stock_driven(db, model, &var_ident); + && flow_is_special_stock_driven(db, model, var_ident.clone()); let mut fatal_diags: Vec = Vec::new(); for err in &errors { if suppress_empty && err.code == crate::common::ErrorCode::EmptyEquation { @@ -709,6 +644,22 @@ pub(crate) fn lower_var_fragment( ModuleInputSet::empty(db), ); + // Per-call memo over `model_variable_by_name`. The salsa firewall query is + // the SOURCE of truth for the lookup (that is what gives the fragment a + // dependency on the named variable rather than on the whole variables map); + // this only stops the three name loops below from asking it the same + // question three times, which on a dependency-heavy model was the whole + // measurable cost of the narrowing. + let mut resolved: HashMap> = HashMap::new(); + let mut resolve_var = |name: &str| -> Option { + if let Some(hit) = resolved.get(name) { + return *hit; + } + let found = model_variable_by_name(db, model, name.to_string()); + resolved.insert(name.to_string(), found); + found + }; + // A bare reference to a standalone lookup-only table -- the table used as a // value rather than called via `LOOKUP(table, x)` -- has no scalar value of // its own and is rejected (issue #606). After the table-reference / @@ -717,7 +668,6 @@ pub(crate) fn lower_var_fragment( // reference (a real call lands in `referenced_tables`), so its presence in // the dependency sets is exactly this error. { - let source_vars = model.variables(db); let referenced: BTreeSet<&String> = deps .dt_deps .iter() @@ -726,8 +676,8 @@ pub(crate) fn lower_var_fragment( let bare_table_diags: Vec = referenced .into_iter() .filter_map(|dep| { - let dep_sv = source_vars.get(dep.as_str())?; - crate::db::source_var_is_table_only(db, *dep_sv).then(|| Diagnostic { + let dep_sv = resolve_var(dep)?; + crate::db::source_var_is_table_only(db, dep_sv).then(|| Diagnostic { model: model.name(db).clone(), variable: Some(var.ident(db).clone()), error: DiagnosticError::Model(crate::common::Error::new( @@ -779,7 +729,6 @@ pub(crate) fn lower_var_fragment( // this, SUM(arr[*] + 1) fails because the Op2's ArrayBounds are // never computed (get_dimensions returns None for dependencies). let model_name_str = model.name(db); - let source_vars = model.variables(db); let mut stage0_vars: HashMap, crate::model::VariableStage0> = HashMap::new(); @@ -801,10 +750,10 @@ pub(crate) fn lower_var_fragment( if effective.contains('\u{00B7}') { continue; } - if let Some(dep_sv) = source_vars.get(effective) { + if let Some(dep_sv) = resolve_var(effective) { let dep_parsed = parse_source_variable_with_module_context( db, - *dep_sv, + dep_sv, project, module_ident_context, ); @@ -863,30 +812,28 @@ pub(crate) fn lower_var_fragment( // Arena for sub-model stub variables allocated by build_submodel_metadata let arena = bumpalo::Bump::new(); - // Assign sequential offsets for the minimal context. The mini-layout is - // always body-relative (offset 0): the implicit globals - // (time/dt/initial_time/final_time) are NOT inserted because they lower - // to `LoadGlobalVar` at fixed absolute slots, never through this - // metadata/rmap. Symbolization round-trips each offset back to its - // `SymVarRef { name, element_offset }`, so the produced symbolic - // fragment is independent of where the body starts -- the root - // +IMPLICIT_VAR_COUNT shift is applied later, at assembly, via - // `VariableLayout::root_shifted`. This is what makes the diagnostic pass - // and assembly share one salsa cache entry per variable. + // The minimal symbol table for this variable: itself plus its + // dependencies, carrying no offsets at all. The variables of the model + // being compiled have no layout yet -- assembly assigns one -- and lowering + // does not need one, because every reference it emits names its variable. + // That is what makes a fragment position-independent, and hence what lets + // ONE salsa cache entry per variable serve both the diagnostic pass and + // assembly and survive unrelated variables coming and going. (The four + // implicit globals are absent for a separate reason: they lower to + // `LoadGlobalVar` at fixed absolute slots and never go through a symbol + // table at all.) let mut mini_metadata: HashMap, crate::compiler::VariableMetadata<'_>> = HashMap::new(); - let mut mini_offset = 0; // Add self mini_metadata.insert( var_ident_canonical.clone(), crate::compiler::VariableMetadata { - offset: mini_offset, + offset: None, size: var_size, var: &lowered, }, ); - mini_offset += var_size; // Walk dependencies + implicit modules (single shared walk). An // unknown dependency is fatal here, exactly as before. @@ -912,12 +859,11 @@ pub(crate) fn lower_var_fragment( mini_metadata.insert( dep_ident.clone(), crate::compiler::VariableMetadata { - offset: mini_offset, + offset: None, size: *dep_size, var: dep_var, }, ); - mini_offset += dep_size; } } @@ -926,12 +872,11 @@ pub(crate) fn lower_var_fragment( mini_metadata.insert( im_ident.clone(), crate::compiler::VariableMetadata { - offset: mini_offset, + offset: None, size: *im_size, var: im_var, }, ); - mini_offset += im_size; } } @@ -947,12 +892,6 @@ pub(crate) fn lower_var_fragment( build_submodel_metadata(&arena, db, *sub_model, project, &mut all_metadata); } - // Build the mini VariableLayout for symbolization - let mini_layout = - crate::compiler::symbolic::layout_from_metadata(&all_metadata, model_name_ident) - .unwrap_or_else(|_| VariableLayout::new(HashMap::new(), 0)); - let rmap = ReverseOffsetMap::from_layout(&mini_layout); - // Build tables for compilation -- propagate errors rather than // silently dropping them, which would shift table indices and cause // lookups to read the wrong table at runtime. @@ -988,7 +927,6 @@ pub(crate) fn lower_var_fragment( // functions. When a variable uses LOOKUP(dep, x), the dep's table // data must be in the mini-Module's tables map so the bytecode // compiler can emit the correct Lookup opcodes. - let source_vars = model.variables(db); let all_dep_names: BTreeSet<&String> = deps .dt_deps .iter() @@ -1021,8 +959,8 @@ pub(crate) fn lower_var_fragment( if tables.contains_key(&dep_canonical) { continue; } - if let Some(dep_sv) = source_vars.get(effective) { - let dep_tables = extract_tables_from_source_var(db, dep_sv, project); + if let Some(dep_sv) = resolve_var(effective) { + let dep_tables = extract_tables_from_source_var(db, &dep_sv, project); if !dep_tables.is_empty() { tables.insert(dep_canonical, dep_tables); } @@ -1041,12 +979,22 @@ pub(crate) fn lower_var_fragment( } } + // Project the owned extent view out of this model's metadata. The map + // borrows `lowered` and the arena and must stay internal; this projection + // reads only owned data (a reference and a size), never the borrowed + // variable, so it crosses back to the caller freely. It is built BEFORE the + // lowering context because both halves of the compile read it: lowering's + // GH #578 ELM MAP fold through `Context`, and emission through `ModuleCtx`. + let var_sizes: crate::db::assemble::PerVarSizes = + crate::compiler::whole_variable_extents(&all_metadata, model_name_ident); + // Build Var for each phase this variable participates in let core = crate::compiler::ContextCore { dimensions: converted_dims, dimensions_ctx: dim_context, model_name: model_name_ident, metadata: &all_metadata, + var_sizes: &var_sizes, module_models, inputs, }; @@ -1064,22 +1012,6 @@ pub(crate) fn lower_var_fragment( let initial = build_var(true); let noninitial = build_var(false); - // Project the owned (offset, size) view out of the metadata map. The - // map borrows `lowered` and the arena and must stay internal; this - // projection reads only the owned offset/size pair, never the - // borrowed variable, so it crosses back to the caller freely. - let offsets: VarOffsets = all_metadata - .iter() - .map(|(k, v)| { - ( - k.clone(), - v.iter() - .map(|(vk, vm)| (vk.clone(), (vm.offset, vm.size))) - .collect(), - ) - }) - .collect(); - LoweredVarFragment::Lowered { unit_diags, per_phase_lowered: PerPhaseLowered { @@ -1087,8 +1019,6 @@ pub(crate) fn lower_var_fragment( noninitial, }, tables, - offsets, - rmap, - mini_offset, + var_sizes, } } diff --git a/src/simlin-engine/src/lib.rs b/src/simlin-engine/src/lib.rs index 0c8fe0f6f..c2aaef704 100644 --- a/src/simlin-engine/src/lib.rs +++ b/src/simlin-engine/src/lib.rs @@ -6,8 +6,8 @@ // for unchecked array access in the hot dispatch loop. Rust's forbid() cannot // be overridden by inner #[allow] attributes (even in submodules), so deny() // is the strongest level that still permits a single opt-in. The unsafe stack -// access is proven safe by ByteCodeBuilder::finish(), which statically validates -// that compiled bytecode cannot exceed STACK_CAPACITY. +// access is proven safe by compiler::symbolic::resolve_bytecode(), which +// statically validates that compiled bytecode cannot exceed STACK_CAPACITY. #![deny(unsafe_code)] pub use prost; diff --git a/src/simlin-engine/src/model.rs b/src/simlin-engine/src/model.rs index 8fb0ba60e..133eb6b1c 100644 --- a/src/simlin-engine/src/model.rs +++ b/src/simlin-engine/src/model.rs @@ -8,30 +8,24 @@ use std::hash::Hash; use crate::ast::{Expr0, lower_ast}; use crate::common::{ Canonical, EquationError, EquationResult, Error, ErrorCode, ErrorKind, Ident, Result, - UnitError, canonicalize, + canonicalize, }; use crate::dimensions::DimensionsContext; use crate::variable::{ModuleInput, Variable, identifier_set}; use crate::{datamodel, eqn_err, model_err}; -use { - crate::common::topo_sort, crate::datamodel::Dimension, crate::var_eqn_err, crate::vm::StepPart, - std::result::Result as StdResult, -}; - #[cfg(test)] use { + crate::datamodel::Dimension, crate::db, crate::units::Context, crate::variable::{parse_var, parse_var_with_module_context}, }; #[cfg(test)] -use crate::testutils::{aux, flow, stock, x_aux, x_flow, x_model, x_module, x_stock}; +use crate::testutils::{x_aux, x_flow, x_model, x_module, x_stock}; pub type ModuleInputSet = BTreeSet>; -pub type DependencySet = BTreeSet>; -pub type DependencyMap = HashMap, BTreeSet>>; pub type VariableStage0 = Variable; @@ -56,6 +50,17 @@ pub struct ModelStage0 { pub ident: Ident, pub display_name: String, pub variables: HashMap, VariableStage0>, + /// Model-level errors recorded while building this stage: today only the + /// duplicate-canonical-ident collision (GH #891), which the canonical-keyed + /// `variables` map above would otherwise swallow last-wins. + /// + /// Read only by [`ModelStage1::new`], which copies it into + /// [`ModelStage1::errors`] -- the monolithic path's simulatability gate. + /// Production's own duplicate-ident diagnostic is a separate derivation + /// (`db::model_duplicate_variables` -> `emit_duplicate_variable_diagnostics`) + /// over the raw `declared_variable_idents` input, so the two are + /// independent, and `db::stages_tests` compares this field against the + /// salsa-free `ModelStage0::new_in_project` oracle. pub errors: Option>, /// implicit is true if this model was implicitly added to the project /// by virtue of it being in the stdlib (or some similar reason) @@ -79,18 +84,17 @@ pub struct ModelStage1 { pub name: Ident, pub display_name: String, pub variables: HashMap, Variable>, - /// Model-level errors are also accumulated via the salsa accumulator in - /// `compile_var_fragment` and `check_model_units`. This field is retained - /// because several test helpers inspect it directly. - pub errors: Option>, - /// Unit warnings are also accumulated via the salsa accumulator in - /// `check_model_units`. This field is retained for the test-only - /// `Project::from_salsa` construction path. + /// The monolithic path's simulatability gate: `compiler::Module::new` + /// refuses to build a module from a model with a non-empty list here. /// - /// Contains unit-related issues that should be surfaced to users but - /// should NOT block simulation. Unit mismatches are common in real-world - /// models and should not prevent running simulations. - pub unit_warnings: Option>, + /// Filled by [`ModelStage1::set_dependencies`] from three sources -- the + /// Stage0 duplicate-ident collision, the production dependency graph's + /// `has_cycle` verdict, and a roll-up of the equation errors the variables + /// themselves carry. It is NOT an alternative to the salsa diagnostics: it + /// is deliberately coarser (a code, no location), it exists only on this + /// test-only construction path, and everything that reports errors to a + /// user goes through `db::collect_all_diagnostics` instead. + pub errors: Option>, /// model_deps is the transitive set of model names referenced from modules in this model pub model_deps: Option>>, pub instantiations: Option>, @@ -106,6 +110,14 @@ pub struct ModelStage1 { pub macro_params: Vec>, } +/// One module instantiation's evaluation order, as the monolithic +/// `compiler::Module` consumes it. +/// +/// The three runlists are copied verbatim out of the production dependency +/// graph (`db::dep_graph::model_dependency_graph`); this type carries no +/// dependency analysis of its own. It used to also carry that graph's +/// `dt_dependencies` / `initial_dependencies` maps, whose only readers were the +/// second, now-deleted dependency walk that produced them. #[cfg_attr(feature = "debug-derive", derive(Debug))] #[derive(Clone, PartialEq, Eq)] pub struct ModuleStage2 { @@ -113,416 +125,11 @@ pub struct ModuleStage2 { /// inputs is the set of variables overridden (provided as input) in this /// module instantiation. pub inputs: ModuleInputSet, - /// initial_dependencies contains variables dependencies needed to calculate the initial values of stocks - pub initial_dependencies: HashMap, DependencySet>, - /// dt_dependencies contains the variable dependencies used during normal "dt" iterations/calculations. - pub dt_dependencies: HashMap, DependencySet>, pub runlist_initials: Vec>, pub runlist_flows: Vec>, pub runlist_stocks: Vec>, } -impl ModelStage1 { - pub(crate) fn dt_deps( - &self, - inputs: &ModuleInputSet, - ) -> Option<&HashMap, DependencySet>> { - self.instantiations - .as_ref() - .and_then(|instances| instances.get(inputs).map(|module| &module.dt_dependencies)) - } - - pub(crate) fn initial_deps( - &self, - inputs: &ModuleInputSet, - ) -> Option<&HashMap, DependencySet>> { - self.instantiations.as_ref().and_then(|instances| { - instances - .get(inputs) - .map(|module| &module.initial_dependencies) - }) - } - - /// Collect the set of variables referenced by INIT() calls across all - /// equations in this model. These variables must be included in the - /// Initials runlist so that INIT(x) can read x's initial value even - /// when x is not a stock or module. - /// - /// Parallel logic exists in db.rs variable_direct_dependencies_impl for - /// the salsa incremental path. - fn init_referenced_vars(&self) -> HashSet> { - self.variables - .values() - .filter_map(|v| v.ast()) - .flat_map(|ast| crate::variable::classify_dependencies(ast, &[], None).init_referenced) - .map(|s| Ident::new(&s)) - .collect() - } -} - -fn module_deps( - ctx: &DepContext, - var: &Variable, - is_stock: &dyn Fn(&Ident) -> bool, -) -> Vec> { - if let Variable::Module { - inputs, model_name, .. - } = var - { - if ctx.is_initial { - // A module may reference a model that is not present (an empty or - // dangling `model_name`: a freshly-drawn module or a reference to a - // deleted model). Skip its submodel deps gracefully rather than - // indexing `ctx.models` and panicking -- mirroring the guarded twin - // in `module_output_deps` (GH #806). - let Some(model) = ctx.models.get(model_name).copied() else { - return vec![]; - }; - // FIXME: do this higher up - let module_inputs = &inputs.iter().map(|mi| mi.dst.clone()).collect(); - if let Some(initial_deps) = model.initial_deps(module_inputs) { - let mut stock_deps = HashSet::>::new(); - - for var in model.variables.values() { - if let Variable::Stock { .. } = var - && let Some(deps) = initial_deps.get(var.ident()) - { - stock_deps.extend(deps.iter().cloned()); - } - } - - // During initialization, modules need their stock - // inputs initialized first (e.g. SMOOTH3 needs its - // input stock's initial value). Unlike the DT phase - // where stocks use their previous-timestep value, the - // initial phase must respect stock dependencies. - inputs - .iter() - .flat_map(|input| { - let src = &input.src; - if stock_deps.contains(&input.dst) { - let direct_dep = match src.as_str().find('.') { - Some(pos) => &src.as_str()[..pos], - None => src.as_str(), - }; - - Some(Ident::new(direct_dep)) - } else { - None - } - }) - .collect() - } else { - panic!("internal compiler error: invariant broken"); - } - } else { - inputs - .iter() - .flat_map(|r| { - let src = &r.src; - let direct_dep = match src.as_str().find('.') { - Some(pos) => &src.as_str()[..pos], - None => src.as_str(), - }; - - if is_stock(src) { - None - } else { - Some(Ident::new(direct_dep)) - } - }) - .collect() - } - } else { - unreachable!(); - } -} - -fn module_output_deps<'a>( - ctx: &DepContext, - model_name: &Ident, - output_ident: &str, - inputs: &'a [ModuleInput], - module_ident: &'a str, -) -> Result> { - if !ctx.models.contains_key(model_name) { - return model_err!(BadModelName, model_name.to_string()); - } - let model = ctx.models[model_name]; - - let module_inputs = &inputs.iter().map(|mi| mi.dst.clone()).collect(); - let deps = if ctx.is_initial { - model.initial_deps(module_inputs) - } else { - model.dt_deps(module_inputs) - }; - - if deps.is_none() { - return model_err!(Generic, output_ident.to_owned()); - } - let deps = deps.unwrap(); - if !deps.contains_key(output_ident) { - return model_err!(UnknownDependency, output_ident.to_owned()); - } - - let output_var = &model.variables[output_ident]; - let output_deps = &deps[output_ident]; - - let mut final_deps: BTreeSet<&str> = BTreeSet::new(); - - if ctx.is_initial || !output_var.is_stock() { - final_deps.insert(module_ident); - } - - for dep in output_deps.iter() { - for module_input in inputs.iter() { - if &module_input.dst == dep { - final_deps.insert(module_input.src.as_str()); - } - } - } - - Ok(final_deps) -} - -fn direct_deps(ctx: &DepContext, var: &Variable) -> Vec> { - let is_stock = |ident: &Ident| -> bool { - matches!( - resolve_relative2(ctx, ident.as_str()), - Some(Variable::Stock { .. }) - ) - }; - if var.is_module() { - module_deps(ctx, var, &is_stock) - } else { - let ast = if ctx.is_initial { - var.init_ast() - } else { - var.ast() - }; - match ast { - Some(ast) => { - let converted_dims: Vec = ctx - .dimensions - .iter() - .map(crate::dimensions::Dimension::from) - .collect(); - let classification = - crate::variable::classify_dependencies(ast, &converted_dims, ctx.module_inputs); - let mut deps = classification.all; - if !ctx.is_initial { - deps.retain(|dep| !classification.init_only.contains(dep.as_str())); - } - deps.retain(|dep| !classification.previous_only.contains(dep.as_str())); - deps - } - .into_iter() - .collect(), - None => vec![], - } - } -} - -struct DepContext<'a> { - is_initial: bool, - model_name: &'a str, // this needs to be a str, not an Ident for lifetime reasons when recursing - models: &'a HashMap, &'a ModelStage1>, - sibling_vars: &'a HashMap, Variable>, - module_inputs: Option<&'a ModuleInputSet>, - dimensions: &'a [Dimension], -} - -// to ensure we sort the list of variables in O(n*log(n)) time, we -// need to iterate over the set of variables we have and compute -// their recursive dependencies. (assuming this function runs -// in <= O(n*log(n))) -fn all_deps<'a, Iter>( - ctx: &DepContext, - vars: Iter, -) -> StdResult, EquationError)> -where - Iter: Iterator, -{ - // we need to use vars multiple times, so collect it into a Vec once - let vars = vars.collect::>(); - let mut processing: BTreeSet> = BTreeSet::new(); - let mut all_vars: HashMap<&'a str, &'a Variable> = - vars.iter().map(|v| (v.ident(), *v)).collect(); - let mut all_var_deps: HashMap, Option>>> = vars - .iter() - .map(|v| (Ident::from_str_unchecked(v.ident()), None)) - .collect(); - - fn all_deps_inner<'a>( - ctx: &DepContext, - id: &str, - processing: &mut BTreeSet>, - all_vars: &mut HashMap<&'a str, &'a Variable>, - all_var_deps: &mut HashMap, Option>>>, - ) -> StdResult<(), (Ident, EquationError)> { - let var = all_vars[id]; - - // short circuit if we've already figured this out - let canonical_id = Ident::from_str_unchecked(id); - if all_var_deps[&canonical_id].is_some() { - return Ok(()); - } - - // dependency chains break at stocks, as we use their value from the - // last dt timestep. BUT if we are calculating dependencies in the - // initial dt, then we need to treat stocks as ordinary variables. - if var.is_stock() && !ctx.is_initial { - all_var_deps.insert(canonical_id.clone(), Some(BTreeSet::new())); - return Ok(()); - } - - processing.insert(canonical_id.clone()); - - // all deps start out as the direct deps - let mut all_deps: BTreeSet> = BTreeSet::new(); - - for dep in direct_deps(ctx, var).into_iter() { - // TODO: we could potentially handle this by passing around some context - // variable, but its just terrible. - if dep.as_str().starts_with("\\·") { - let loc = var - .ast() - .unwrap() - .get_var_loc(dep.as_str()) - .unwrap_or_default(); - return var_eqn_err!( - Ident::from_str_unchecked(var.ident()), - NoAbsoluteReferences, - loc.start, - loc.end - ); - } - - // in the case of a dependency on a module output, this one dep may - // turn into several: we'll need to depend on the inputs to that module - let filtered_deps: Vec> = if dep.as_str().contains('·') { - // if the dependency was e.g. "submodel.output", do a dataflow analysis to - // figure out which of the set of (inputs + module) we depend on - let parts = dep.as_str().splitn(2, '·').collect::>(); - let module_ident = parts[0]; - let output_ident = parts[1]; - - if !all_vars.contains_key(module_ident) { - let loc = var - .ast() - .unwrap() - .get_var_loc(dep.as_str()) - .unwrap_or_default(); - return var_eqn_err!( - Ident::from_str_unchecked(var.ident()), - UnknownDependency, - loc.start, - loc.end - ); - } - - if let Variable::Module { - model_name, inputs, .. - } = all_vars[module_ident] - { - // XXX: I don't remember why we do this differently here - // and then special case modules below (end of this - // for loop) - match module_output_deps(ctx, model_name, output_ident, inputs, module_ident) { - Ok(deps) => deps.into_iter().map(Ident::from_str_unchecked).collect(), - Err(err) => { - return Err((Ident::from_str_unchecked(var.ident()), err.into())); - } - } - } else { - let loc = var - .ast() - .unwrap() - .get_var_loc(dep.as_str()) - .unwrap_or_default(); - return var_eqn_err!( - Ident::from_str_unchecked(var.ident()), - ExpectedModule, - loc.start, - loc.end - ); - } - } else { - vec![dep] - }; - - for dep in filtered_deps { - if !all_vars.contains_key(dep.as_str()) { - let loc = var - .ast() - .unwrap() - .get_var_loc(dep.as_str()) - .unwrap_or_default(); - return var_eqn_err!( - Ident::from_str_unchecked(var.ident()), - UnknownDependency, - loc.start, - loc.end - ); - } - - if ctx.is_initial || !all_vars[dep.as_str()].is_stock() { - all_deps.insert(dep.clone()); - - // ensure we don't blow the stack - if processing.contains(&dep) { - let loc = match var.ast() { - Some(ast) => ast.get_var_loc(dep.as_str()).unwrap_or_default(), - None => Default::default(), - }; - return var_eqn_err!( - Ident::from_str_unchecked(var.ident()), - CircularDependency, - loc.start, - loc.end - ); - } - - if all_var_deps[&dep].is_none() { - all_deps_inner(ctx, dep.as_str(), processing, all_vars, all_var_deps)?; - } - - // we actually don't want the module's dependencies here; - // we handled that above in module_output_deps() - if !all_vars[dep.as_str()].is_module() { - let dep_deps = all_var_deps[&dep].as_ref().unwrap(); - all_deps.extend(dep_deps.iter().cloned()); - } - } - } - } - - processing.remove(&canonical_id); - - all_var_deps.insert(canonical_id, Some(all_deps)); - - Ok(()) - } - - for var in vars { - all_deps_inner( - ctx, - var.ident(), - &mut processing, - &mut all_vars, - &mut all_var_deps, - )?; - } - - // this unwrap is safe, because of the full iteration over vars directly above - let var_deps: HashMap, BTreeSet>> = all_var_deps - .into_iter() - .map(|(k, v)| (k, v.unwrap())) - .collect(); - - Ok(var_deps) -} - fn resolve_relative<'a>( models: &'a HashMap, &'a ModelStage0>, model_name: &str, @@ -550,41 +157,6 @@ fn resolve_relative<'a>( } } -// the ident arg must be from a CanonicalIdent, but is a &str here for lifetime reasons around recursion. -fn resolve_relative2<'a>(ctx: &DepContext<'a>, ident: &'a str) -> Option<&'a Variable> { - let model_name = ctx.model_name; - let ident = if model_name == "main" && ident.starts_with('·') { - &ident['·'.len_utf8()..] - } else { - ident - }; - - let input_prefix = format!("{model_name}·"); - // TODO: this is weird to do here and not before we call into this fn - let ident = ident.strip_prefix(&input_prefix).unwrap_or(ident); - - // if the identifier is still dotted, its a further submodel reference - // TODO: this will have to change when we break `module ident == model name` - if let Some(pos) = ident.find('·') { - let submodel_name = &ident[..pos]; - let submodel_var = &ident[pos + '·'.len_utf8()..]; - let ctx = DepContext { - is_initial: ctx.is_initial, - model_name: submodel_name, - models: ctx.models, - sibling_vars: &ctx - .models - .get(&Ident::::from_str_unchecked(submodel_name))? - .variables, - module_inputs: None, - dimensions: ctx.dimensions, - }; - resolve_relative2(&ctx, submodel_var) - } else { - Some(ctx.sibling_vars.get(ident)?) - } -} - /// lower_variable takes a stage 0 variable and turns it into a stage 1 variable. /// This involves resolving both module inputs and dimension indexes. pub(crate) fn lower_variable(scope: &ScopeStage0, var_s0: &VariableStage0) -> Variable { @@ -1102,7 +674,6 @@ impl ModelStage1 { .map(|(ident, v)| (ident.clone(), lower_variable(&model_scope, v))) .collect(), errors: model_s0.errors.clone(), - unit_warnings: None, model_deps: Some(model_deps), instantiations: None, implicit: model_s0.implicit, @@ -1111,159 +682,124 @@ impl ModelStage1 { } } + /// Fill `instantiations` -- the per-module-instance evaluation order the + /// monolithic `compiler::Module` consumes -- from the production + /// dependency-graph query. + /// + /// This used to be a second, independent dependency analysis: its own + /// transitive-closure walk (`all_deps`), its own cross-model output + /// resolution, its own `CircularDependency` gate and its own topological + /// runlists. That is the divergence GH #568 tracked. The two gates did not + /// merely *risk* disagreeing -- they DID: an element-acyclic recurrence SCC + /// (`ecc[t2] = ecc[t1] + 1`) that `db::dep_graph::resolve_recurrence_sccs` + /// resolves was still a whole-variable `CircularDependency` here, so this + /// path rejected models production compiles and simulates. There is now one + /// gate, and it is the one production uses. + /// + /// The runlists are therefore the production runlists, including everything + /// the second walk did not have: the resolved-SCC contiguous blocks, the + /// dt stock-submodel-output chain break, and the `INITIAL()`-backed + /// initials seeding (GH #584). + /// + /// # Two model classes are refused, for two different reasons + /// + /// A rejected graph (`has_cycle`) records a model-level + /// `CircularDependency`. Note it does NOT get empty runlists, despite the + /// dependency MAP being emptied: `topo_sort_str` over an empty map still + /// emits every allowed name, in sorted order, so the runlists come out + /// populated and mutually incoherent (a stock's init can read a variable + /// that is absent from the initials runlist). `ModelStage1::errors` is what + /// stops `compiler::Module::new` from compiling them; the gate is not + /// redundant with an empty-runlist check, because there is no such thing. + /// + /// A RESOLVED recurrence SCC (`resolved_sccs` non-empty) is refused too, + /// with `NotSimulatable`. Unifying the gate did not unify the emitter: + /// production compiles such an SCC by interleaving its members' per-element + /// segments (`db::assemble::combine_scc_fragment`, reached only from + /// `assemble_module`), and `compiler::Module` has no equivalent -- it would + /// emit the members whole, in runlist order, so a member reads a co-member's + /// element before it is assigned. That is a silent wrong answer, not a + /// failure. Before the unification the second gate happened to refuse these + /// models by calling them circular; refusing them deliberately keeps the + /// monolith honest as an oracle, which is the only reason it still exists. + /// The distinct code matters: `CircularDependency` here would contradict + /// `project::tests::the_circular_dependency_gate_is_the_production_one`, + /// which asserts the two gates agree that this model class is NOT circular. pub(crate) fn set_dependencies( &mut self, - models: &HashMap, &ModelStage1>, - dimensions: &[Dimension], + db: &dyn crate::db::Db, + source_model: crate::db::SourceModel, + project: crate::db::SourceProject, instantiations: &BTreeSet, ) { - // used when building runlists - give us a stable order to start with - let mut var_names: Vec<&Ident> = self.variables.keys().collect(); - var_names.sort_unstable(); - - // use a Set to deduplicate problems we see in dt_deps and initial_deps - let mut var_errors: HashMap, HashSet> = HashMap::new(); // Model errors: seed with any pre-existing model-level errors recorded // at Stage0 construction (e.g. DuplicateVariable, GH #891) so this // recompute extends rather than clobbers them. `set_dependencies` runs // once per model (Project::from_salsa), so taking the list cannot // double-report. let mut errors: Vec = self.errors.take().unwrap_or_default(); + let mut has_cycle = false; + let mut has_resolved_scc = false; - let instantiations = instantiations - .iter() - .map(|instantiation| { - let mut ctx = DepContext { - is_initial: false, - model_name: self.name.as_str(), - sibling_vars: &self.variables, - models, - module_inputs: Some(instantiation), - dimensions, - }; - - let dt_deps = match all_deps(&ctx, self.variables.values()) { - Ok(deps) => Some(deps), - Err((ident, err)) => { - var_errors.entry(ident).or_default().insert(err); - None - } - }; - - ctx.is_initial = true; - - let initial_deps = match all_deps(&ctx, self.variables.values()) { - Ok(deps) => Some(deps), - Err((ident, err)) => { - var_errors.entry(ident).or_default().insert(err); - None - } - }; - - let init_referenced = self.init_referenced_vars(); - - let build_runlist = |deps: &HashMap< - Ident, - BTreeSet>, - >, - part: StepPart, - predicate: &dyn Fn(&Ident) -> bool| - -> Vec> { - let canonical_var_names: Vec> = var_names - .iter() - .filter(|id| predicate(id)) - .map(|id| (*id).clone()) - .collect(); - let runlist: Vec<&Ident> = canonical_var_names.iter().collect(); - let runlist = match part { - StepPart::Initials => { - let needed: HashSet<&Ident> = runlist - .iter() - .cloned() - .filter(|id| { - let v = &self.variables[*id]; - v.is_stock() || v.is_module() || init_referenced.contains(*id) - }) - .collect(); - let mut runlist: HashSet<&Ident> = - needed.iter().flat_map(|id| &deps[*id]).collect(); - runlist.extend(needed); - // Sort before the topo sort: the set above is a - // `HashSet`, and `topo_sort` breaks ties by visit - // order, so feeding it the raw iteration order made - // the Initials runlist -- and therefore any initial - // value depending on an unordered pair -- vary run - // to run. Same fix, same reason, as GH #595 in - // `db::dep_graph::model_dependency_graph_impl`; the - // Flows/Stocks arms are already deterministic - // because they filter the pre-sorted `var_names`. - let mut runlist: Vec<&Ident> = runlist.into_iter().collect(); - runlist.sort_unstable(); - topo_sort(runlist, deps) - } - StepPart::Flows => topo_sort(runlist, deps), - StepPart::Stocks => runlist, - }; - - let runlist: Vec> = runlist.into_iter().cloned().collect(); - - runlist - }; - - let runlist_initials = if let Some(deps) = initial_deps.as_ref() { - build_runlist(deps, StepPart::Initials, &|_| true) - } else { - vec![] - }; - - let runlist_flows = if let Some(deps) = dt_deps.as_ref() { - build_runlist(deps, StepPart::Flows, &|id| { - instantiation.contains(id) || !self.variables[id].is_stock() - }) - } else { - vec![] - }; - - let runlist_stocks = if let Some(deps) = dt_deps.as_ref() { - build_runlist(deps, StepPart::Stocks, &|id| { - let v = &self.variables[id]; - // modules need to be called _both_ during Flows and Stocks, as - // they may contain _both_ flows and Stocks - !instantiation.contains(id) && (v.is_stock() || v.is_module()) - }) - } else { - vec![] - }; + let to_idents = |names: &[String]| -> Vec> { + // The graph's runlists are canonical names by construction, so + // interning them needs no re-canonicalization scan. + names.iter().map(|n| Ident::from_str_unchecked(n)).collect() + }; + let instantiations: HashMap = instantiations + .iter() + .map(|inputs| { + let interned = crate::db::ModuleInputSet::from_canonical_set(db, inputs); + let graph = crate::db::model_dependency_graph(db, source_model, project, interned); + has_cycle |= graph.has_cycle; + has_resolved_scc |= !graph.resolved_sccs.is_empty(); ( - instantiation.clone(), + inputs.clone(), ModuleStage2 { model_ident: self.name.clone(), - inputs: instantiation.clone(), - dt_dependencies: dt_deps.unwrap_or_default(), - initial_dependencies: initial_deps.unwrap_or_default(), - runlist_initials, - runlist_flows, - runlist_stocks, + inputs: inputs.clone(), + runlist_initials: to_idents(&graph.runlist_initials), + runlist_flows: to_idents(&graph.runlist_flows), + runlist_stocks: to_idents(&graph.runlist_stocks), }, ) }) - .collect::>(); + .collect(); self.instantiations = Some(instantiations); - let mut variables_have_errors = false; - for (ident, var) in self.variables.iter_mut() { - if var_errors.contains_key(ident) { - let errors = std::mem::take(var_errors.get_mut(ident).unwrap()); - for error in errors.into_iter() { - var.push_error(error); - } - variables_have_errors = true; - } + if has_cycle { + errors.push(Error::new( + ErrorKind::Model, + ErrorCode::CircularDependency, + None, + )); + } + + // The gate is unified; the EMITTER is not. See the rustdoc above: the + // monolith cannot lower an interleaved per-element SCC, so refuse rather + // than emit members whole and read a co-member's element early. + if has_resolved_scc { + errors.push(Error::new( + ErrorKind::Model, + ErrorCode::NotSimulatable, + Some(format!( + "model '{}' contains a resolved recurrence SCC, which only \ + the per-element interleaving fragment compiler can lower", + self.name + )), + )); } - if variables_have_errors { + // Equation errors already ride on the variables themselves, recorded by + // parsing and by `lower_variable`. Roll them up to the model level so + // the `Module::new` gate still refuses a model with a broken variable. + if self + .variables + .values() + .any(|var| var.equation_errors().is_some()) + { errors.push(Error::new( ErrorKind::Model, ErrorCode::VariablesHaveErrors, @@ -1271,27 +807,19 @@ impl ModelStage1 { )); } - let maybe_errors = match errors.len() { - 0 => None, - _ => Some(errors), + self.errors = if errors.is_empty() { + None + } else { + Some(errors) }; - - self.errors = maybe_errors; } - /// Returns unit errors from variables in this model. The salsa incremental - /// path emits unit errors through `CompilationDiagnostic` accumulators; - /// prefer `db::collect_model_diagnostics` for new code. - pub fn get_unit_errors(&self) -> HashMap, Vec> { - self.variables - .iter() - .flat_map(|(ident, var)| var.unit_errors().map(|errs| (ident.clone(), errs))) - .collect() - } - - /// Returns equation errors from variables in this model. The salsa - /// incremental path emits equation errors through `CompilationDiagnostic` - /// accumulators; prefer `db::collect_model_diagnostics` for new code. + /// The equation errors this model's variables carry, keyed by variable. + /// + /// A projection of [`Variable::equation_errors`] over the model, used by + /// the roll-up above and by the tests that check it. User-facing reporting + /// goes through `db::collect_all_diagnostics`, which reports the same + /// errors with a source location attached. pub fn get_variable_errors(&self) -> HashMap, Vec> { self.variables .iter() @@ -1550,58 +1078,86 @@ fn test_module_parse() { assert_eq!(expected, actual); } +/// A variable carrying an equation error rolls up to a model-level +/// `VariablesHaveErrors`, which is what stops `compiler::Module::new` from +/// compiling a model with a broken variable. +/// +/// The errors rolled up are the ones parsing and `lower_variable` recorded on +/// the variables themselves (`Variable::errors`). `set_dependencies` used to +/// contribute a second source -- its own dependency walk's +/// `UnknownDependency` / `CircularDependency` / `ExpectedModule` -- which was +/// the GH #568 divergence; that walk is gone and its diagnostics come from the +/// production gate instead (see +/// `unknown_dependency_reaches_the_one_remaining_gate`). #[test] -fn test_errors() { +fn variable_equation_errors_roll_up_to_the_model() { let units_ctx = Context::new(&[], &Default::default()).0; - let main_model = x_model( - "main", - vec![x_aux("aux_3", "unknown_variable * 3.14", None)], - ); - let owned_models: HashMap, ModelStage0> = - vec![("main".to_string(), &main_model)] - .into_iter() - .map(|(name, m)| { - ( - Ident::new(&name), - ModelStage0::new(m, &[], &units_ctx, false), - ) - }) - .collect(); + let main_model = x_model("main", vec![x_aux("aux_3", "1 +", None)]); + let direct = ModelStage0::new(&main_model, &[], &units_ctx, false); let models: HashMap, &ModelStage0> = - owned_models.iter().map(|(k, v)| (k.clone(), v)).collect(); + std::iter::once((Ident::new("main"), &direct)).collect(); - let model = { - let no_module_inputs: ModuleInputSet = BTreeSet::new(); - let default_instantiation = [no_module_inputs].iter().cloned().collect(); - let scope = ScopeStage0 { - models: &models, - dimensions: &Default::default(), - model_name: "main", - }; - let mut model = ModelStage1::new(&scope, models[&*canonicalize("main")]); - model.set_dependencies(&HashMap::new(), &[], &default_instantiation); - model + let db = db::SimlinDb::default(); + let project_datamodel = datamodel::Project { + name: "errors".to_string(), + sim_specs: datamodel::SimSpecs::default(), + dimensions: vec![], + units: vec![], + models: vec![main_model.clone()], + source: None, + ai_information: None, }; + let sync = db::sync_from_datamodel(&db, &project_datamodel); + + let scope = ScopeStage0 { + models: &models, + dimensions: &Default::default(), + model_name: "main", + }; + let mut model = ModelStage1::new(&scope, &direct); + let no_module_inputs: ModuleInputSet = BTreeSet::new(); + let default_instantiation = [no_module_inputs].iter().cloned().collect(); + model.set_dependencies( + &db, + sync.models["main"].source, + sync.project, + &default_instantiation, + ); - assert!(model.errors.is_some()); assert_eq!( - &Error::new(ErrorKind::Model, ErrorCode::VariablesHaveErrors, None), - &model.errors.as_ref().unwrap()[0] + Some(&Error::new( + ErrorKind::Model, + ErrorCode::VariablesHaveErrors, + None + )), + model.errors.as_ref().and_then(|errs| errs.first()), ); let var_errors = model.get_variable_errors(); - assert_eq!(1, var_errors.len()); let aux_3_key = Ident::new("aux_3"); - assert!(var_errors.contains_key(&aux_3_key)); - assert_eq!(1, var_errors[&aux_3_key].len()); - let err = &var_errors[&aux_3_key][0]; assert_eq!( - &EquationError { - start: 0, - end: 16, - code: ErrorCode::UnknownDependency - }, - err + 1, + var_errors.len(), + "exactly the broken variable carries errors, got: {var_errors:?}" + ); + assert!(var_errors.contains_key(&aux_3_key)); +} + +/// The `UnknownDependency` the deleted second dependency walk used to raise is +/// still raised -- by the production gate, which is now the only one. +#[test] +fn unknown_dependency_reaches_the_one_remaining_gate() { + use crate::test_common::TestProject; + + let tp = TestProject::new("main").aux("aux_3", "unknown_variable * 3.14", None); + let errs = match tp.compile() { + Ok(_) => panic!("a model referencing an undefined variable must not compile"), + Err(errs) => errs, + }; + assert!( + errs.iter() + .any(|(loc, code)| loc == "main.aux_3" && *code == ErrorCode::UnknownDependency), + "expected a main.aux_3 UnknownDependency, got: {errs:?}" ); } @@ -1730,7 +1286,12 @@ fn test_stage0_records_duplicate_variable_error() { let mut model = ModelStage1::new(&scope, &direct); let no_module_inputs: ModuleInputSet = BTreeSet::new(); let default_instantiation = [no_module_inputs].iter().cloned().collect(); - model.set_dependencies(&HashMap::new(), &[], &default_instantiation); + model.set_dependencies( + &db, + sync.models["main"].source, + sync.project, + &default_instantiation, + ); assert!( model .errors @@ -1978,241 +1539,6 @@ fn test_collect_module_idents_skips_apply_to_all_previous() { ); } -#[test] -fn test_all_deps() { - use rand::rng; - use rand::seq::SliceRandom; - - fn verify_all_deps( - expected_deps_list: &[(&Variable, &[&str])], - is_initial: bool, - models: &HashMap, &ModelStage1>, - module_inputs: Option<&BTreeSet>>, - ) { - let default_inputs = BTreeSet::>::new(); - let expected_deps: HashMap, BTreeSet>> = - expected_deps_list - .iter() - .map(|(v, deps)| { - ( - Ident::new(v.ident()), - deps.iter().map(|s| Ident::new(s)).collect(), - ) - }) - .collect(); - - let mut all_vars: Vec = expected_deps_list - .iter() - .map(|(v, _)| (*v).clone()) - .collect(); - let ctx = DepContext { - is_initial, - model_name: "test", - models, - sibling_vars: &HashMap::new(), - module_inputs: Some(module_inputs.unwrap_or(&default_inputs)), - dimensions: &[], - }; - let deps = all_deps(&ctx, all_vars.iter()).unwrap(); - - if expected_deps != deps { - let failed_dep_order: Vec<_> = all_vars.iter().map(|v| v.ident()).collect(); - eprintln!("failed order: {failed_dep_order:?}"); - for (v, expected) in expected_deps_list.iter() { - eprintln!("{}", v.ident()); - let mut expected: Vec<_> = expected.to_vec(); - expected.sort(); - eprintln!(" expected: {expected:?}"); - let mut actual: Vec<_> = deps[&*canonicalize(v.ident())].iter().collect(); - actual.sort(); - eprintln!(" actual : {actual:?}"); - } - }; - assert_eq!(expected_deps, deps); - - let mut rng = rng(); - // no matter the order of variables in the list, we should get the same all_deps - // (even though the order of recursion might change) - for _ in 0..16 { - all_vars.shuffle(&mut rng); - let ctx = DepContext { - is_initial, - model_name: "test", - models, - sibling_vars: &HashMap::new(), - module_inputs: Some(module_inputs.unwrap_or(&default_inputs)), - dimensions: &[], - }; - let deps = all_deps(&ctx, all_vars.iter()).unwrap(); - assert_eq!(expected_deps, deps); - } - } - - let mod_1_model = x_model( - "mod_1", - vec![ - x_aux("input", "{expects to be set with module input}", None), - x_aux("output", "3 * TIME", None), - x_aux("flow", "2 * input", None), - x_stock("output_2", "input", &["flow"], &[], None), - ], - ); - - let main_model = x_model( - "main", - vec![ - x_module("mod_1", &[("aux_3", "mod_1.input")], None), - x_aux("aux_3", "6", None), - x_flow("inflow", "mod_1.flow", None), - x_aux("aux_4", "mod_1.output", None), - ], - ); - let units_ctx = Context::new(&[], &Default::default()).0; - let owned_x_models: HashMap, ModelStage0> = vec![ - ("mod_1".to_owned(), &mod_1_model), - ("main".to_owned(), &main_model), - ] - .into_iter() - .map(|(name, m)| { - ( - Ident::new(&name), - ModelStage0::new(m, &[], &units_ctx, false), - ) - }) - .collect(); - let x_models: HashMap, &ModelStage0> = - owned_x_models.iter().map(|(k, v)| (k.clone(), v)).collect(); - - let mut model_list = vec!["mod_1", "main"] - .into_iter() - .map(|name| { - let model_s0 = x_models[&*canonicalize(name)]; - let scope = ScopeStage0 { - models: &x_models, - dimensions: &Default::default(), - model_name: name, - }; - ModelStage1::new(&scope, model_s0) - }) - .collect::>(); - - let module_instantiations = { - let models = model_list.iter().map(|m| (m.name.as_str(), m)).collect(); - // FIXME: ignoring the result here because if we have errors, it doesn't really matter - enumerate_modules(&models, "main", |model| model.name.clone()).unwrap() - }; - - let models = { - let no_instantiations = BTreeSet::new(); - let mut models: HashMap, &ModelStage1> = HashMap::new(); - for model in model_list.iter_mut() { - let instantiations = module_instantiations - .get(&model.name) - .unwrap_or(&no_instantiations); - model.set_dependencies(&models, &[], instantiations); - models.insert(model.name.clone(), model); - } - models - }; - - let mut implicit_vars: Vec = Vec::new(); - let unit_ctx = crate::units::Context::new(&[], &Default::default()).0; - let mod_1_orig = &main_model.variables[0]; - assert_eq!("mod_1", mod_1_orig.get_ident()); - let mod_1 = parse_var(&[], mod_1_orig, &mut implicit_vars, &unit_ctx, |mi| { - Ok(Some(mi.clone())) - }); - let scope = ScopeStage0 { - models: &x_models, - dimensions: &Default::default(), - model_name: "main", - }; - let mod_1 = lower_variable(&scope, &mod_1); - assert!(implicit_vars.is_empty()); - let aux_3 = aux("aux_3", "6"); - let aux_4 = aux("aux_4", "mod_1.output"); - let inflow = flow("inflow", "mod_1.flow"); - let expected_deps_list: Vec<(&Variable, &[&str])> = vec![ - (&inflow, &["mod_1", "aux_3"]), - (&mod_1, &["aux_3"]), - (&aux_3, &[]), - (&aux_4, &["mod_1"]), - ]; - - verify_all_deps(&expected_deps_list, false, &models, None); - - let aux_used_in_initial = aux("aux_used_in_initial", "7"); - let aux_2 = aux("aux_2", "aux_used_in_initial"); - let aux_3 = aux("aux_3", "aux_2"); - let aux_4 = aux("aux_4", "aux_2"); - let inflow = flow("inflow", "aux_3 + aux_4"); - let outflow = flow("outflow", "stock_1"); - let stock_1 = stock("stock_1", "aux_used_in_initial", &["inflow"], &["outflow"]); - let expected_deps_list: Vec<(&Variable, &[&str])> = vec![ - (&aux_used_in_initial, &[]), - (&aux_2, &["aux_used_in_initial"]), - (&aux_3, &["aux_used_in_initial", "aux_2"]), - (&aux_4, &["aux_used_in_initial", "aux_2"]), - (&inflow, &["aux_used_in_initial", "aux_2", "aux_3", "aux_4"]), - (&outflow, &[]), - (&stock_1, &[]), - ]; - - verify_all_deps(&expected_deps_list, false, &models, None); - - // test circular references return an error and don't do something like infinitely - // recurse - let aux_a = aux("aux_a", "aux_b"); - let aux_b = aux("aux_b", "aux_a"); - let all_vars = [aux_a, aux_b]; - let ctx = DepContext { - is_initial: false, - model_name: "test", - models: &models, - sibling_vars: &HashMap::new(), - module_inputs: None, - dimensions: &[], - }; - let deps_result = all_deps(&ctx, all_vars.iter()); - assert!(deps_result.is_err()); - - // also self-references should return an error and not blow stock - let aux_a = aux("aux_a", "aux_a"); - let all_vars = [aux_a]; - let deps_result = all_deps(&ctx, all_vars.iter()); - assert!(deps_result.is_err()); - - // test initials - let expected_deps_list: Vec<(&Variable, &[&str])> = vec![ - (&aux_used_in_initial, &[]), - (&aux_2, &["aux_used_in_initial"]), - (&aux_3, &["aux_used_in_initial", "aux_2"]), - (&aux_4, &["aux_used_in_initial", "aux_2"]), - (&inflow, &["aux_used_in_initial", "aux_2", "aux_3", "aux_4"]), - (&outflow, &["stock_1", "aux_used_in_initial"]), - (&stock_1, &["aux_used_in_initial"]), - ]; - - verify_all_deps(&expected_deps_list, true, &models, None); - - let aux_if = aux( - "aux_if", - "if isModuleInput(aux_true) THEN aux_true ELSE aux_false", - ); - let aux_true = aux("aux_true", "TIME * 3"); - let aux_false = aux("aux_false", "TIME * 4"); - let expected_deps_list: Vec<(&Variable, &[&str])> = vec![ - (&aux_if, &["aux_true"]), - (&aux_true, &[]), - (&aux_false, &[]), - ]; - - let module_inputs = [Ident::new("aux_true")].iter().cloned().collect(); - verify_all_deps(&expected_deps_list, true, &models, Some(&module_inputs)); - - // test non-existant variables -} - /// Compile-time proof that the two name-keyed, pre-layout model-compilation /// stages and the plain diagnostic `Error` implement `salsa::Update`. /// @@ -2238,14 +1564,24 @@ fn test_all_deps() { /// return). Keeping the derives is cheap and forward-compatible; the point of the /// test is that nothing else forces them to stay. /// -/// The offset-keyed lowered-expression layer (`compiler::Expr` and everything -/// downstream of variable layout) must NEVER derive `salsa::Update`, even -/// though it mechanically could: those values are only meaningful relative to -/// ONE model-global slot layout, so caching one and reusing it after a layout -/// change would silently return a value keyed to a stale layout. `ModelStage0` -/// and `ModelStage1` are keyed by canonical name and built before layout, and -/// `Error` is layout-independent diagnostic data, so the derive is -/// semantically sound only for these three. +/// This used to carry a prohibition: `compiler::Expr` and "everything +/// downstream of variable layout" must never derive `salsa::Update`, because +/// those values were keyed to ONE model-global slot layout and caching one +/// across a layout change would silently return a stale address. **That premise +/// no longer holds.** Since GH #964's symbolic emission, a lowered `Expr` +/// references variables by NAME (`compiler::VarRef`) and carries no offsets at +/// all; addresses are assigned exactly once, at assembly, by +/// `symbolic::resolve_module`. `Expr` is now as layout-independent as the two +/// stages above it, and caching one across a layout change is sound -- which is +/// the whole reason a per-variable fragment survives its neighbours moving. +/// +/// The derive is still absent from `Expr`, and this test still does not assert +/// it, because nothing needs it yet: making the *lowering* itself a tracked +/// query is a separate change. The point is that the obstacle is gone, so +/// whoever wants it does not have to relitigate a soundness argument. +/// `ModelStage0` and `ModelStage1` are keyed by canonical name and built before +/// layout, and `Error` is layout-independent diagnostic data, so the derive is +/// sound for all three. #[test] fn stage_types_and_error_implement_salsa_update() { fn assert_update() {} diff --git a/src/simlin-engine/src/project.rs b/src/simlin-engine/src/project.rs index b4a25ca07..1adb38e49 100644 --- a/src/simlin-engine/src/project.rs +++ b/src/simlin-engine/src/project.rs @@ -86,10 +86,7 @@ impl Project { F: FnMut(&HashMap, &ModelStage1>, &Context, &mut ModelStage1), { use crate::common::{ErrorCode, ErrorKind, topo_sort}; - use crate::db::{ - model_stage1, project_datamodel_dims, project_dimensions_context, - project_units_context_result, - }; + use crate::db::{model_stage1, project_dimensions_context, project_units_context_result}; use crate::model::enumerate_modules; let units_result = project_units_context_result(db, source_project); @@ -115,7 +112,6 @@ impl Project { }) }) .collect(); - let dm_dims = project_datamodel_dims(db, source_project); // Read the project-global dimension context from the salsa-cached query // rather than rebuilding it here (it is canonicalized once per project). let dims_ctx = project_dimensions_context(db, source_project); @@ -130,23 +126,31 @@ impl Project { // // The values are CLONED because everything below is destructive and the // memo is shared with every other reader: `model_deps.take()` empties - // that field, `set_dependencies` fills `instantiations`, rewrites - // `errors`, and pushes equation errors onto the variables THEMSELVES, - // and `model_cb` takes `&mut ModelStage1`. A `ModelStage1` is one - // model's lowered equations, so this is the same allocation the deleted - // code paid for building them -- the saving is the parse and lowering - // work, not the copy. - let mut models_list: Vec = source_project + // that field, `set_dependencies` fills `instantiations` and rewrites + // `errors`, and `model_cb` takes `&mut ModelStage1`. A `ModelStage1` is + // one model's lowered equations, so this is the same allocation the + // deleted code paid for building them -- the saving is the parse and + // lowering work, not the copy. + // + // Each stage is kept paired with the `SourceModel` handle it came from: + // `set_dependencies` reads the production dependency graph, which is + // keyed on that handle rather than on the model's name. + let mut models_list: Vec<(crate::db::SourceModel, ModelStage1)> = source_project .models(db) .values() - .map(|src_model| model_stage1(db, *src_model, source_project).clone()) + .map(|src_model| { + ( + *src_model, + model_stage1(db, *src_model, source_project).clone(), + ) + }) .collect(); // Topo-sort by model dependencies. let model_order = { let model_deps: HashMap, BTreeSet>> = models_list .iter_mut() - .map(|model| { + .map(|(_, model)| { let deps = model.model_deps.take().unwrap(); (model.name.clone(), deps) }) @@ -167,22 +171,56 @@ impl Project { .map(|(i, n)| (n.clone(), i)) .collect::, usize>>() }; - models_list.sort_unstable_by(|a, b| model_order[&a.name].cmp(&model_order[&b.name])); + models_list + .sort_unstable_by(|(_, a), (_, b)| model_order[&a.name].cmp(&model_order[&b.name])); let module_instantiations = { - let models = models_list.iter().map(|m| (m.name.as_str(), m)).collect(); + let models = models_list + .iter() + .map(|(_, m)| (m.name.as_str(), m)) + .collect(); enumerate_modules(&models, "main", |model| model.name.clone()).unwrap_or_default() }; // Dependency resolution + model callbacks (unit inference etc.). + // + // A model that can REACH a module cycle is gated out first, exactly as + // the production entry points gate it (`db::diagnostic`'s + // `module_cycle_diagnostic`, GH #806): resolving its dependencies takes + // the recurrence-SCC refinement into the recursive `model_module_map` + // query, which salsa turns into an unrecoverable dependency-graph panic + // -- a process abort under `panic = abort`, on a project a user can + // draw. It records the cycle as this model's error and gives it no + // instantiations at all, which is what a `NotSimulatable` model looks + // like here. Scoped to REACHABILITY, so an unrelated draft cycle + // elsewhere in the project does not gate a valid model out. + // + // The `continue` also skips `model_cb`, so a cycle-reaching model gets + // no unit inference on this path. Nothing is lost today -- the only + // non-trivial callback is the `units_infer` test harness, and the + // production unit pass (`db::units::check_model_units`) is gated the + // same way one level down -- but it is a consequence of the gate, not + // an oversight, and a callback that needs to see every model would have + // to run before it. { + let module_graph = crate::db::project_module_graph(db, source_project); let no_instantiations = BTreeSet::new(); let mut models: HashMap, &ModelStage1> = HashMap::new(); - for model in models_list.iter_mut() { + for (src_model, model) in models_list.iter_mut() { + if let Some((code, message)) = module_graph.cycle_error_from(model.name.as_str()) { + model.errors.get_or_insert_with(Vec::new).push(Error::new( + ErrorKind::Model, + code, + Some(message), + )); + model.instantiations = Some(HashMap::new()); + models.insert(model.name.clone(), model); + continue; + } let instantiations = module_instantiations .get(&model.name) .unwrap_or(&no_instantiations); - model.set_dependencies(&models, dm_dims.as_slice(), instantiations); + model.set_dependencies(db, *src_model, source_project, instantiations); if !model.implicit { model_cb(&models, units_ctx, model); } @@ -192,12 +230,12 @@ impl Project { let ordered_models = models_list .iter() - .map(|m| m.name.clone()) + .map(|(_, m)| m.name.clone()) .collect::>(); let models = models_list .into_iter() - .map(|m| (m.name.clone(), Arc::new(m))) + .map(|(_, m)| (m.name.clone(), Arc::new(m))) .collect(); Project { @@ -276,15 +314,20 @@ mod tests { /// Building the same project repeatedly must produce the same model order /// and the same Initials runlists. /// - /// Two `HashMap`/`HashSet` iteration orders used to leak into `topo_sort`, + /// Two `HashMap`/`HashSet` iteration orders used to leak into a `topo_sort`, /// which breaks ties by visit order: the model runlist seeded from - /// `model_deps.keys()` here, and the Initials runlist set in - /// `ModelStage1::set_dependencies`. On two fresh `SimlinDb`s in ONE process - /// this produced `["sub_a","sub_b","main"]` on one construction and - /// `["sub_b","sub_a","main"]` on the next, with `runlist_initials` flipping - /// to match -- so an initial value depending on an unordered pair was - /// order-dependent. This is the `Project`-path twin of GH #595, fixed the - /// same way (sort before the topo sort). + /// `model_deps.keys()` here, and the Initials runlist set in the second + /// dependency walk `set_dependencies` used to run. On two fresh `SimlinDb`s + /// in ONE process this produced `["sub_a","sub_b","main"]` on one + /// construction and `["sub_b","sub_a","main"]` on the next, with + /// `runlist_initials` flipping to match -- so an initial value depending on + /// an unordered pair was order-dependent. This is the `Project`-path twin + /// of GH #595, fixed the same way (sort before the topo sort). + /// + /// The runlist half now rides on the production dependency graph's own + /// determinism (GH #568 unified the two gates), so this covers the model + /// order directly and the runlists as a consequence -- which is why it + /// still asserts on them rather than on the model order alone. /// /// Repeated across several independent constructions because each fresh /// `HashMap` gets its own `RandomState`: one agreeing pair proves nothing, @@ -396,9 +439,10 @@ mod tests { let model = x_model("main", vec![x_aux("x", "1", None), module]); let dm = x_project(sim_specs_with_units("years"), &[model]); - // Drives Project::from_datamodel -> from_salsa -> set_dependencies -> - // module_deps / topo_sort. Before the guards these panicked on the - // dangling model_name; now construction returns without crashing. + // Drives Project::from_datamodel -> from_salsa -> enumerate_modules / + // topo_sort. Before the guards a dangling `model_name` panicked here + // (and in the second dependency walk, since deleted); construction now + // returns without crashing. let _project = Project::from(dm); } @@ -444,6 +488,211 @@ mod tests { let _project = Project::from(dm); } + /// GH #568: there is one circular-dependency gate, and it is production's. + /// + /// This path used to run its own dependency walk with its own + /// `CircularDependency`, which had no element-acyclicity refinement. A + /// staged subscript-shift recurrence (`ecc[t2] = ecc[t1] + 1`) is exactly + /// the class where the two verdicts parted: `db::dep_graph` resolves the + /// SCC per element and compiles the model, while the second walk saw a + /// whole-variable self-edge and rejected it. So this path refused models + /// production simulates. + /// + /// Written as *agreement* rather than as "it compiles", in both directions: + /// if the production gate's verdict on either shape changes, the assertion + /// tracks it instead of going stale, and a re-introduced second gate reds it. + #[test] + fn the_circular_dependency_gate_is_the_production_one() { + use crate::common::ErrorCode; + use crate::db::{ModuleInputSet, SimlinDb, model_dependency_graph, sync_from_datamodel}; + use crate::test_common::TestProject; + + // (fixture, description) -- one shape the production gate resolves, one + // it rejects. + let element_acyclic = TestProject::new("main") + .named_dimension("t", &["t1", "t2", "t3"]) + .array_with_ranges( + "ecc[t]", + vec![("t1", "1"), ("t2", "ecc[t1] + 1"), ("t3", "ecc[t2] + 1")], + ); + let genuine_cycle = TestProject::new("main") + .aux("a", "b + 1", None) + .aux("b", "a + 1", None); + + for (tp, label) in [ + (element_acyclic, "element-acyclic recurrence"), + (genuine_cycle, "genuine two-variable cycle"), + ] { + let db = SimlinDb::default(); + let dm = tp.build_datamodel(); + let sync = sync_from_datamodel(&db, &dm); + let graph = model_dependency_graph( + &db, + sync.models["main"].source, + sync.project, + ModuleInputSet::empty(&db), + ); + + let monolithic = Project::from(dm); + let monolithic_rejects = monolithic.models[&Ident::new("main")] + .errors + .as_ref() + .is_some_and(|errs| errs.iter().any(|e| e.code == ErrorCode::CircularDependency)); + + assert_eq!( + graph.has_cycle, monolithic_rejects, + "the two paths must agree on whether the {label} is circular \ + (production gate: {}, monolithic path: {monolithic_rejects})", + graph.has_cycle + ); + } + } + + /// Unifying the cycle GATE did not unify the EMITTER, so the monolith must + /// refuse a resolved recurrence SCC rather than mis-compile it. + /// + /// Production lowers such an SCC by cutting each member's bytecode into + /// per-element segments and interleaving them along the SCC's verified + /// element order (`db::assemble::combine_scc_fragment`, reached only from + /// `assemble_module`). `compiler::Module` has no equivalent: it emits each + /// member whole, in runlist order, so `ce`'s second element reads `ecc[0]` + /// before `ecc` has run at all. That produces `ce = [1,1,1] / ecc = [2,2,2]` + /// where the model means `[1,3,5] / [2,4,6]` -- a silent wrong answer. + /// + /// Before GH #568 the second dependency walk refused these models by calling + /// them circular, so the hazard was covered by accident. It is now refused + /// on purpose, because an oracle that answers wrongly is worse than one that + /// declines -- and this is precisely the model class the element-acyclicity + /// refinement exists for, so it is exactly where a wrong oracle would hide a + /// real regression. + /// + /// The negative half is the load-bearing one: an ordinary model must still + /// build, or "refuses everything" would pass this test. + #[test] + fn build_module_refuses_a_resolved_recurrence_scc() { + use crate::test_common::TestProject; + + let resolved_scc = TestProject::new("main") + .named_dimension("t", &["t1", "t2", "t3"]) + .array_with_ranges( + "ce[t]", + vec![("t1", "1"), ("t2", "ecc[t1] + 1"), ("t3", "ecc[t2] + 1")], + ) + .array_with_ranges( + "ecc[t]", + vec![ + ("t1", "ce[t1] + 1"), + ("t2", "ce[t2] + 1"), + ("t3", "ce[t3] + 1"), + ], + ); + + // The production gate accepts it -- this is NOT a circular model, which + // is what makes the monolith's inability to lower it a silent hazard + // rather than a shared refusal. + { + use crate::db::{ + ModuleInputSet, SimlinDb, model_dependency_graph, sync_from_datamodel, + }; + let db = SimlinDb::default(); + let dm = resolved_scc.build_datamodel(); + let sync = sync_from_datamodel(&db, &dm); + let graph = model_dependency_graph( + &db, + sync.models["main"].source, + sync.project, + ModuleInputSet::empty(&db), + ); + assert!(!graph.has_cycle, "fixture must not be circular"); + assert!( + !graph.resolved_sccs.is_empty(), + "fixture must actually produce a resolved SCC, or this test \ + pins nothing" + ); + } + + let err = match resolved_scc.build_module() { + Ok(_) => panic!( + "the monolithic path must refuse a resolved recurrence SCC: it \ + cannot lower the per-element interleave and would emit a \ + program that reads a co-member's element before assigning it" + ), + Err(err) => err, + }; + assert!( + err.contains("NotSimulatable"), + "expected a NotSimulatable refusal, got: {err}" + ); + + // Control: an ordinary model still builds, so the refusal above is + // attributable to the SCC rather than to a blanket rejection. + let ordinary = TestProject::new("main") + .aux("k", "10", None) + .aux("derived", "k * 3", None); + assert!( + ordinary.build_module().is_ok(), + "a model with no resolved SCC must still build a monolithic module" + ); + } + + /// A module cycle whose members ALSO contain a variable-level cycle must + /// not abort the process. + /// + /// Since GH #568 this path resolves dependencies through the production + /// dependency graph, and that query's recurrence-SCC refinement descends + /// into the recursive `model_module_map` -- which salsa turns into an + /// unrecoverable dependency-graph cycle panic on a cyclic module graph + /// (GH #806), i.e. a process abort under `panic = abort`, from a public + /// `From` on a project a user can draw. The production + /// entry points have always gated reaching models out first; this path now + /// does too, and reports the module cycle as the model's error. + /// + /// The variable cycle in `middle` is load-bearing: without it the dep-graph + /// query never reaches the refinement and the shape does not panic, which is + /// why the plain module-cycle test above did not catch this. + #[test] + fn from_salsa_module_cycle_with_variable_cycle_does_not_abort() { + use crate::common::ErrorCode; + use crate::testutils::{sim_specs_with_units, x_aux, x_model, x_module_named, x_project}; + + let main = x_model( + "main", + vec![ + x_aux("driver", "1", None), + x_module_named("mid", "middle", &[("driver", "mid.input")], None), + ], + ); + let middle = x_model( + "middle", + vec![ + x_aux("input", "0", None), + x_aux("a", "b + 1", None), + x_aux("b", "a + 1", None), + x_module_named("other", "leaf", &[("input", "other.input")], None), + ], + ); + let leaf = x_model( + "leaf", + vec![ + x_aux("input", "0", None), + x_module_named("back", "middle", &[("input", "back.input")], None), + ], + ); + let dm = x_project(sim_specs_with_units("years"), &[main, middle, leaf]); + let project = Project::from(dm); + + for name in ["main", "middle", "leaf"] { + let model = &project.models[&Ident::new(name)]; + assert!( + model.errors.as_ref().is_some_and(|errs| errs + .iter() + .any(|e| e.code == ErrorCode::CircularDependency)), + "every model reaching the module cycle must record it, {name} did not: {:?}", + model.errors + ); + } + } + /// GH #891: the legacy `from_salsa` path builds each model's variable map /// from the canonical-keyed salsa sync maps, where two variables whose /// names canonicalize identically already collapsed last-wins. The @@ -480,7 +729,9 @@ mod tests { // The TestProject::compile surface (the main test-helper consumer of // this path) must report the failure rather than compiling a silently - // different model. + // different model. It reports the SALSA diagnostic, which attributes + // the collision to the surviving variable rather than to the model as a + // whole -- so the location is `main.net_flow`, not `main`. let result = crate::test_common::TestProject::new("dup") .aux("net flow", "1", None) .aux("net_flow", "2", None) @@ -491,8 +742,8 @@ mod tests { }; assert!( errs.iter() - .any(|(loc, code)| loc == "main" && *code == ErrorCode::DuplicateVariable), - "expected a main-model DuplicateVariable, got: {errs:?}" + .any(|(loc, code)| loc == "main.net_flow" && *code == ErrorCode::DuplicateVariable), + "expected a main.net_flow DuplicateVariable, got: {errs:?}" ); } } diff --git a/src/simlin-engine/src/test_common.rs b/src/simlin-engine/src/test_common.rs index 3078d6762..9b8451af7 100644 --- a/src/simlin-engine/src/test_common.rs +++ b/src/simlin-engine/src/test_common.rs @@ -9,6 +9,8 @@ use crate::common::{Canonical, ErrorCode, Ident, UnitError}; use crate::datamodel::{self, Dimension, Equation, Project, SimSpecs, Variable}; +#[cfg(test)] +use crate::db::sync_from_datamodel; use crate::db::{ DiagnosticError, DiagnosticSeverity, SimlinDb, collect_all_diagnostics, compile_project_incremental, sync_from_datamodel_incremental, @@ -541,79 +543,81 @@ impl TestProject { /// expressions via `Module::get_flow_exprs`). #[cfg(test)] impl TestProject { - /// Build and compile the project via `Project::from`. - pub fn compile(&self) -> Result> { - use std::sync::Arc; - + /// Build the monolithic `Project`, or the `Error`-severity salsa + /// diagnostics that stop it, as `(location, code)` pairs. + /// + /// The error set comes from `collect_all_diagnostics` rather than from the + /// monolithic path's own embedded error fields. Those fields are a strict + /// subset -- they never carried the unknown-dependency, bare-lookup-table + /// or assembly diagnostics, and their cycle gate was a second + /// implementation that disagreed with production's (GH #568) -- so this + /// helper used to report a model as broken that production compiles, and + /// as fine when production rejects it. + /// + /// `location` is `model.variable` for a variable-attributed diagnostic, the + /// model name for a model-level one, and `"project"` for the project-level + /// ones (the macro-registry build error and the unit definition errors, + /// which name no model). The `Project` is built against the SAME database + /// the diagnostics came from, so the datamodel is synced once. + fn compile_checked(&self) -> Result> { let datamodel = self.build_datamodel(); - let compiled = Arc::new(crate::project::Project::from(datamodel)); - - let mut errors = Vec::new(); + let db = SimlinDb::default(); + let sync = sync_from_datamodel(&db, &datamodel); - if !compiled.errors.is_empty() { - for err in &compiled.errors { - errors.push(("project".to_string(), err.code)); - } - } - - for (model_name, model) in &compiled.models { - if let Some(model_errors) = &model.errors { - for err in model_errors { - errors.push((model_name.to_string(), err.code)); - } - } - - for (var_name, var_errors) in model.get_variable_errors() { - for err in var_errors { - errors.push((format!("{model_name}.{var_name}"), err.code)); - } - } - - for (var_name, unit_errors) in model.get_unit_errors() { - for err in unit_errors { - let code = match err { + let errors: Vec<(String, ErrorCode)> = collect_all_diagnostics(&db, sync.project) + .iter() + .filter(|d| d.severity == DiagnosticSeverity::Error) + .map(|d| { + let location = match (d.model.as_str(), d.variable.as_deref()) { + ("", _) => "project".to_string(), + (model, Some(var)) => format!("{model}.{var}"), + (model, None) => model.to_string(), + }; + let code = match &d.error { + DiagnosticError::Equation(eq_err) => eq_err.code, + DiagnosticError::Model(err) => err.code, + DiagnosticError::Unit(unit_err) => match unit_err { UnitError::DefinitionError(eq_err, _) => eq_err.code, - UnitError::ConsistencyError(code, _, _) => code, - UnitError::InferenceError { code, .. } => code, - }; - errors.push((format!("{model_name}.{var_name}"), code)); - } - } + UnitError::ConsistencyError(code, _, _) => *code, + UnitError::InferenceError { code, .. } => *code, + }, + DiagnosticError::Assembly(_) => ErrorCode::NotSimulatable, + }; + (location, code) + }) + .collect(); + if !errors.is_empty() { + return Err(errors); } - if errors.is_empty() { - Ok(Arc::try_unwrap(compiled).unwrap_or_else(|arc| (*arc).clone())) - } else { - Err(errors) - } + Ok(crate::project::Project::from_salsa( + datamodel, + &db, + sync.project, + |_models, _units_ctx, _model| {}, + )) + } + + /// Build and compile the project, reporting the diagnostics production + /// would report; see [`TestProject::compile_checked`]. + pub fn compile(&self) -> Result> { + self.compile_checked() } /// Build a Module for testing lowered expressions. pub fn build_module(&self) -> Result { use crate::common::Canonical; use std::collections::BTreeSet; - use std::sync::Arc; - - let datamodel = self.build_datamodel(); - let compiled = Arc::new(crate::project::Project::from(datamodel)); - - if !compiled.errors.is_empty() { - return Err(format!( - "Project has compilation errors: {:?}", - compiled.errors - )); - } + let compiled = self + .compile_checked() + .map_err(|errors| format!("Project has compilation errors: {errors:?}"))?; let main_ident = Ident::::from_str_unchecked("main"); let model = compiled .models .get(&main_ident) .ok_or_else(|| "Model 'main' not found in compiled project".to_string())?; - if model.errors.is_some() { - return Err(format!("Model has errors: {:?}", model.errors)); - } - let inputs: BTreeSet> = BTreeSet::new(); crate::compiler::Module::new(&compiled, model.clone(), &inputs, true) .map_err(|e| format!("Failed to create module: {e:?}")) diff --git a/src/simlin-engine/src/testutils.rs b/src/simlin-engine/src/testutils.rs index 4764952b6..8ee274bf5 100644 --- a/src/simlin-engine/src/testutils.rs +++ b/src/simlin-engine/src/testutils.rs @@ -3,8 +3,6 @@ // Version 2.0, that can be found in the LICENSE file. use crate::datamodel::{self, ModuleReference}; -use crate::model::{ScopeStage0, lower_variable}; -use crate::variable::{Variable, parse_var}; #[cfg(test)] fn optional_vec(slice: &[&str]) -> Vec { @@ -26,24 +24,6 @@ pub(crate) fn x_aux(ident: &str, eqn: &str, units: Option<&str>) -> datamodel::V }) } -#[cfg(test)] -pub(crate) fn aux(ident: &str, eqn: &str) -> Variable { - let var = x_aux(ident, eqn, None); - let mut implicit_vars: Vec = Vec::new(); - let unit_ctx = crate::units::Context::new(&[], &Default::default()).0; - let var = parse_var(&[], &var, &mut implicit_vars, &unit_ctx, |mi| { - Ok(Some(mi.clone())) - }); - assert!(var.equation_errors().is_none()); - assert!(implicit_vars.is_empty()); - let scope = ScopeStage0 { - models: &Default::default(), - dimensions: &Default::default(), - model_name: "main", - }; - lower_variable(&scope, &var) -} - #[cfg(test)] pub(crate) fn x_stock( ident: &str, @@ -66,24 +46,6 @@ pub(crate) fn x_stock( }) } -#[cfg(test)] -pub(crate) fn stock(ident: &str, eqn: &str, inflows: &[&str], outflows: &[&str]) -> Variable { - let var = x_stock(ident, eqn, inflows, outflows, None); - let mut implicit_vars: Vec = Vec::new(); - let unit_ctx = crate::units::Context::new(&[], &Default::default()).0; - let var = parse_var(&[], &var, &mut implicit_vars, &unit_ctx, |mi| { - Ok(Some(mi.clone())) - }); - assert!(var.equation_errors().is_none()); - assert!(implicit_vars.is_empty()); - let scope = ScopeStage0 { - models: &Default::default(), - dimensions: &Default::default(), - model_name: "main", - }; - lower_variable(&scope, &var) -} - #[cfg(test)] pub(crate) fn x_model(ident: &str, variables: Vec) -> datamodel::Model { datamodel::Model { @@ -174,24 +136,6 @@ pub(crate) fn x_flow(ident: &str, eqn: &str, units: Option<&str>) -> datamodel:: }) } -#[cfg(test)] -pub(crate) fn flow(ident: &str, eqn: &str) -> Variable { - let var = x_flow(ident, eqn, None); - let mut implicit_vars: Vec = Vec::new(); - let unit_ctx = crate::units::Context::new(&[], &Default::default()).0; - let var = parse_var(&[], &var, &mut implicit_vars, &unit_ctx, |mi| { - Ok(Some(mi.clone())) - }); - assert!(var.equation_errors().is_none()); - assert!(implicit_vars.is_empty()); - let scope = ScopeStage0 { - models: &Default::default(), - dimensions: &Default::default(), - model_name: "main", - }; - lower_variable(&scope, &var) -} - #[cfg(test)] pub(crate) fn sim_specs_with_units(time_units: &str) -> crate::datamodel::SimSpecs { crate::datamodel::SimSpecs { diff --git a/src/simlin-engine/src/variable.rs b/src/simlin-engine/src/variable.rs index a46db1065..7b23ae585 100644 --- a/src/simlin-engine/src/variable.rs +++ b/src/simlin-engine/src/variable.rs @@ -83,13 +83,11 @@ pub enum Variable { inflows: Vec>, outflows: Vec>, non_negative: bool, - /// Legacy field from the monolithic compilation path. In the salsa incremental - /// path, equation errors are emitted via `CompilationDiagnostic` accumulators - /// instead of being stored here. New code should use `db::collect_model_diagnostics`. + /// How parsing and lowering report a failure on this variable; see the + /// note on [`Variable::equation_errors`]. errors: Vec, - /// Legacy field from the monolithic compilation path. In the salsa incremental - /// path, unit errors are emitted via `CompilationDiagnostic` accumulators - /// instead of being stored here. New code should use `db::collect_model_diagnostics`. + /// How parsing reports a malformed `` string on this variable; + /// see the note on [`Variable::unit_errors`]. unit_errors: Vec, }, Var { @@ -102,13 +100,11 @@ pub enum Variable { non_negative: bool, is_flow: bool, is_table_only: bool, - /// Legacy field from the monolithic compilation path. In the salsa incremental - /// path, equation errors are emitted via `CompilationDiagnostic` accumulators - /// instead of being stored here. New code should use `db::collect_model_diagnostics`. + /// How parsing and lowering report a failure on this variable; see the + /// note on [`Variable::equation_errors`]. errors: Vec, - /// Legacy field from the monolithic compilation path. In the salsa incremental - /// path, unit errors are emitted via `CompilationDiagnostic` accumulators - /// instead of being stored here. New code should use `db::collect_model_diagnostics`. + /// How parsing reports a malformed `` string on this variable; + /// see the note on [`Variable::unit_errors`]. unit_errors: Vec, }, Module { @@ -117,13 +113,11 @@ pub enum Variable { model_name: Ident, units: Option, inputs: Vec, - /// Legacy field from the monolithic compilation path. In the salsa incremental - /// path, equation errors are emitted via `CompilationDiagnostic` accumulators - /// instead of being stored here. New code should use `db::collect_model_diagnostics`. + /// How parsing and lowering report a failure on this variable; see the + /// note on [`Variable::equation_errors`]. errors: Vec, - /// Legacy field from the monolithic compilation path. In the salsa incremental - /// path, unit errors are emitted via `CompilationDiagnostic` accumulators - /// instead of being stored here. New code should use `db::collect_model_diagnostics`. + /// How parsing reports a malformed `` string on this variable; + /// see the note on [`Variable::unit_errors`]. unit_errors: Vec, }, } @@ -212,9 +206,27 @@ impl Variable { matches!(self, Variable::Module { .. }) } - /// Returns equation errors stored in the legacy monolithic-path fields. - /// In the salsa incremental path, errors are emitted as `CompilationDiagnostic` - /// accumulators rather than stored here; prefer `db::collect_model_diagnostics`. + /// The equation errors parsing and lowering recorded on this variable. + /// + /// **This is a live error channel, not residue from the monolithic + /// compiler.** `parse_var` writes an equation's parse errors here and + /// `model::lower_variable` appends the errors `lower_ast` raises, because + /// both produce a `Variable` and have nowhere else to put a failure. The + /// salsa path READS it: `db::var_fragment::lower_var_fragment` turns each + /// entry into a `Diagnostic`, at two sites. The read of the LOWERED + /// variable is the one nothing else covers -- drop it and every + /// `MismatchedDimensions` disappears + /// (`db::diagnostic_tests::variable_error_fields_are_the_lowering_channel` + /// is the standing gate). The read of the PARSED variable sees a strict + /// subset, since `lower_variable` clones the parse errors forward, but it + /// is where the conveyor/queue driven-flow `EmptyEquation` suppression + /// applies, so dropping it turns a spec-sanctioned empty equation into a + /// phantom error (`db::diagnostic_tests`' + /// `test_conveyor_driven_flow_empty_equation_suppressed` and its two + /// siblings). + /// + /// So `db::collect_model_diagnostics` is not an ALTERNATIVE source for + /// these -- it is the same errors, downstream of this field. pub fn equation_errors(&self) -> Option> { let errors = match self { Variable::Stock { errors, .. } @@ -228,9 +240,13 @@ impl Variable { } } - /// Returns unit errors stored in the legacy monolithic-path fields. - /// In the salsa incremental path, errors are emitted as `CompilationDiagnostic` - /// accumulators rather than stored here; prefer `db::collect_model_diagnostics`. + /// The malformed-``-string errors parsing recorded on this variable. + /// + /// Live for the same reason as [`Variable::equation_errors`]: `parse_var` + /// is where a unit string is parsed, and `lower_var_fragment` reads this + /// field to emit the non-fatal `DiagnosticError::Unit` rows. Unit + /// *consistency* mismatches are a different pass (`db::units`) and never + /// land here -- nothing appends to this field after parsing. pub fn unit_errors(&self) -> Option> { let errors = match self { Variable::Stock { unit_errors, .. } @@ -244,28 +260,6 @@ impl Variable { } } - /// Appends an equation error to the legacy monolithic-path storage on this variable. - /// The monolithic path (non-salsa) uses mutation to collect errors after parsing; - /// the salsa path accumulates them via `CompilationDiagnostic` instead. - pub fn push_error(&mut self, err: EquationError) { - match self { - Variable::Stock { errors, .. } - | Variable::Var { errors, .. } - | Variable::Module { errors, .. } => errors.push(err), - } - } - - /// Appends a unit error to the legacy monolithic-path storage on this variable. - /// The monolithic path (non-salsa) uses mutation to collect errors after unit checking; - /// the salsa path accumulates them via `CompilationDiagnostic` instead. - pub fn push_unit_error(&mut self, err: UnitError) { - match self { - Variable::Stock { unit_errors, .. } - | Variable::Var { unit_errors, .. } - | Variable::Module { unit_errors, .. } => unit_errors.push(err), - } - } - pub fn table(&self) -> Option<&Table> { match self { Variable::Stock { .. } => None, diff --git a/src/simlin-engine/src/vm.rs b/src/simlin-engine/src/vm.rs index fc38b77a7..d3f29926c 100644 --- a/src/simlin-engine/src/vm.rs +++ b/src/simlin-engine/src/vm.rs @@ -421,10 +421,16 @@ impl Stack { #[inline(always)] fn push(&mut self, value: f64) { debug_assert!(self.top < STACK_CAPACITY, "stack overflow"); - // SAFETY: ByteCodeBuilder::finish() statically validates that the max - // stack depth of all compiled bytecode is < STACK_CAPACITY, so this - // bound cannot be exceeded at runtime. The debug_assert serves as a - // belt-and-suspenders check during development. + // SAFETY: compiler::symbolic::resolve_bytecode() statically validates + // that the max stack depth of all compiled bytecode is < STACK_CAPACITY, + // so this bound cannot be exceeded at runtime. That function is the ONLY + // place concrete bytecode is produced, so nothing can reach the VM + // without passing the check; it reports over-depth as a compile Err + // rather than aborting, so an unchecked program is not executed, it is + // rejected. (The check lived in the per-fragment `ByteCodeBuilder` until + // GH #964 made codegen emit symbolic bytecode; it did not go away with + // the builder.) The debug_assert serves as a belt-and-suspenders check + // during development. unsafe { *self.data.get_unchecked_mut(self.top) = value; } @@ -434,10 +440,13 @@ impl Stack { fn pop(&mut self) -> f64 { debug_assert!(self.top > 0, "stack underflow"); self.top -= 1; - // SAFETY: ByteCodeBuilder::finish() validates via checked_sub that no - // opcode sequence pops more values than have been pushed (i.e. the stack - // depth never goes negative). This guarantees top > 0 before every pop - // at runtime. The debug_assert is a belt-and-suspenders check. + // SAFETY: compiler::symbolic::resolve_bytecode() validates via + // ByteCode::max_stack_depth's checked_sub that no opcode sequence pops + // more values than have been pushed (i.e. the stack depth never goes + // negative). Producing concrete bytecode is the only way to reach the + // VM, so this guarantees top > 0 before every pop at runtime; an + // underflow is reported as a compile Err and the program never runs. + // The debug_assert is a belt-and-suspenders check. unsafe { *self.data.get_unchecked(self.top) } } #[inline(always)] @@ -649,7 +658,7 @@ fn collect_stock_offsets( let mut offsets = Vec::new(); for op in module.compiled_stocks.code.iter() { match op { - Opcode::AssignNext { off } | Opcode::BinOpAssignNext { off, .. } => { + Opcode::BinOpAssignNext { off, .. } => { offsets.push(base_off + *off as usize); } Opcode::EvalModule { id, .. } => { @@ -2024,10 +2033,6 @@ impl Vm { curr[module_off + *off as usize] = stack.pop(); debug_assert_eq!(0, stack.len()); } - Opcode::AssignNext { off } => { - next[module_off + *off as usize] = stack.pop(); - debug_assert_eq!(0, stack.len()); - } // === SUPERINSTRUCTIONS === Opcode::AssignConstCurr { off, literal_id } => { curr[module_off + *off as usize] = bytecode.literals[*literal_id as usize]; @@ -2315,24 +2320,6 @@ impl Vm { // ========================================================= // VIEW STACK OPERATIONS // ========================================================= - Opcode::PushVarView { - base_off, - dim_list_id, - } => { - let (n_dims, dim_ids) = context.get_dim_list(*dim_list_id); - let n = n_dims as usize; - let dims: SmallVec<[u16; 4]> = (0..n) - .map(|i| context.dimensions[dim_ids[i] as usize].size) - .collect(); - let dim_id_vec: SmallVec<[DimId; 4]> = dim_ids[..n].iter().copied().collect(); - let view = RuntimeView::for_var( - (module_off + *base_off as usize) as u32, - dims, - dim_id_vec, - ); - view_stack.push(view); - } - Opcode::PushTempView { temp_id, dim_list_id, @@ -4729,7 +4716,7 @@ mod superinstruction_tests { #[test] fn test_fused_binop_next_sub() { - // stock with only outflow exercises Sub in AssignNext + // stock with only outflow exercises Sub inside the stock update let tp = TestProject::new("fused_next_sub") .with_sim_time(0.0, 3.0, 1.0) .flow("outflow", "5", None) diff --git a/src/simlin-engine/src/wasmgen/lower.rs b/src/simlin-engine/src/wasmgen/lower.rs index 2685d21b2..64c458150 100644 --- a/src/simlin-engine/src/wasmgen/lower.rs +++ b/src/simlin-engine/src/wasmgen/lower.rs @@ -13,7 +13,7 @@ //! values live in one flat f64 "slab" in linear memory, addressed by slot //! offset. A model runs over two chunks at a time -- `curr` (the values at the //! current timestep) and `next` (the values being computed for the following -//! timestep). `LoadVar` reads from `curr`; `AssignCurr`/`AssignNext` store into +//! timestep). `LoadVar` reads from `curr`; `AssignCurr`/`BinOpAssignNext` store into //! `curr`/`next`. //! //! Each `Opcode` lowers to a short, mostly 1:1 wasm instruction sequence over @@ -21,22 +21,30 @@ //! (`vm.rs:1257+`). //! //! Three compound assignment opcodes beyond the bare scalar set reach a -//! `CompiledSimulation` consumer, and they all lower here: +//! `CompiledSimulation` consumer, and they all lower here. All three are +//! produced in the SYMBOLIC domain and carried through `resolve_bytecode` +//! unchanged (it is strictly 1:1), so they arrive here exactly as codegen +//! emitted them: //! - `AssignConstCurr` arrives by *two* routes: `compiler::codegen` emits it -//! directly for any constant-RHS `AssignCurr` (`codegen.rs:1164`), and the -//! **peephole** pass also fuses a `LoadConstant; AssignCurr` pair into it -//! (`bytecode.rs:1830`). Either way it rides through the symbolic layer into -//! `CompiledSimulation`; every model with a constant initial/aux carries it. -//! - `BinOpAssignCurr` / `BinOpAssignNext` are *only* peephole output -//! (`bytecode.rs:1837`/`1841`, fusing `Op2; Assign{Curr,Next}`). The peephole -//! pass (`ByteCode::peephole_optimize`, run inside -//! `Module::compile`/`ByteCodeBuilder::finish`) runs per-variable-fragment in -//! the incremental pipeline *before* symbolization, so these ride through -//! too. Every scalar Euler stock integration (`stock + delta`) is one, so -//! they are part of the scalar core. +//! directly for any constant-RHS `AssignCurr`, and the **peephole** pass also +//! fuses a `LoadConstant; AssignCurr` pair into it. Every model with a +//! constant initial/aux carries one. +//! - `BinOpAssignCurr` is *only* peephole output (fusing `Op2; AssignCurr`); +//! `BinOpAssignNext` is emitted at once by +//! `SymbolicByteCodeBuilder::fuse_trailing_op2_into_assign_next`, since there +//! is no un-fused next-assign opcode to fall back to. Every scalar Euler +//! stock integration (`stock + delta`) is one, so they are part of the scalar +//! core. +//! +//! The peephole runs per variable fragment, inside +//! `compiler::symbolic::SymbolicByteCodeBuilder::finish` -- i.e. before +//! assembly rather than, as it once did, on concrete per-fragment bytecode that +//! was then symbolized. The observable result is the same opcode stream; what +//! changed (GH #964) is that there is now no concrete-bytecode stage before +//! assembly for it to run on. //! //! The late **3-address** pass (`ByteCode::fuse_three_address`) instead runs -//! only on the VM's private execution copy (`vm.rs:395-398`), so its +//! only on the VM's private execution copy (`vm.rs`, in `Vm::new`), so its //! `BinVarVar` / `AssignAddVarVarCurr` / ... family never reaches a consumer. //! //! Anything outside the supported scalar core -- an array/module/lookup opcode @@ -138,7 +146,7 @@ pub(crate) struct EmitCtx<'a> { pub final_time: f64, /// wasm local index holding this instance's `module_off` (i32). pub module_off_local: u32, - /// wasm local index of a scratch f64, used by `AssignCurr`/`AssignNext` to + /// wasm local index of a scratch f64, used by `AssignCurr`/`BinOpAssign*` to /// hold the value while the store address is pushed under it. pub scratch_local: u32, /// wasm local indices reserved for the `SetCond`/`If` condition register. @@ -1319,9 +1327,6 @@ fn emit_ops( Opcode::AssignCurr { off } => { emit_assign(ctx.curr_base, *off, ctx, f); } - Opcode::AssignNext { off } => { - emit_assign(ctx.next_base, *off, ctx, f); - } // `AssignConstCurr` reaches a `CompiledSimulation` by two routes // (see the module docstring): `compiler::codegen` emits it directly // for any constant-RHS `AssignCurr` (`codegen.rs:1164`), and the @@ -1415,21 +1420,6 @@ fn emit_ops( })?; state.view_stack.push(ViewDesc::from_static(view)); } - // `PushVarView` builds a full contiguous view over a variable array; - // the VM folds `module_off` into the base (`vm.rs:1749`), so the base - // is module-relative. - Opcode::PushVarView { - base_off, - dim_list_id, - } => { - let (dims, dim_ids) = resolve_dim_list_dims(ctx, *dim_list_id)?; - state.view_stack.push(ViewDesc::contiguous( - u32::from(*base_off), - ViewBase::CurrModuleRelative, - dims, - dim_ids, - )); - } // `PushTempView` builds a full contiguous view over a temp array // (`vm.rs:1757`). Opcode::PushTempView { @@ -1445,8 +1435,11 @@ fn emit_ops( )); } // `PushVarViewDirect` builds a contiguous view from raw dim sizes - // (dim_ids all 0), the base for a dynamic subscript (`vm.rs:1776`). - // Module-relative, like `PushVarView`. + // (dim_ids all 0), the base for a dynamic subscript. It is the only + // `CurrModuleRelative` view opcode: the VM folds the runtime `module_off` + // into the base, where `PushStaticView` bakes an absolute slot in + // and `PushTempView` addresses `temp_storage` with no `module_off` + // at all. Opcode::PushVarViewDirect { base_off, dim_list_id, @@ -2732,8 +2725,8 @@ fn view_top_mut(view_stack: &mut [ViewDesc]) -> Result<&mut ViewDesc, WasmGenErr }) } -/// Resolve a dim-list id to `(dim sizes, dim ids)` for `PushVarView`/ -/// `PushTempView`: each entry is a `DimId`, and the size comes from +/// Resolve a dim-list id to `(dim sizes, dim ids)` for `PushTempView`: +/// each entry is a `DimId`, and the size comes from /// `ctx.dimensions[DimId].size` (`vm.rs:1745`). fn resolve_dim_list_dims( ctx: &EmitCtx, diff --git a/src/simlin-engine/src/wasmgen/lower_tests.rs b/src/simlin-engine/src/wasmgen/lower_tests.rs index f671d694d..bdf1bb4d4 100644 --- a/src/simlin-engine/src/wasmgen/lower_tests.rs +++ b/src/simlin-engine/src/wasmgen/lower_tests.rs @@ -531,7 +531,7 @@ fn lowers_nested_if() { assert_eq!(eval(1.0, 0.0, 9.0, 100.0, 200.0), 200.0); } -// ── AssignCurr / AssignNext ─────────────────────────────────────────── +// ── AssignCurr / BinOpAssignNext ────────────────────────────────────── #[test] fn lowers_assign_curr_constant() { @@ -675,8 +675,12 @@ fn lowers_assign_next_euler_update() { op2(Op2::Sub), // births - deaths Opcode::LoadConstant { id: 0 }, // dt op2(Op2::Mul), // (births - deaths) * dt - op2(Op2::Add), // pop + ... - Opcode::AssignNext { off: 4 }, + // `pop + ...` and the store into next[] are one fused opcode: codegen + // emits `BinOpAssignNext` directly for every stock update. + Opcode::BinOpAssignNext { + op: Op2::Add, + off: 4, + }, ]; // pop=100, births=10, deaths=2.5 -> 100 + 7.5*0.5 = 103.75 let seed = &[(32, 100.0), (40, 10.0), (48, 2.5)]; @@ -685,14 +689,18 @@ fn lowers_assign_next_euler_update() { #[test] fn assign_next_honors_module_off() { - // With module_off=2, AssignNext{off:0} writes next[2]; next_base=4096, + // With module_off=2, BinOpAssignNext{off:0} writes next[2]; next_base=4096, // so byte 4096 + 2*8 = 4112. let ctx = ctx_with_cond_depth(0); let program = bc( - vec![7.0], + vec![3.5], vec![ Opcode::LoadConstant { id: 0 }, - Opcode::AssignNext { off: 0 }, + Opcode::LoadConstant { id: 0 }, + Opcode::BinOpAssignNext { + op: Op2::Add, + off: 0, + }, ], ); let bytes = build_module(&program, &ctx, false, 0); @@ -2199,7 +2207,8 @@ fn load_temp_dynamic_floors_fractional_index() { // `StaticArrayView::to_runtime_view`) folded per the matching VM reducer arm // (`vm.rs:2216-2309`). The view transform opcodes the production codegen does // not emit directly (it bakes constant subscripts into one `PushStaticView`) -// are exercised here on a `PushVarView` base so each `apply_*` is reduced +// are exercised here on a full-array `PushStaticView` base so each `apply_*` +// is reduced // over and checked against the VM. Reuses `TEMP_BASE` / `ctx_with_arrays` // from the Task 1 section above. // ════════════════════════════════════════════════════════════════════════ @@ -2421,13 +2430,15 @@ fn static_temp_view_honors_temp_offset() { // ── Task 1: view transform opcodes (mirror RuntimeView::apply_*) ────── // -// Build a full var view with PushVarView, apply one transform, reduce, and -// compare to the VM's RuntimeView with the same transform applied. These are -// the opcodes production codegen bakes into a single PushStaticView, so they -// are exercised here directly to pin each `apply_*` against the VM. - -/// A `ByteCodeContext` with a single dimension of `size` (DimId 0) and a -/// dim-list `[DimId 0]` (DimListId 0) for a 1-D `PushVarView`. +// Build a full var view, apply one transform, reduce, and compare to the VM's +// RuntimeView with the same transform applied. Production codegen bakes these +// transforms into a single `PushStaticView`, so the base view here is a +// `PushStaticView` too -- carrying the same real `DimId`s the VM oracle's +// `RuntimeView::for_var` is built with, which a `PushVarViewDirect` base +// (dim_ids all zero) would not. + +/// A `ByteCodeContext` with a single dimension of `size` (DimId 0), plus the +/// dim-list `[DimId 0]` some transform opcodes resolve against. fn ctx_one_dim(size: u16) -> ByteCodeContext { let mut context = ByteCodeContext::default(); let name_id = context.intern_name("D"); @@ -2436,19 +2447,39 @@ fn ctx_one_dim(size: u16) -> ByteCodeContext { context } -/// Run `PushVarView(base 0, dims) ; ; ; PopView` and -/// also build the VM `RuntimeView` the same way for the addressing oracle. +/// A full contiguous view over `curr` slots `0..product(dims)` whose shape is +/// `context`'s dim-list 0, resolved through `context.dimensions` -- i.e. the +/// view `RuntimeView::for_var(0, dims, dim_ids)` describes. A context may +/// declare dimensions the view does not span (a star-range's child dim), so +/// the dim-list, not the dimension table, is what defines the view. +fn full_var_view(context: &ByteCodeContext) -> StaticArrayView { + let (n_dims, dim_ids) = context.get_dim_list(0); + let dim_ids: Vec = dim_ids[..n_dims as usize].to_vec(); + let dims: Vec = dim_ids + .iter() + .map(|&id| { + context + .get_dimension(id) + .expect("dim-list 0 must reference declared dimensions") + .size + }) + .collect(); + let mut view = dense_view(0, &dims); + view.dim_ids = dim_ids.into_iter().collect(); + view +} + +/// Run `PushStaticView(full var view) ; ; ; PopView`. fn run_var_view_reduce( context: &ByteCodeContext, transforms: &[Opcode], reduce: Opcode, data: &[f64], ) -> f64 { - let ctx = ctx_with_arrays(context); - let mut code = vec![Opcode::PushVarView { - base_off: 0, - dim_list_id: 0, - }]; + let mut context = context.clone(); + let view_id = context.add_static_view(full_var_view(&context)); + let ctx = ctx_with_arrays(&context); + let mut code = vec![Opcode::PushStaticView { view_id }]; code.extend_from_slice(transforms); code.push(reduce); code.push(Opcode::PopView {}); @@ -2590,12 +2621,11 @@ fn dup_view_then_reduce_matches_single() { // The duplicate must leave the stack balanced for the trailing PopView; // a second PopView would underflow, so add one more here to drain the // dup and confirm both pops succeed. + let mut context = context.clone(); + let view_id = context.add_static_view(full_var_view(&context)); let ctx = ctx_with_arrays(&context); let code = vec![ - Opcode::PushVarView { - base_off: 0, - dim_list_id: 0, - }, + Opcode::PushStaticView { view_id }, Opcode::DupView {}, Opcode::ArraySum {}, Opcode::PopView {}, // pop dup diff --git a/src/simlin-engine/src/wasmgen/module.rs b/src/simlin-engine/src/wasmgen/module.rs index ddf106db0..99cda6876 100644 --- a/src/simlin-engine/src/wasmgen/module.rs +++ b/src/simlin-engine/src/wasmgen/module.rs @@ -1405,8 +1405,8 @@ fn new_opcode_fn( /// Collect absolute offsets of all stock variables across the whole simulation, /// recursing into child modules via `EvalModule` so submodule (SMOOTH/DELAY) /// stocks are included. Mirrors the VM's `collect_stock_offsets` -/// (`vm.rs:512-543`) exactly: a stock writes via `AssignNext` or its -/// peephole-fused `BinOpAssignNext` (most integrations are `stock + delta`), and +/// (`vm.rs:512-543`) exactly: a stock writes via `BinOpAssignNext` (codegen's +/// fused form of `stock + delta`, the only shape a stock update takes), and /// an `EvalModule` recurses with `base_off + decl.off` (each instance addresses /// its slot at `base_off + off`). After each step these slots are copied `next -> /// curr`; the RK loops index `rk_scratch[saved/accum]` by their sorted position. @@ -1422,7 +1422,7 @@ fn collect_all_stock_offsets( let mut offsets: Vec = Vec::new(); for op in module.compiled_stocks.code.iter() { match op { - Opcode::AssignNext { off } | Opcode::BinOpAssignNext { off, .. } => { + Opcode::BinOpAssignNext { off, .. } => { offsets.push(base_off + *off as usize); } Opcode::EvalModule { id, .. } => { diff --git a/src/simlin-engine/src/wasmgen/module_tests.rs b/src/simlin-engine/src/wasmgen/module_tests.rs index c2ab795a8..d0c6b35be 100644 --- a/src/simlin-engine/src/wasmgen/module_tests.rs +++ b/src/simlin-engine/src/wasmgen/module_tests.rs @@ -2517,7 +2517,7 @@ fn compile_simulation_reset_no_override_restores_defaults() { /// `set_value` on a non-constant offset returns the error code and does not /// write. A stock's offset (`level`) is not an overridable constant (its -/// initial is a constant, but it is assigned via `AssignNext`, not an +/// initial is a constant, but it is assigned via `BinOpAssignNext`, not an /// `AssignConstCurr` in flows), so `set_value` must reject it. After the /// rejected call the default run must be unchanged. #[test] diff --git a/src/simlin-engine/src/wasmgen/views.rs b/src/simlin-engine/src/wasmgen/views.rs index 8aef2b9cd..e55f8c4a2 100644 --- a/src/simlin-engine/src/wasmgen/views.rs +++ b/src/simlin-engine/src/wasmgen/views.rs @@ -32,9 +32,9 @@ pub(crate) enum ViewBase { /// `base_off` verbatim (no `module_off` added), so the byte address is /// `curr_base + (base_off + flat) * 8` with no runtime addend. CurrAbsolute, - /// `curr[module_off + base_off + ..]`. `PushVarView` / `PushVarViewDirect` - /// fold the runtime `module_off` into the base (`vm.rs:1749` / `1784`), so a - /// read adds `module_off * 8` to the constant address. In the current + /// `curr[module_off + base_off + ..]`. `PushVarViewDirect` folds the + /// runtime `module_off` into the base (`vm.rs`'s `PushVarViewDirect` arm), + /// so a read adds `module_off * 8` to the constant address. In the current /// single-root scope `module_off == 0`, but the distinction is preserved so /// Phase 7 can thread a real `module_off` without changing addressing. CurrModuleRelative, @@ -119,7 +119,7 @@ impl ViewDesc { /// Build a contiguous view over a full variable/temp array from a dim-list /// (the `(n_dims, sizes)` for `PushVarViewDirect`, or dim sizes resolved - /// from `ctx.dimensions` for `PushVarView`/`PushTempView`). Strides are + /// from `ctx.dimensions` for `PushTempView`). Strides are /// row-major, built right-to-left, exactly as `RuntimeView::for_var`. pub fn contiguous(base_off: u32, base: ViewBase, dims: Vec, dim_ids: Vec) -> Self { let mut strides = Vec::with_capacity(dims.len()); @@ -421,7 +421,7 @@ impl ViewDesc { /// `module_relative = false`. /// - `CurrModuleRelative`: `const = curr_base + (base_off + flat) * 8`, /// `module_relative = true` (the caller adds `module_off * 8`). The VM - /// folds `module_off` into the base at `PushVarView` time (`vm.rs:1749`); + /// folds `module_off` into the base at `PushVarViewDirect` time; /// in the single-root scope `module_off == 0`, so the read is the same as /// `CurrAbsolute` today, but the flag keeps Phase 7 correct. ///