From 89ecb91c3e24d7f9a38e1c3f2b0061310d77b32b Mon Sep 17 00:00:00 2001 From: Bobby Powers Date: Mon, 27 Jul 2026 12:49:12 -0700 Subject: [PATCH 01/17] engine: teach needs_quoting the lexer's keyword table `ast::needs_quoting` is the single "can this canonical name be spelled bare in an equation" predicate, but it tested only the XID character classes and the leading-character rule. The lexer resolves a bare word against KEYWORDS before it ever considers an identifier, so a variable legally named `"if"` -- neither XMILE nor Vensim reserves anything, and both our readers admit one with zero diagnostics -- printed BARE and no longer re-parsed as a reference to itself. A `patch` rename of an unrelated variable reprints every dependent equation through `print_ident` and persists the result, so an edit that touched neither the variable nor its reader silently rewrote a valid saved model into an unparseable one. That predicate has three consumers, one per producer of equation text, and all three were reached: `print_ident`, `ltm_augment::quote_ident`, and the MDL importer's reference formatter. The importer had its own rule that quoted only on `.`, which left every other unreadable name bare -- `data revenue k$` emitted a bare `$`, `Marg Capital'Energy per Capital` emitted a bare `'` that our lexer reads as the TRANSPOSE operator, and seven of the eight keywords imported to an `UnrecognizedToken` on every reference (all are names in the checked-in corpus). It now delegates to `needs_quoting` like the other two; the shared `quoted_space_to_underbar` is untouched, because it spells a definition-side ident rather than equation text. `nan` is excluded, and that is a disclosed residual rather than an oversight: the importer represents Vensim's `A FUNCTION OF(...)` placeholder as the stored equation text `NAN`, so quoting `nan` would bind that placeholder to any like-named variable. The writer cannot spell it differently either -- Vensim documents `A FUNCTION OF` as not intended for use in writing equations, and as precluding simulation. So a bare `nan` reference naming a declared variable still reads as the NaN literal: pre-existing, silent, narrower than the seven this fixes, pinned by a characterization test, and properly fixed by changing how the importer represents "no equation". Renaming a variable TO a name containing `"` is refused at the same front door, because it is the same class arriving through the same entry point. `Lexer::quoted_identifier` ends at the first `"` and the equation grammar has no escape, so such a name can be written neither bare nor quoted -- yet both readers will define one from ``. The rename used to return Ok and persist reprinted dependents that no longer compile. The check is on the target name only, so renaming AWAY from such a name still works and remains how an affected model is repaired; it reuses `ErrorCode::UnclosedQuotedIdent`, the code recompiling would have produced, and an `x$y` control pins that the rejection stays narrow. The new proptest states the completeness property against the LEXER rather than a second copy of the character rules, which is the check both past holes here -- leading digit, keyword -- would have failed. --- src/simlin-engine/CLAUDE.md | 5 +- src/simlin-engine/src/ast/mod.rs | 197 +++++++- src/simlin-engine/src/db/ltm_char_tests.rs | 9 +- src/simlin-engine/src/db/ltm_tests.rs | 53 +++ src/simlin-engine/src/keyword_ident_tests.rs | 454 +++++++++++++++++++ src/simlin-engine/src/lexer/mod.rs | 25 + src/simlin-engine/src/lexer/test.rs | 73 +++ src/simlin-engine/src/lib.rs | 2 + src/simlin-engine/src/ltm_augment.rs | 31 +- src/simlin-engine/src/ltm_augment_tests.rs | 56 +++ src/simlin-engine/src/mdl/CLAUDE.md | 2 +- src/simlin-engine/src/mdl/xmile_compat.rs | 72 ++- src/simlin-engine/src/patch.rs | 27 ++ 13 files changed, 974 insertions(+), 32 deletions(-) create mode 100644 src/simlin-engine/src/keyword_ident_tests.rs diff --git a/src/simlin-engine/CLAUDE.md b/src/simlin-engine/CLAUDE.md index 73eb2dcb1..6252d6ec9 100644 --- a/src/simlin-engine/CLAUDE.md +++ b/src/simlin-engine/CLAUDE.md @@ -8,9 +8,9 @@ Core simulation engine for system dynamics models. Compiles, type-checks, unit-c Equation text flows through these stages in order: -1. **`src/lexer/`** - Tokenizer for equation syntax +1. **`src/lexer/`** - Tokenizer for equation syntax. `KEYWORDS` is the equation language's reserved-word table (`if`/`then`/`else`/`not`/`mod`/`and`/`or`/`nan`; the units lexer shares it and differs only in also admitting `$` inside identifiers), and `identifierish` resolves a bare word against it BEFORE falling back to `Token::Ident`. `is_reserved_word` exposes that verdict so `ast::needs_quoting` can read this table rather than restate it -- without it a variable legally named `"if"` printed BARE and no longer re-parsed, so an unrelated `patch` rename persisted an unparseable equation (GH #976). 2. **`src/parser/`** - Recursive descent parser producing `Expr0` AST -3. **`src/ast/`** - AST type system with progressive lowering: `Expr0` (parsed) -> `Expr1` (modules expanded) -> `Expr2` (dimensions resolved) -> `Expr3` (subscripts expanded). `array_view.rs` tracks array dimensions and sparsity. `Expr2Context` trait includes `has_mapping_to()` for cross-dimension mapping lookups during `find_matching_dimension`. +3. **`src/ast/`** - AST type system with progressive lowering: `Expr0` (parsed) -> `Expr1` (modules expanded) -> `Expr2` (dimensions resolved) -> `Expr3` (subscripts expanded). `array_view.rs` tracks array dimensions and sparsity. `Expr2Context` trait includes `has_mapping_to()` for cross-dimension mapping lookups during `find_matching_dimension`. `needs_quoting` is the single "can this canonical name be spelled bare in an EQUATION" predicate -- character classes, the leading-character rule, and `lexer::is_reserved_word` -- and it has THREE consumers, one per producer of equation text: `print_ident` (the `print_eqn` path), `ltm_augment::quote_ident` (LTM's generated guard forms), and `mdl::xmile_compat::XmileFormatter::quote_reference` (the MDL importer, which writes equation text into `datamodel::Equation` that our own lexer reads back; it excludes `nan` alone, a disclosed residual explained in the MDL module doc). `print_eqn_proptest::a_bare_spellable_name_lexes_as_one_identifier` pins its completeness against the lexer itself, which is what makes the restatement checkable. The MDL *writer*'s `needs_mdl_quoting` is a DIFFERENT language's rule and deliberately separate -- as is `quoted_space_to_underbar`, which spells a definition-side *ident*, not equation text. A canonical name containing `"` is the one shape with no spelling at all (the lexer's quoted identifier has no escape); `patch::apply_rename_variable` refuses to rename TO one rather than persisting an equation that cannot be re-read. 4. **`src/builtins.rs`** - Builtin function definitions (e.g. `MIN`, `PULSE`, `LOOKUP`, `QUANTUM`, `SSHAPE`, `VECTOR SELECT`, `VECTOR ELM MAP`, `VECTOR SORT ORDER`, `VECTOR RANK`, `ALLOCATE AVAILABLE`, `ALLOCATE BY PRIORITY`, `NPV`, `MODULO`, `PREVIOUS`, `INIT`). `is_stdlib_module_function()` is the authoritative predicate for deciding whether a function name expands to a stdlib module; the **module-function** generalization that also resolves project macros lives in **`src/module_functions.rs`** -- the pure (Functional-Core) `ModuleFunctionDescriptor`/`MacroRegistry` resolver+validator unifying stdlib functions and macros (`stdlib_descriptor`, `MacroRegistry::build`/`resolve_macro`, the shared `is_renamed_opcode_intrinsic`/`is_renamed_stdlib_module_builtin`/`is_renamed_builtin_macro_collision` precedence predicates). `MacroRegistry::build` validates in four passes: duplicate macro name and macro/model name collision (macros.AC5.3), macro-to-macro recursion (macros.AC5.2), and -- Pass 4 -- a `Variable::Module` inside a macro-marked model (macros.AC5.7, `ErrorCode::MacroContainsModule`). Pass 4 is a CYCLE-SAFETY rule and is not redundant with Pass 3: `db::project_module_graph` records only EXPLICIT module edges, so a cycle `mac ->(explicit module)-> u ->(macro call, an IMPLICIT edge)-> mac` was invisible to the gate and drove the recursive queries into salsa's dependency-graph cycle panic, while Pass 3's macro-to-macro graph cannot even express the edge through the non-macro `u`. The rejection is deliberately broader than the cycle -- an acyclic module inside a macro is rejected too -- and that cost is REAL, not zero: the shape is reachable from an ordinary XMILE file (the `` content model is shared with ``, so the reader passes a `` through unfiltered), our own XMILE writer round-trips it, and an acyclic instance compiles and SIMULATES correctly today. It is rejected anyway because narrowing to only-when-cyclic needs a second reachability analysis that must agree with `project_module_graph`'s, whose back edge is the macro CALL -- discoverable only by parsing every model's equations, which is the dependency-list cost widening the graph was rejected for; and because a macro is a TEMPLATE, so instantiating a sub-model inside one is dubious on its own terms and reads as a language rule rather than a workaround. The MDL importer cannot produce it, but not because "Vensim macros have no modules": `src/mdl/convert/multi_output.rs` DOES mint a `Variable::Module` for a multi-output `:`-list invocation, and it is unreachable from a macro body only because the scoped body sub-context hard-codes an empty materialization (`src/mdl/convert/macros.rs`). A module *targeting* a macro (how multi-output invocations work) is untouched -- only a module *inside* a macro-marked model is rejected. A **genuine passthrough macro** -- a single-parameter, single-output macro whose body is exactly `out = BUILTIN(param)` self-calling its own renamed-builtin-collision name (`:MACRO: INIT(x) = INITIAL(x)`, stored after the importer's `INITIAL`->`INIT` rename as `init = init(x)`) -- is classified once at `MacroRegistry::build` time (`classify_passthrough`, the only place the body AST is available) into a `passthrough: Option` on the descriptor; `builtins_visitor.rs` reads it to *collapse the call to the builtin opcode* (`init`->`LoadInitial`) instead of expanding the buggy per-element synthetic module (#591), falling through to the same renamed-builtin intrinsic routing the #554 self-call exception takes. Strict criteria (no additional outputs, bare-parameter argument, self-call, renamed-builtin-collision name) keep it from misfiring on a near-miss like `INIT = INIT(x) + 1`. `equation_is_module_call()` (pre-scan, renamed from `equation_is_stdlib_call`) and `contains_module_call()` (walk-time, renamed from `contains_stdlib_call`) both consult the `MacroRegistry` (a passthrough caller is still classified module-backed here -- benign, since it collapses to a flat-slot variable). `builtins_visitor.rs` handles implicit module instantiation, single-output macro inlining (an arrayed macro invocation enters the per-element path), and PREVIOUS/INIT helper rewriting: unary `PREVIOUS(x)` desugars to `PREVIOUS(x, 0)`, direct scalar args compile to `LoadPrev`, and module-backed or expression args are first rewritten through synthesized scalar helper auxes. `INIT(x)` compiles to `LoadInitial`, using the same helper rewrite when needed. Tracks `module_idents` so `PREVIOUS(module_var)` never reads a multi-slot module directly. 5. **`src/compiler/`** - Multi-pass compilation to bytecode: - `mod.rs` - Orchestration; includes A2A hoisting logic that detects array-producing builtins (VectorElmMap, VectorSortOrder, Rank, AllocateAvailable, AllocateByPriority) during array expansion, hoists them into `AssignTemp` pre-computations, and emits per-element `TempArrayElement` reads. Treats a **standalone lookup-only variable** -- a graphical-function holder whose equation is empty or the legacy MDL `"0+0"` sentinel (`crate::variable::var_is_lookup_only`/`is_empty_or_sentinel`, mirroring `mdl::writer::is_lookup_only_equation`'s "empty or sentinel" rule) -- as a non-value-bearing **static table** (`Variable::Var::is_table_only`): it is excluded from every runlist and from the saved output, so it produces NO series of its own (issue #606). Its data is reached only through `LOOKUP(table, x)` call sites (resolved by ident -> `base_gf`); a *bare* reference with no argument is a compile error (`ErrorCode::LookupReferencedWithoutArgument`). WITH LOOKUP (`var = WITH LOOKUP(input, table)`: tables present *and* a real input) is NOT lookup-only -- it is a value-bearing variable that lowers to `LOOKUP(self, input)` for every equation shape (`apply_implicit_with_lookup`): per element for arrayed variables, where a per-element gf applies each element's OWN table and a gf-less element keeps its raw input equation (GH #909) @@ -218,6 +218,7 @@ Integration tests are consolidated into the single `tests/integration` harness ( - **`src/array_tests.rs`** - Array-specific tests (feature-gated) - **`src/per_element_gf_tests.rs`** - Per-element graphical-function (arrayed GF) layout invariant: a per-element GF table must land at the element's *declared dimension index* (row-major), not its `Equation::Arrayed` `elems` Vec position. Uses deliberately NON-alphabetical declared orders (where position != dim-index) so a positional mis-map is observable; covers the scalar `Lookup` path, the `LookupArray` (VECTOR SELECT) path, the MDL-importer-sort twin, sparse + multi-dim, the salsa dependency-table path (`extract_tables_from_source_var`), and a compile-time/perf structural guard that the reorder is materialized at compile time and the VM opcode carries no element name. The LTM-polarity twin lives in the `ltm` tests module (`test_per_element_gf_link_polarity_fixed_index_non_sorted_order`). - **`src/lookup_only_tests.rs`** - End-to-end tests for the standalone-lookup-only **static table** contract (#606): a lookup-only holder (scalar, A2A, arrayed) produces NO saved series; a consumer that calls it (`LOOKUP(g, x)`) reads the table, two calls at different arguments are independent, a bare reference is a compile error, and the legacy `"0+0"` form is still accepted. The pure `is_lookup_only`/`var_is_lookup_only` predicate units live in `src/variable.rs`. +- **`src/keyword_ident_tests.rs`** - End-to-end contract for a variable whose canonical name IS an equation-language keyword (GH #976). Pins the reachability premise the issue left unverified -- both the XMILE and MDL readers admit `if`/`mod`/`nan`/... as a variable name and the model compiles with zero diagnostics -- and then the consequence: a `patch` rename of an UNRELATED variable reprints every dependent equation, and the reprint must keep the quotes (bare, it re-parses as the keyword and the persisted datamodel no longer compiles). Both rename directions are covered, since renaming TO a keyword reprints the new name at every reference. - **`src/json_proptest.rs`**, **`src/json_sdai_proptest.rs`** - Property-based tests - **`src/unit_checking_test.rs`** - Unit checking regression tests - **`src/test_sir_xmile.rs`** - SIR epidemiology model integration tests diff --git a/src/simlin-engine/src/ast/mod.rs b/src/simlin-engine/src/ast/mod.rs index de97fd9d9..4520727d3 100644 --- a/src/simlin-engine/src/ast/mod.rs +++ b/src/simlin-engine/src/ast/mod.rs @@ -456,11 +456,27 @@ fn binary_op_latex(op: BinaryOp) -> BinaryOpLatex { } /// Check whether a canonicalized identifier needs double-quoting to be -/// re-parseable. Names containing characters outside XID_Start/XID_Continue -/// (like `$`, `⁚`, `/`) must be quoted -- as must a name whose FIRST character -/// is not `XID_Start` even if every character is alphanumeric (`1stock`, a legal -/// quoted XMILE name: bare, the lexer reads the number `1` then the identifier -/// `stock`). +/// re-parseable **in the equation language** (`LexerType::Equation`). Three ways +/// a name fails to be bare-spellable, one per clause below: +/// +/// * a character outside XID_Start/XID_Continue (`$`, `⁚`, `/`); +/// * a FIRST character that is not `XID_Start`, even when every character is +/// alphanumeric (`1stock`, a legal quoted XMILE name: bare, the lexer reads +/// the number `1` then the identifier `stock`); +/// * a name the lexer resolves to a KEYWORD instead of an identifier +/// ([`crate::lexer::is_reserved_word`] -- `if`, `mod`, `nan`, ...). XMILE +/// lets a modeler quote any name, so `"if"` is a legal variable and +/// canonicalization keeps it as `if`; printed bare it re-parses as the `if` +/// of a conditional (`nan` re-parses as the NaN *literal*), which is how a +/// `patch` rename silently rewrote a valid model into an unparseable one +/// (GH #976). The predicate delegates to the lexer's own table rather than +/// restating it, so printer and lexer cannot disagree about what a keyword +/// is. +/// +/// The units lexer shares that keyword table and differs only in also admitting +/// `$` inside identifiers, so this predicate is *conservative* -- never wrong -- +/// if it is ever asked about a unit expression. It is not today: every caller +/// prints equation text. /// /// `pub(crate)` because this is the single "can this name be spelled bare" /// predicate: `print_ident` uses it for the `print_eqn` path and @@ -480,7 +496,7 @@ pub(crate) fn needs_quoting(canonical: &str) -> bool { return true; } } - false + crate::lexer::is_reserved_word(canonical) } /// Canonicalize an identifier for display, re-quoting if the canonical form @@ -1543,7 +1559,8 @@ fn test_latex_printers_agree_on_if_under_an_operator() { ); } -/// `parse(print_eqn(e)) == e` over the FULL operator set. +/// `parse(print_eqn(e)) == e` over the FULL operator set and over a NAME POOL +/// that reaches every clause of [`needs_quoting`]. /// /// The MDL writer's fixpoint proptest (`mdl::writer_proptest`) re-reads with /// `mdl::parser`, whose binary precedence table is inverted (GH #914), so its @@ -1556,9 +1573,76 @@ fn test_latex_printers_agree_on_if_under_an_operator() { mod print_eqn_proptest { use super::*; use crate::common::RawIdent; - use crate::lexer::LexerType; + use crate::lexer::{LexerType, Token}; use proptest::prelude::*; + /// Identifiers reaching every clause of [`needs_quoting`], so the round-trip + /// property is sensitive to the quoting decision and not only to operator + /// placement: bare-legal names, all eight equation-language KEYWORDS + /// (GH #976), a leading-digit name (`1stock`, the case `17d4e7c0` fixed), a + /// name carrying the synthetic `⁚`/`→` characters LTM mints, and a `·` + /// module-qualified name (`XID_Continue`, so it stays bare). + /// + /// Spelled out rather than read from `lexer::KEYWORDS`: a fixture derived + /// from the table under test could not notice that table losing an entry. + const NAME_POOL: [&str; 13] = [ + "a", "b", "_c", "if", "then", "else", "not", "mod", "and", "or", "nan", "1stock", "m·out", + ]; + + /// Rewrite every identifier to its canonical form. + /// + /// `print_ident` canonicalizes as it prints, and a quoted name comes back + /// from the parser with its quotes still attached to the `RawIdent`, so RAW + /// ident equality is not the property `print_eqn` promises -- CANONICAL + /// ident equality is. On the bare names this is the identity, so the + /// operator coverage is unaffected. The match is exhaustive with no `_` arm, + /// so a new `Expr0` variant is a compile error here. + fn canonicalize_idents(expr: Expr0) -> Expr0 { + fn canon(raw: &RawIdent) -> RawIdent { + RawIdent::new(canonicalize(raw.as_str()).into_owned()) + } + match expr { + Expr0::Const(text, value, loc) => Expr0::Const(text, value, loc), + Expr0::Var(id, loc) => Expr0::Var(canon(&id), loc), + Expr0::App(UntypedBuiltinFn(func, args), loc) => Expr0::App( + UntypedBuiltinFn(func, args.into_iter().map(canonicalize_idents).collect()), + loc, + ), + Expr0::Subscript(id, args, loc) => Expr0::Subscript( + canon(&id), + args.into_iter().map(canonicalize_index_idents).collect(), + loc, + ), + Expr0::Op1(op, l, loc) => Expr0::Op1(op, Box::new(canonicalize_idents(*l)), loc), + Expr0::Op2(op, l, r, loc) => Expr0::Op2( + op, + Box::new(canonicalize_idents(*l)), + Box::new(canonicalize_idents(*r)), + loc, + ), + Expr0::If(c, t, f, loc) => Expr0::If( + Box::new(canonicalize_idents(*c)), + Box::new(canonicalize_idents(*t)), + Box::new(canonicalize_idents(*f)), + loc, + ), + } + } + + fn canonicalize_index_idents(index: IndexExpr0) -> IndexExpr0 { + match index { + IndexExpr0::Wildcard(loc) => IndexExpr0::Wildcard(loc), + IndexExpr0::StarRange(dim, loc) => { + IndexExpr0::StarRange(RawIdent::new(canonicalize(dim.as_str()).into_owned()), loc) + } + IndexExpr0::Range(l, r, loc) => { + IndexExpr0::Range(canonicalize_idents(l), canonicalize_idents(r), loc) + } + IndexExpr0::DimPosition(n, loc) => IndexExpr0::DimPosition(n, loc), + IndexExpr0::Expr(e) => IndexExpr0::Expr(canonicalize_idents(e)), + } + } + /// Every `BinaryOp`, so a new variant cannot be silently skipped: the match is /// exhaustive and adding a variant is a compile error here. fn all_binary_ops() -> Vec { @@ -1577,7 +1661,7 @@ mod print_eqn_proptest { fn expr_strategy() -> impl Strategy { let leaf = prop_oneof![ - prop::sample::select(&["a", "b", "c"][..]) + prop::sample::select(&NAME_POOL[..]) .prop_map(|n| Expr0::Var(RawIdent::new_from_str(n), Loc::default())), prop::sample::select(&[0.0f64, 1.0, 2.5][..]).prop_map(|v| Expr0::Const( format!("{v}"), @@ -1624,6 +1708,74 @@ mod print_eqn_proptest { }) } + /// Does `text` lex as ONE identifier covering the whole input? + /// + /// This is the lexer-side statement of "bare-spellable", derived by running + /// the lexer rather than by restating its character classes -- which is the + /// point: `needs_quoting` restates them, and the restatement was incomplete + /// (GH #976). + fn lexes_as_one_whole_ident(text: &str) -> bool { + let mut lexer = crate::lexer::Lexer::new(text, LexerType::Equation); + match (lexer.next(), lexer.next()) { + (Some(Ok((start, Token::Ident(word), end))), None) => { + start == 0 && end == text.len() && word == text + } + _ => false, + } + } + + /// Names in the shapes canonicalization can produce, plus arbitrary short + /// strings over the alphabet those shapes draw from. `"` is excluded, and + /// [`a_canonical_name_containing_a_quote_is_unspellable`] says why. + fn name_strategy() -> impl Strategy { + prop_oneof![ + prop::sample::select(&NAME_POOL[..]).prop_map(str::to_string), + "[a-zA-Z0-9_·⁚$→]{1,6}", + ] + } + + /// The one name shape `needs_quoting` cannot rescue, pinned rather than + /// quietly excluded from the property above. + /// + /// `canonicalize` preserves an embedded `"` (an XMILE `name="a"b"` + /// reaches the compiler as the canonical `a"b`), and `Lexer::quoted_ident` + /// terminates on the FIRST `"` with no escape sequence of any kind -- so + /// `a"b` has no bare spelling AND no quoted spelling. `needs_quoting` + /// correctly says "quote it"; there is simply nothing to say. + /// + /// Such a name IS reachable -- the XMILE reader admits + /// `` with zero diagnostics -- and a rename TO one + /// used to persist a corrupted model through the same `patch.rs` path as + /// GH #976: the rename reprints every dependent equation, so `c = a + 1` + /// became `c = "x"y" + 1` and the previously-valid model stopped compiling + /// with `UnclosedQuotedIdent`. That direction is now refused at the front + /// door by `patch::apply_rename_variable`, which is where the loudness + /// belongs; giving the name a spelling instead would mean an escape in the + /// lexer's quoted-identifier rule, a language change and not a printer one. + /// + /// So what remains is exactly this: a name that can be DEFINED (by either + /// reader) but never REFERENCED. This test is the characterization pin for + /// that state, and it reds if the lexer ever grows an escape -- which is + /// when `print_ident` needs revisiting. + #[test] + fn a_canonical_name_containing_a_quote_is_unspellable() { + let canonical = canonicalize("a\"b"); + assert_eq!( + "a\"b", + canonical.as_ref(), + "the quote survives canonicalization" + ); + assert!(needs_quoting(&canonical)); + assert!(!lexes_as_one_whole_ident(&canonical), "no bare spelling"); + + let printed = print_ident(&canonical); + assert_eq!("\"a\"b\"", printed); + assert!( + !lexes_as_one_whole_ident(&printed), + "no quoted spelling either: the lexer has no escape inside a quoted ident" + ); + } + proptest! { #[test] fn print_eqn_roundtrips_over_the_full_operator_set(expr in expr_strategy()) { @@ -1634,12 +1786,35 @@ mod print_eqn_proptest { "print_eqn output did not re-parse: {printed:?} ({reparsed:?})" ); prop_assert_eq!( - expr.clone().strip_loc(), - reparsed.unwrap().unwrap().strip_loc(), + canonicalize_idents(expr.clone()).strip_loc(), + canonicalize_idents(reparsed.unwrap().unwrap()).strip_loc(), "print_eqn output re-parsed to a DIFFERENT AST: {}", printed ); } + + /// The completeness guard for [`needs_quoting`]: a name it calls + /// bare-spellable must ACTUALLY lex as one identifier. Stating it against + /// the lexer (rather than against a second copy of the character rules) + /// is what makes the predicate checkable: every past hole here -- + /// leading digit, keyword -- was a clause the printer never knew about. + /// + /// The converse is deliberately not asserted. Over-quoting is always + /// safe, and `ltm_augment::quote_ident` relies on that to keep quoting + /// `·`-qualified names the lexer would happily read bare. + #[test] + fn a_bare_spellable_name_lexes_as_one_identifier(name in name_strategy()) { + let canonical = canonicalize(&name); + prop_assume!(!canonical.is_empty()); + if !needs_quoting(&canonical) { + prop_assert!( + lexes_as_one_whole_ident(&canonical), + "needs_quoting says `{}` can be spelled bare, but the lexer does not \ + read it as a single identifier", + canonical + ); + } + } } } diff --git a/src/simlin-engine/src/db/ltm_char_tests.rs b/src/simlin-engine/src/db/ltm_char_tests.rs index f4f117405..cfaac9672 100644 --- a/src/simlin-engine/src/db/ltm_char_tests.rs +++ b/src/simlin-engine/src/db/ltm_char_tests.rs @@ -1619,9 +1619,12 @@ fn scalar_feeder_bare_in_arrayed_reducer_compiles_and_simulates() { // Verified: re-introducing the `quote_ident` regression leaves every other // fixture green and fails THIS one, via the `AllCompile` expectation. // -// Deliberately a leading DIGIT and not a keyword: a keyword-named source would -// fail today against open GH #976 (`ast::needs_quoting` has no keyword check), -// which is out of scope here. This fixture must be green on correct code. +// A leading DIGIT rather than a keyword, and one fixture rather than two: since +// GH #976 both classes are decided by the SAME `ast::needs_quoting` delegation, +// so a second golden would re-test the same branch of `quote_ident` with a +// different input. The keyword class is pinned where the distinction actually +// lives -- `ltm_tests::link_score_quotes_every_keyword_named_source`, which +// ranges over the whole keyword table and asserts the generated arm PARSES. // --------------------------------------------------------------------------- fn unquotable_source_name_model() -> datamodel::Project { diff --git a/src/simlin-engine/src/db/ltm_tests.rs b/src/simlin-engine/src/db/ltm_tests.rs index 7d4eb0859..d9511612e 100644 --- a/src/simlin-engine/src/db/ltm_tests.rs +++ b/src/simlin-engine/src/db/ltm_tests.rs @@ -620,6 +620,59 @@ fn link_score_quotes_a_canonical_name_that_cannot_be_bare() { ); } +/// The keyword twin of the test above (GH #976), and the LTM half of that +/// issue's damage. +/// +/// `if` is a legal XMILE variable name that canonicalizes to `if`, but the lexer +/// resolves a bare `if` to the keyword before it ever considers an identifier. +/// `quote_ident` tested "every char is alphanumeric or `_`" plus +/// `ast::needs_quoting`, and a keyword satisfied BOTH -- so the generated guard +/// form spelled the source bare, the arm did not parse, the fragment compiled to +/// no bytecode, and the link score read a constant 0 behind a Warning nobody has +/// to look at. The keyword clause now lives in `needs_quoting`, so `quote_ident` +/// inherits it through the delegation it already had. +/// +/// Ranges over every keyword rather than sampling `if`: the predicate reads a +/// table, and a table is exactly the thing that can be right for one entry. +#[test] +fn link_score_quotes_every_keyword_named_source() { + for keyword in ["if", "then", "else", "not", "mod", "and", "or", "nan"] { + let project = TestProject::new("keyword_ident") + .stock(keyword, "0", &["inflow"], &[], None) + .flow("inflow", &format!("\"{keyword}\" * 0.1"), None) + .build_datamodel(); + + let db = SimlinDb::default(); + let sync = sync_from_datamodel(&db, &project); + let model = sync.models["main"].source; + + let link_id = LtmLinkId::new(&db, keyword.to_string(), "inflow".to_string()); + let scored = + link_score_equation_text_shaped(&db, link_id, RefShape::Bare, model, sync.project); + let ShapedLinkScore::Scored(lsv) = scored else { + panic!("the {keyword} -> inflow link score should be scored, got: {scored:?}"); + }; + + let crate::db::LtmEquation::Scalar(arm) = &lsv.equation else { + panic!( + "a scalar target's link score should be Scalar, got: {:?}", + lsv.name + ); + }; + assert!( + arm.text.contains(&format!("\"{keyword}\"")), + "a keyword-named source must be quoted in the generated text, got: {}", + arm.text + ); + assert!( + arm.expr.is_some(), + "the generated link-score equation must PARSE (an unparseable arm carries no \ + AST, compiles to no bytecode, and silently zeroes the score); text was: {}", + arm.text + ); + } +} + /// An unparseable generated arm degrades LOUDLY and WITHOUT PANICKING, for the /// scalar shape AND for an `Arrayed` equation whose OTHER arms parse fine. /// diff --git a/src/simlin-engine/src/keyword_ident_tests.rs b/src/simlin-engine/src/keyword_ident_tests.rs new file mode 100644 index 000000000..0977c64b1 --- /dev/null +++ b/src/simlin-engine/src/keyword_ident_tests.rs @@ -0,0 +1,454 @@ +// Copyright 2026 The Simlin Authors. All rights reserved. +// Use of this source code is governed by the Apache License, +// Version 2.0, that can be found in the LICENSE file. + +//! End-to-end contract for a variable whose canonical name collides with an +//! equation-language KEYWORD (`if`, `then`, `else`, `not`, `mod`, `and`, `or`, +//! `nan` -- `lexer::KEYWORDS`). +//! +//! Neither XMILE nor Vensim reserves those words as variable names, and +//! canonicalization preserves them, so such a variable reaches the compiler +//! intact from an ordinary file. But `lexer::identifierish` resolves a bare word +//! against the keyword table BEFORE falling back to `Ident`, so the variable can +//! only ever be *referenced* double-quoted. Every producer of equation text has +//! to agree with the lexer about that; the single predicate that decides it is +//! `ast::needs_quoting`, which had no keyword clause until GH #976. +//! +//! What that cost: `patch`'s rename reprints every dependent equation through +//! `print_eqn`, so an unrelated rename dropped the quotes and wrote unparseable +//! text back into the persisted `datamodel::Project` -- a silently-corrupted +//! save, the worst outcome in this codebase. The tests below pin the whole +//! chain: both readers admit the name, the printer quotes it, a rename keeps the +//! model compiling, and the corrupting spelling never appears in stored text. + +use crate::common::Ident; +use crate::datamodel; +use crate::db::{ + SimlinDb, collect_all_diagnostics, compile_project_incremental, sync_from_datamodel_incremental, +}; +use crate::patch::{ModelOperation, ModelPatch, ProjectPatch, apply_patch}; + +/// The equation-language keywords in their `name=` spelling. +/// +/// Deliberately a literal list rather than a read of `lexer::KEYWORDS`: these +/// tests are the independent side of the contract, and a fixture derived from +/// the table under test could not notice that table losing an entry. +const KEYWORD_NAMES: [&str; 8] = ["if", "then", "else", "not", "mod", "and", "or", "nan"]; + +fn xmile_doc(vars: &str) -> String { + format!( + r#" + +
ttt
+ + 01
1
+ + {vars} +"# + ) +} + +/// The issue's model: `if = 3`, `b = "if" * 2`, `c = b + 1`. +const KEYWORD_MODEL_VARS: &str = r#" + 3 + "if" * 2 + b + 1"#; + +fn read_xmile(vars: &str) -> datamodel::Project { + crate::compat::open_xmile(&mut xmile_doc(vars).as_bytes()).expect("XMILE must parse") +} + +/// Diagnostics production would report for `project`, via the incremental path. +fn diagnostics(project: &datamodel::Project) -> Vec { + let mut db = SimlinDb::default(); + let sync = sync_from_datamodel_incremental(&mut db, project, None); + collect_all_diagnostics(&db, sync.project) +} + +fn assert_compiles_clean(project: &datamodel::Project, what: &str) { + let diags = diagnostics(project); + assert!( + diags.is_empty(), + "{what} must compile with no diagnostics, got: {diags:?}" + ); + + let mut db = SimlinDb::default(); + let sync = sync_from_datamodel_incremental(&mut db, project, None); + compile_project_incremental(&db, sync.project, "main") + .unwrap_or_else(|e| panic!("{what} must compile: {e:?}")); +} + +/// Reachability, XMILE side: the reader admits every keyword as a variable name +/// and hands the canonical name through unchanged. +#[test] +fn xmile_reader_admits_every_keyword_as_a_variable_name() { + for name in KEYWORD_NAMES { + let project = read_xmile(&format!(r#"3"#)); + let idents: Vec<&str> = project.models[0] + .variables + .iter() + .map(|v| v.get_ident()) + .collect(); + assert_eq!( + vec![name], + idents, + "canonicalization must preserve the keyword-shaped name" + ); + } +} + +/// Run `project` and return the final value of `var`. +fn final_value(project: &datamodel::Project, var: &str) -> f64 { + let mut vm = crate::queue_compile::build_vm(project, "main").expect("model must build"); + vm.run_to_end().expect("simulation must run"); + let results = vm.into_results(); + let offset = *results + .offsets + .get(var) + .unwrap_or_else(|| panic!("no results column for `{var}`")); + results + .iter() + .next_back() + .unwrap_or_else(|| panic!("`{var}` has an empty timeseries"))[offset] +} + +/// Reachability, MDL side: Vensim reserves none of our keywords either, so the +/// importer has to emit a reference to one QUOTED. +/// +/// The bare spelling is the one that matters and the one that was broken +/// (GH #976, review finding F1): a modeler writing plain Vensim does not quote, +/// because Vensim has nothing to quote against, so every reference to such a +/// variable imported to an `UnrecognizedToken`. Both spellings are covered here, +/// and the numeric assertion is what distinguishes "compiles" from "computes the +/// right thing". +/// +/// `nan` is excluded and handled by +/// [`a_bare_nan_reference_in_mdl_is_still_the_literal`]: it is the one keyword +/// the importer deliberately does NOT quote, because the stored text `NAN` is +/// also how we represent Vensim's `A FUNCTION OF` placeholder. Enumerated as an +/// explicit exclusion rather than a shortened list so the asymmetry is visible. +#[test] +fn mdl_reader_admits_a_keyword_named_variable() { + for keyword in KEYWORD_NAMES.into_iter().filter(|k| *k != "nan") { + for reference in [keyword.to_string(), format!("\"{keyword}\"")] { + let mdl = format!( + "{keyword} = 3\n\t~\t\n\t~\t|\n\n\ + b = {reference} * 2\n\t~\t\n\t~\t|\n\n\ + c = b + 1\n\t~\t\n\t~\t|\n\n\ + \\\\\\---/// Sketch information - do not modify anything except names\n" + ); + let project = crate::compat::open_vensim(&mdl) + .unwrap_or_else(|e| panic!("MDL must parse ({keyword}, {reference}): {e:?}")); + + let mut idents: Vec<&str> = project.models[0] + .variables + .iter() + .map(|v| v.get_ident()) + .collect(); + idents.sort_unstable(); + let mut expected = vec!["b", "c", keyword]; + expected.sort_unstable(); + assert_eq!(expected, idents); + + assert_compiles_clean( + &project, + &format!("the MDL-imported model referencing `{reference}`"), + ); + assert_eq!( + 6.0, + final_value(&project, "b"), + "`b = {reference} * 2` must be 6, not a bare-keyword misreading" + ); + } + } +} + +/// The residual: a bare `nan` reference naming a declared variable STILL reads +/// as the NaN literal. Known, pre-existing, silent, and deliberately not fixed +/// here. +/// +/// Every other keyword is quoted by the importer now, so `b = mod * 2` resolves +/// to the variable. `nan` cannot join them, and the reason is a representation +/// choice two layers away: our importer stores Vensim's `A FUNCTION OF(...)` -- +/// "this variable has no equation" -- as the equation text `NAN`, and the MDL +/// writer prints that back as a bare `NaN`. Quoting `nan` here would bind that +/// placeholder to any variable named `nan`, so a round-tripped model would +/// compute a value for a variable that has none. That is a strictly worse trade +/// than the one this test pins. +/// +/// Nor can the writer fix it. Vensim's documentation says `A FUNCTION OF` "is +/// not intended for use in writing equations, and precludes simulation", so +/// emitting the marker in place of a bare `NaN` produces a file Vensim cannot +/// run at all. There is no writer-side spelling that resolves this. +/// +/// Fixing it properly means changing how the importer represents "no equation" +/// -- its own piece of work, not this one. This test exists so the residual is +/// recorded rather than rediscovered; it asserts today's WRONG behavior on +/// purpose, and reds if that ever changes. +#[test] +fn a_bare_nan_reference_in_mdl_is_still_the_literal() { + let mdl = concat!( + "nan = 3\n\t~\t\n\t~\t|\n\n", + "b = nan * 2\n\t~\t\n\t~\t|\n\n", + "\\\\\\---/// Sketch information - do not modify anything except names\n" + ); + let project = crate::compat::open_vensim(mdl).expect("MDL must parse"); + + // Compiles clean either way, which is what makes the case silent: the + // diagnostics channel cannot see it, only the number can. + assert_compiles_clean(&project, "the bare-`nan` MDL model"); + assert!( + final_value(&project, "b").is_nan(), + "KNOWN RESIDUAL: `b = nan * 2` still reads the NaN literal rather than \ + the declared variable `nan` (3). If this now computes 6, the importer's \ + `nan` exclusion was removed -- check that the `A FUNCTION OF` \ + placeholder round trip still works before accepting the change." + ); + + // The contrast that makes the exclusion narrow: every other keyword IS + // resolved to the variable. + let mdl_mod = concat!( + "mod = 3\n\t~\t\n\t~\t|\n\n", + "b = mod * 2\n\t~\t\n\t~\t|\n\n", + "\\\\\\---/// Sketch information - do not modify anything except names\n" + ); + let project = crate::compat::open_vensim(mdl_mod).expect("MDL must parse"); + assert_compiles_clean(&project, "the bare-`mod` MDL model"); + assert_eq!(6.0, final_value(&project, "b")); +} + +/// What the `nan` exclusion buys: an `A FUNCTION OF()` placeholder survives +/// `project_to_mdl` -> re-import as a placeholder, **including in a model that +/// also declares a variable named `nan`**. +/// +/// That combination is the reason `nan` is excluded from the importer's keyword +/// quoting. The writer spells a placeholder as a bare `NaN`, so quoting `nan` +/// would bind it to the like-named variable and `marketing` would compute `3` +/// instead of `NaN`, with zero diagnostics. One model can be both a declarer of +/// `nan` and a carrier of a placeholder, so no rule that inspects the model can +/// separate them -- and per Vensim's documentation `A FUNCTION OF` "precludes +/// simulation", so the writer cannot emit the marker instead. Excluding the one +/// keyword is the only move that costs nothing here. +/// +/// The no-`nan` control is what keeps this from passing vacuously: it is the arm +/// that works regardless, so only the declared arm can catch a regression. +#[test] +fn a_placeholder_round_trips_even_when_nan_names_a_variable() { + let with_nan = concat!( + "nan = 3\n\t~\t\n\t~\t|\n\n", + "marketing = A FUNCTION OF( nan )\n\t~\t\n\t~\t|\n\n", + "\\\\\\---/// Sketch information - do not modify anything except names\n" + ); + let without_nan = concat!( + "other = 3\n\t~\t\n\t~\t|\n\n", + "marketing = A FUNCTION OF( other )\n\t~\t\n\t~\t|\n\n", + "\\\\\\---/// Sketch information - do not modify anything except names\n" + ); + + for (label, src) in [("declares `nan`", with_nan), ("control", without_nan)] { + let first = crate::compat::open_vensim(src).expect("MDL must parse"); + assert_eq!( + Some(datamodel::Equation::Scalar("NAN".to_string())), + placeholder_equation(&first), + "{label}: the placeholder must import as the NaN placeholder" + ); + assert!( + final_value(&first, "marketing").is_nan(), + "{label}: a variable with no equation must evaluate to NaN" + ); + + let written = crate::mdl::project_to_mdl(&first).expect("MDL must write"); + let second = crate::compat::open_vensim(&written).expect("re-import must parse"); + // Compared case-insensitively: the placeholder arrives as `NAN` from the + // importer's `A FUNCTION OF` arm and comes back as `NaN` from the + // writer's `Const` text. Both lex to the NaN literal, and that -- not + // the spelling -- is the property. The value assertion below is the one + // that would catch the reference binding the review found. + let round_tripped = placeholder_equation(&second).expect("placeholder must survive"); + let datamodel::Equation::Scalar(text) = &round_tripped else { + panic!("{label}: expected a scalar equation, got {round_tripped:?}"); + }; + assert!( + text.eq_ignore_ascii_case("nan"), + "{label}: the placeholder must survive the round trip as the NaN \ + literal, not as a reference; got {text:?}" + ); + assert!( + final_value(&second, "marketing").is_nan(), + "{label}: the round-tripped placeholder must still be NaN, not the \ + value of a like-named variable" + ); + } +} + +fn placeholder_equation(project: &datamodel::Project) -> Option { + project.models[0] + .variables + .iter() + .find(|v| v.get_ident() == "marketing") + .and_then(|v| v.get_equation().cloned()) +} + +/// The keyword-named model compiles clean from XMILE too, and computes the +/// right number. A loud rejection here would be a wrong rejection: the name is +/// legal in both source formats. +#[test] +fn a_keyword_named_variable_compiles_clean() { + let project = read_xmile(KEYWORD_MODEL_VARS); + assert_compiles_clean(&project, "the keyword-named model"); + + let mut db = SimlinDb::default(); + let sync = sync_from_datamodel_incremental(&mut db, &project, None); + let compiled = + compile_project_incremental(&db, sync.project, "main").expect("model must compile"); + assert!( + compiled.offsets.contains_key(&Ident::new("if")), + "the keyword-named variable must reach the compiled model" + ); + + // `if = 3`, `b = "if" * 2`, `c = b + 1`: the numbers are what separate + // "compiles" from "means what the modeler wrote". + assert_eq!(6.0, final_value(&project, "b")); + assert_eq!(7.0, final_value(&project, "c")); +} + +/// GH #976's verified repro. Renaming an UNRELATED variable (`b` -> `bee`) +/// reprints every dependent equation through `expr2_to_string` -> `print_ident`. +/// With no keyword clause in `needs_quoting` the reference to `if` came back +/// bare, so the patched -- and PERSISTED -- datamodel held `bee = if * 2`, which +/// no longer parses: an edit that touched neither `if` nor its reader silently +/// destroyed a valid saved model. +#[test] +fn renaming_an_unrelated_variable_preserves_a_keyword_reference() { + let mut project = read_xmile(KEYWORD_MODEL_VARS); + assert_compiles_clean(&project, "the pre-rename model"); + + apply_patch( + &mut project, + ProjectPatch { + project_ops: vec![], + models: vec![ModelPatch { + name: "main".to_string(), + ops: vec![ModelOperation::RenameVariable { + from: "b".to_string(), + to: "bee".to_string(), + }], + }], + }, + ) + .expect("rename must apply"); + + assert_eq!( + Some(datamodel::Equation::Scalar("\"if\" * 2".to_string())), + project.models[0] + .variables + .iter() + .find(|v| v.get_ident() == "bee") + .and_then(|v| v.get_equation().cloned()), + "the reprinted equation must keep the quotes the lexer requires" + ); + assert_compiles_clean(&project, "the renamed model"); +} + +/// The mirror direction: renaming a variable TO a keyword. Every equation that +/// referenced it is reprinted with the new name, so the new name is the one that +/// has to come out quoted. +#[test] +fn renaming_a_variable_to_a_keyword_keeps_the_model_compiling() { + for name in KEYWORD_NAMES { + let mut project = read_xmile( + r#" + 3 + a + 1"#, + ); + + apply_patch( + &mut project, + ProjectPatch { + project_ops: vec![], + models: vec![ModelPatch { + name: "main".to_string(), + ops: vec![ModelOperation::RenameVariable { + from: "a".to_string(), + to: name.to_string(), + }], + }], + }, + ) + .expect("rename must apply"); + + assert_eq!( + Some(datamodel::Equation::Scalar(format!("\"{name}\" + 1"))), + project.models[0] + .variables + .iter() + .find(|v| v.get_ident() == "c") + .and_then(|v| v.get_equation().cloned()), + "a reference to the renamed `{name}` must be quoted" + ); + assert_compiles_clean(&project, &format!("the model renamed to `{name}`")); + } +} + +/// The one name shape a rename must REFUSE (review finding F2). +/// +/// A canonical name containing `"` has no spelling at all -- the lexer's quoted +/// identifier ends at the first `"` and there is no escape -- so a rename to one +/// used to return `Ok(())` and persist `c = "x"y" + 1`, turning a valid saved +/// model into one that no longer compiles. Same class and same entry point as +/// GH #976, so it gets the same treatment, at the front door instead of in the +/// printer: nothing is lost by refusing a name that nothing could reference. +/// +/// The `x$y` control is what keeps the rejection narrow: `$` is equally +/// unspellable BARE, but it has a working quoted spelling, so it must still be +/// allowed and must still compile. +#[test] +fn renaming_to_an_unspellable_name_is_refused() { + const MODEL: &str = r#" + 3 + a + 1"#; + + let rename_to = |target: &str| { + let mut project = read_xmile(MODEL); + let result = apply_patch( + &mut project, + ProjectPatch { + project_ops: vec![], + models: vec![ModelPatch { + name: "main".to_string(), + ops: vec![ModelOperation::RenameVariable { + from: "a".to_string(), + to: target.to_string(), + }], + }], + }, + ); + (result, project) + }; + + let (result, project) = rename_to("x\"y"); + let err = result.expect_err("a rename to a `\"`-bearing name must be refused"); + assert_eq!(crate::common::ErrorCode::UnclosedQuotedIdent, err.code); + assert_compiles_clean(&project, "the model a refused rename left alone"); + assert!( + project.models[0] + .variables + .iter() + .any(|v| v.get_ident() == "a"), + "a refused rename must not have partially applied" + ); + + // Control: `$` is unspellable bare but spellable quoted, so it is allowed + // and the reprinted reference compiles. + let (result, project) = rename_to("x$y"); + result.expect("a quotable name must still be renameable"); + assert_eq!( + Some(datamodel::Equation::Scalar("\"x$y\" + 1".to_string())), + project.models[0] + .variables + .iter() + .find(|v| v.get_ident() == "c") + .and_then(|v| v.get_equation().cloned()), + ); + assert_compiles_clean(&project, "the model renamed to `x$y`"); +} diff --git a/src/simlin-engine/src/lexer/mod.rs b/src/simlin-engine/src/lexer/mod.rs index e1758e1c5..23b587444 100644 --- a/src/simlin-engine/src/lexer/mod.rs +++ b/src/simlin-engine/src/lexer/mod.rs @@ -86,6 +86,31 @@ const KEYWORDS: &[(&str, Token<'static>)] = &[ ("nan", Nan), ]; +/// Does `word`, written bare, lex as a keyword rather than as an identifier? +/// +/// [`Lexer::identifierish`] resolves a bare word against [`KEYWORDS`] BEFORE +/// falling back to `Token::Ident`, so a variable whose canonical name is one of +/// these can never be *referenced* bare -- it has to be double-quoted, even +/// though XMILE happily lets a modeler name a variable `"if"`. This is the +/// lexer's half of [`crate::ast::needs_quoting`], exposed so the printer reads +/// this table instead of duplicating it. +/// +/// Case-insensitive because `identifierish` lowercases before the lookup, so the +/// verdict must not depend on the caller having canonicalized first -- and it +/// lowercases the same way, per Unicode `char` rather than per ASCII byte, so +/// the two cannot disagree on any input. Comparing char-wise makes that +/// equivalent to `identifierish`'s `word.to_lowercase() == keyword` while +/// allocating nothing on the printer path; the one place `str::to_lowercase` +/// differs from the char-wise mapping is the Greek final-sigma rule, and a +/// lowercase form containing `ς` matches no keyword either way. +pub(crate) fn is_reserved_word(word: &str) -> bool { + KEYWORDS.iter().any(|(keyword, _)| { + word.chars() + .flat_map(char::to_lowercase) + .eq(keyword.chars()) + }) +} + impl<'input> Lexer<'input> { pub fn new(input: &'input str, lexer_type: LexerType) -> Self { let mut t = Lexer { diff --git a/src/simlin-engine/src/lexer/test.rs b/src/simlin-engine/src/lexer/test.rs index 713e18839..ae9a618e9 100644 --- a/src/simlin-engine/src/lexer/test.rs +++ b/src/simlin-engine/src/lexer/test.rs @@ -272,6 +272,79 @@ fn safediv_mixed_with_div() { ); } +/// [`super::is_reserved_word`] must agree with the lexer it speaks for: a word +/// it calls reserved is exactly a word `identifierish` resolves to something +/// other than [`Token::Ident`]. The verdicts are derived from `KEYWORDS` here +/// so the pairing cannot go stale when an entry is added or removed -- the +/// independent, hand-written keyword list lives with the consumers +/// (`keyword_ident_tests`, `ast::print_eqn_proptest`), which is where a table +/// that silently SHRANK would show up. +mod reserved_word_tests { + use super::super::{KEYWORDS, is_reserved_word}; + use super::*; + + /// The predicate and `identifierish` must give the same verdict for every + /// entry, in every ASCII casing. + #[test] + fn every_keyword_is_reserved_in_any_casing() { + for (keyword, _) in KEYWORDS { + for spelling in [keyword.to_string(), keyword.to_uppercase(), { + let mut s = keyword.to_string(); + s[..1].make_ascii_uppercase(); + s + }] { + assert!( + is_reserved_word(&spelling), + "`{spelling}` must be reserved: the lexer resolves it to a keyword" + ); + let mut lexer = Lexer::new(&spelling, LexerType::Equation); + let token = lexer.next().expect("one token").expect("lexes"); + assert!( + !matches!(token.1, Token::Ident(_)), + "`{spelling}` lexed as an identifier, so it should not be reserved" + ); + } + } + } + + /// A word that is not in the table lexes as an identifier and is not + /// reserved -- including the near-misses that merely CONTAIN a keyword, and + /// the Unicode near-miss `İF`, whose full lowercase is `i` + a combining dot + /// (NOT `if`). That last one is why the predicate lowercases per Unicode + /// `char` exactly as `identifierish` does; a naive byte fold would still + /// answer correctly here, but only by accident of length. + #[test] + fn a_non_keyword_is_not_reserved() { + for word in [ + "a", "iff", "ifx", "notes", "modulo", "_if", "if2", "band", "\u{130}F", + ] { + assert!(!is_reserved_word(word), "`{word}` must not be reserved"); + let mut lexer = Lexer::new(word, LexerType::Equation); + assert_eq!( + Some(Ok((0, Token::Ident(word), word.len()))), + lexer.next(), + "`{word}` must lex as a whole identifier" + ); + } + } + + /// The units lexer shares this keyword table -- only the identifier + /// character classes differ (units additionally admit `$`). Pinned because + /// `ast::needs_quoting` documents itself as an EQUATION-language predicate + /// on the strength of it. + #[test] + fn the_units_lexer_shares_the_keyword_table() { + for (keyword, _) in KEYWORDS { + let mut lexer = Lexer::new(keyword, LexerType::Units); + let token = lexer.next().expect("one token").expect("lexes"); + assert!( + !matches!(token.1, Token::Ident(_)), + "`{keyword}` must be a keyword in units mode too" + ); + } + } +} + mod scan_number_tests { use super::super::scan_number; diff --git a/src/simlin-engine/src/lib.rs b/src/simlin-engine/src/lib.rs index c2aaef704..9e5da3d07 100644 --- a/src/simlin-engine/src/lib.rs +++ b/src/simlin-engine/src/lib.rs @@ -49,6 +49,8 @@ mod json_proptest; pub mod json_sdai; #[cfg(test)] mod json_sdai_proptest; +#[cfg(test)] +mod keyword_ident_tests; pub mod layout; mod lexer; #[cfg(test)] diff --git a/src/simlin-engine/src/ltm_augment.rs b/src/simlin-engine/src/ltm_augment.rs index e8cedbc7a..ee8777715 100644 --- a/src/simlin-engine/src/ltm_augment.rs +++ b/src/simlin-engine/src/ltm_augment.rs @@ -2678,24 +2678,29 @@ fn live_reducer_text_for_agg<'a>( /// Quote a canonical identifier for use in a generated equation string. /// Identifiers the equation lexer cannot read bare need double quotes: special -/// characters (`$`, `⁚`, `·`) and ALSO a name whose first character cannot START -/// an identifier (`1stock` -- a legal quoted XMILE name, but bare the lexer -/// reads the number `1` then the identifier `stock`). -/// -/// The leading-character rule is [`crate::ast::needs_quoting`], the same -/// predicate the `print_eqn` path's `print_ident` uses -- so the two spellings -/// of one name inside a single generated equation cannot disagree. They did: -/// this used to test only "every char is alphanumeric or `_`", which a leading -/// digit satisfies, so a guard form emitted `print_eqn`'s quoted `"1stock"` in -/// the partial beside a bare `1stock` in the `SIGN(...)` factor -- an -/// unparseable equation, hence a silently-zeroed link score on a valid model. +/// characters (`$`, `⁚`, `·`), a name whose first character cannot START an +/// identifier (`1stock` -- a legal quoted XMILE name, but bare the lexer reads +/// the number `1` then the identifier `stock`), and a name the lexer resolves to +/// a KEYWORD (`if`, `mod`, `nan`, ...). +/// +/// The leading-character and keyword rules are [`crate::ast::needs_quoting`], +/// the same predicate the `print_eqn` path's `print_ident` uses -- so the two +/// spellings of one name inside a single generated equation cannot disagree. +/// They did: this used to test only "every char is alphanumeric or `_`", which a +/// leading digit satisfies, so a guard form emitted `print_eqn`'s quoted +/// `"1stock"` in the partial beside a bare `1stock` in the `SIGN(...)` factor -- +/// an unparseable equation, hence a silently-zeroed link score on a valid model. +/// A keyword satisfied that test too, and is now covered by the same delegation +/// (GH #976). /// /// This deliberately stays MORE conservative than `print_ident` rather than /// delegating outright: `·` (U+00B7) IS `XID_Continue`, so `needs_quoting` /// alone would spell a module-output composite bare (`mod·out1`). That parses, /// but it would rewrite the emitted text of every module-composite link score -/// -- a behavior change with no bug behind it. So a name is bare only when BOTH -/// predicates allow it; the conjunction can only ever add quotes. +/// -- a behavior change with no bug behind it. So the alphanumeric conjunct is +/// NOT redundant; a name is bare only when BOTH predicates allow it, and the +/// conjunction can only ever add quotes. `quote_ident_needs_both_of_its_conjuncts` +/// pins each conjunct on the row only it rejects. pub(crate) fn quote_ident(ident: &str) -> String { let bare_spellable = ident.chars().all(|c| c.is_alphanumeric() || c == '_') && !crate::ast::needs_quoting(ident); diff --git a/src/simlin-engine/src/ltm_augment_tests.rs b/src/simlin-engine/src/ltm_augment_tests.rs index 7d197566d..2b286d1c2 100644 --- a/src/simlin-engine/src/ltm_augment_tests.rs +++ b/src/simlin-engine/src/ltm_augment_tests.rs @@ -825,6 +825,62 @@ fn test_generate_rank_keeps_delta_ratio() { assert!(!partial.contains("PREVIOUS("), "partial: {partial}"); } +/// [`quote_ident`]'s two conjuncts, and why neither is redundant. +/// +/// It quotes when EITHER "every char is alphanumeric or `_`" fails or +/// [`crate::ast::needs_quoting`] says the name is not bare-spellable. Since +/// GH #976 the second conjunct subsumes the leading-digit AND keyword cases, so +/// the natural next simplification is to delete the first and delegate outright. +/// That would be a behavior change with no bug behind it: `·` (U+00B7) is +/// `XID_Continue`, so `needs_quoting` alone reads a module-qualified composite +/// as bare-spellable and every module-composite link score's emitted text moves. +/// +/// This test is the tripwire for that simplification. Delete the alphanumeric +/// conjunct and the `·` row reds; delete the `needs_quoting` conjunct and the +/// keyword and leading-digit rows red. +#[test] +fn quote_ident_needs_both_of_its_conjuncts() { + // Bare-spellable by both predicates: no quotes. + for bare in ["x", "some_var", "v2", "_leading_underscore"] { + assert_eq!(bare, quote_ident(bare), "`{bare}` should stay bare"); + } + + // Rejected by `needs_quoting` only -- the alphanumeric conjunct accepts a + // leading digit and accepts a keyword. + for name in [ + "1stock", "if", "then", "else", "not", "mod", "and", "or", "nan", + ] { + assert!( + crate::ast::needs_quoting(name), + "`{name}` must not be bare-spellable" + ); + assert_eq!(format!("\"{name}\""), quote_ident(name)); + } + + // Rejected by BOTH conjuncts: `$` is not XID_Start and `⁚` (U+205A) is not + // XID_Continue, so `needs_quoting` refuses a synthetic name on its own. Here + // for coverage of the shape LTM actually mints, not to isolate a conjunct. + for name in ["$⁚ltm⁚composite⁚out", "$⁚ltm⁚link_score⁚a→b"] { + assert!(crate::ast::needs_quoting(name)); + assert_eq!( + format!("\"{name}\""), + quote_ident(name), + "`{name}` must stay quoted" + ); + } + + // Rejected by the alphanumeric conjunct ONLY. This is the row -- the only + // row -- that makes that conjunct load-bearing: `·` (U+00B7) IS + // XID_Continue, so `needs_quoting` alone would spell a module-qualified + // composite bare. + assert!( + !crate::ast::needs_quoting("m·out1"), + "the `·` case must be carried by the alphanumeric conjunct, not by \ + needs_quoting -- if this flips, the conjunct really has become redundant" + ); + assert_eq!("\"m·out1\"", quote_ident("m·out1")); +} + /// GH #910: the implicit WITH-LOOKUP wrap must be applied EXACTLY to the /// partials that are a full re-evaluation of the reducer (gf-input units) and /// NEVER to the delta-ratio stand-ins (which are the target's own, already diff --git a/src/simlin-engine/src/mdl/CLAUDE.md b/src/simlin-engine/src/mdl/CLAUDE.md index 18ac3eca8..1c4d39b36 100644 --- a/src/simlin-engine/src/mdl/CLAUDE.md +++ b/src/simlin-engine/src/mdl/CLAUDE.md @@ -42,7 +42,7 @@ For design history and detailed implementation notes, see [docs/design/mdl-parse - `processing.rs` -- Coordinate transforms, angle calculation, flow points ### Expression Formatting -- `xmile_compat.rs` -- XMILE-compatible expression formatter (function renames, argument reordering, name formatting, per-element subscript substitution) +- `xmile_compat.rs` -- XMILE-compatible expression formatter (function renames, argument reordering, name formatting, per-element subscript substitution). The output is **equation text our own lexer reads back**, so a variable REFERENCE is spelled by `XmileFormatter::quote_reference`, which delegates to `ast::needs_quoting` -- the same predicate `ast::print_ident` and `ltm_augment::quote_ident` use. It used to quote only on `.`, which left every other unreadable name bare: `Marg Capital'Energy per Capital` emitted a bare `'` (our TRANSPOSE operator) and `data revenue k$` a bare `$`, both in the checked-in corpus, and seven of the eight equation keywords imported to an `UnrecognizedToken` on every reference (`IDch15d.mdl` declares `MOD`, `test_lookups_funcnames.mdl` declares `mod`) -- GH #976. **`nan` is the deliberate exception**, and a disclosed residual: the importer stores Vensim's `A FUNCTION OF(...)` placeholder as the equation text `NAN`, so quoting `nan` would bind that placeholder to any like-named variable, and the writer cannot spell it differently either -- per Vensim's documentation `A FUNCTION OF` "is not intended for use in writing equations, and precludes simulation", so emitting the marker would produce a file Vensim cannot run. A bare `nan` reference naming a declared variable therefore still reads as the NaN literal; pinned by `keyword_ident_tests::a_bare_nan_reference_in_mdl_is_still_the_literal`, and fixing it means changing how the importer represents "no equation". `quoted_space_to_underbar` is the separate DEFINITION-side spelling (a `datamodel` ident, not equation text) and is unchanged. ### Writing (`writer.rs`) diff --git a/src/simlin-engine/src/mdl/xmile_compat.rs b/src/simlin-engine/src/mdl/xmile_compat.rs index 9e95d6daa..a4aaaca5e 100644 --- a/src/simlin-engine/src/mdl/xmile_compat.rs +++ b/src/simlin-engine/src/mdl/xmile_compat.rs @@ -237,8 +237,70 @@ impl XmileFormatter { } } - // Apply space-to-underbar transformation - quoted_space_to_underbar(name) + self.quote_reference(name) + } + + /// Spell an MDL variable name as a REFERENCE inside an emitted equation. + /// + /// The MDL importer writes equation text into `datamodel::Equation`, and + /// that text is read back by *our* equation lexer -- so the quoting rule + /// here belongs to the equation language, not to MDL, and it is + /// [`crate::ast::needs_quoting`]: the same predicate `ast::print_ident` and + /// `ltm_augment::quote_ident` use. This used to be + /// [`quoted_space_to_underbar`], which quotes only on `.`, and that missed + /// every other name our lexer cannot read bare (GH #976). Two examples from + /// the checked-in corpus: `Marg Capital'Energy per Capital` emitted a bare + /// `'`, which our lexer reads as the TRANSPOSE operator, and + /// `data revenue k$` emitted a bare `$`, which it cannot read at all. + /// + /// Widening is safe in one direction by construction: a name + /// `needs_quoting` rejects cannot be read bare, so quoting it converts a + /// broken import into a working one. The `.` case is subsumed rather than + /// dropped (`.` is not `XID_Continue`), and that one is about IDENTITY as + /// much as lexing -- a bare `a.b` canonicalizes to the module-separator + /// `a·b` while the quoted form canonicalizes to the literal-period + /// sentinel, which is what the definition side produced. + /// + /// The keyword clause covers seven of our eight keywords. Vensim reserves + /// none of them, so `IDch15d.mdl` (which declares `MOD`) and + /// `test_lookups_funcnames.mdl` (which declares `mod`) are legal Vensim that + /// imported to an `UnrecognizedToken` on every reference. + /// + /// **`nan` is deliberately excluded, and that is a disclosed residual.** Our + /// importer represents Vensim's `A FUNCTION OF(...)` -- "this variable has no + /// equation" -- as the stored equation text `NAN` + /// ([`XmileFormatter::format_call_ctx`]'s "a function of" arm), and the MDL + /// writer prints that back as a bare `NaN`. Quoting `nan` here would bind + /// that placeholder to any variable actually named `nan`, so a round trip + /// would compute the variable's value for a variable that has none. Fixing + /// it on the writer side does not work either: per Vensim's documentation + /// `A FUNCTION OF` "is not intended for use in writing equations, and + /// precludes simulation", so emitting it would produce a file Vensim cannot + /// run at all -- there is no writer-side spelling that resolves this. + /// + /// So a bare `nan` reference naming a declared variable still reads as the + /// NaN literal: a KNOWN, PRE-EXISTING, silent case, unchanged by this + /// change, narrower than the seven it fixes, and pinned by + /// `keyword_ident_tests::a_bare_nan_reference_in_mdl_is_still_the_literal`. + /// Fixing it properly means changing how the importer represents "no + /// equation" -- a distinct piece of work. + fn quote_reference(&self, name: &str) -> String { + let result = name.replace(' ', "_"); + if result.starts_with('"') { + // Already quoted (#846): re-quoting would grow a second layer. + return result; + } + // `nan` is pure ASCII letters, so the keyword clause is the only reason + // `needs_quoting` could be rejecting it -- excluding it here cannot + // suppress a character-class or leading-character rejection. + if result.eq_ignore_ascii_case("nan") { + return result; + } + if crate::ast::needs_quoting(&result) { + format!("\"{}\"", result) + } else { + result + } } // Each match arm has an inner `if args.len() >= N` guarding access to @@ -789,6 +851,12 @@ pub fn space_to_underbar(name: &str) -> String { } /// Replace spaces with underscores, quoting if the name contains periods. +/// +/// This is the DEFINITION-side spelling: it produces a `datamodel` variable +/// *ident* (see `mdl::convert::helpers::variable_ident`), which is a name and +/// not equation text. A reference to that variable inside an equation goes +/// through [`XmileFormatter::quote_reference`] instead, because that text has to +/// survive our own lexer. pub fn quoted_space_to_underbar(name: &str) -> String { let result = name.replace(' ', "_"); if result.contains('.') && !result.starts_with('"') { diff --git a/src/simlin-engine/src/patch.rs b/src/simlin-engine/src/patch.rs index 4ee7b6879..ebee75a97 100644 --- a/src/simlin-engine/src/patch.rs +++ b/src/simlin-engine/src/patch.rs @@ -477,6 +477,33 @@ fn apply_rename_variable( let old_ident = Ident::new(from); let new_ident = Ident::new(to); + // Refuse a target name that has NO spelling in the equation language. + // + // `Lexer::quoted_identifier` terminates on the first `"` and the grammar has + // no escape of any kind, so a canonical name containing `"` can be written + // neither bare nor quoted. Renaming TO one is not merely useless: this + // function reprints every dependent equation, so it would persist + // `c = "x"y" + 1` into the datamodel and the previously-valid model would + // stop compiling with `UnclosedQuotedIdent` -- the same silent, saved + // corruption GH #976 fixed for keyword names, through this same entry point. + // + // Rejecting at the front door rather than teaching the lexer an escape is + // deliberate: nothing is lost by refusing a name that nothing could ever + // reference, and the alternative is a grammar change. Note the check is on + // `to` only -- renaming AWAY from such a name is how a model that already + // has one gets repaired. The error code is the one recompiling would have + // produced, so the rejection and the failure it prevents read alike. + if new_ident.as_str().contains('"') { + return Err(Error::new( + ErrorKind::Model, + ErrorCode::UnclosedQuotedIdent, + Some(format!( + "cannot rename to `{to}`: a name containing a double quote cannot be \ + referenced in an equation" + )), + )); + } + if old_ident == new_ident { // Canonically-identical rename: only the display spelling changes // (e.g. "students" -> "Students"). Every reference resolves through From 2f68396422c0eaf70db3a9c29ec1a8fee3702d2f Mon Sep 17 00:00:00 2001 From: Bobby Powers Date: Mon, 27 Jul 2026 12:53:35 -0700 Subject: [PATCH 02/17] engine: make Ident's salsa::Update delegation real The doc on `Ident` said its derived `salsa::Update` delegates to `CanonicalStorage`'s manual impl. It could not: salsa 0.26's derive adds an `Update` bound to every type parameter, and the `Canonical`/`Raw` markers had no impl, so `Ident: salsa::Update` was unsatisfiable and a container's per-field dispatch resolved the field through salsa's `'static + PartialEq` fallback instead. That fallback is value-correct here -- `Ident`'s derived `PartialEq` bottoms out in the same interned comparison -- so this was never a wrong answer, only a comment describing a path that did not exist. The markers now derive `Update` themselves. On a field-less struct the derive generates the "nothing can differ, report no change" body, structurally identical to salsa's own `PhantomData` impl, so the unsafe trait's obligation is discharged vacuously and no `unsafe` is hand-written. Scope, stated so nobody over-reads it: salsa calls a field's `maybe_update` only for tracked STRUCTS, while a tracked FUNCTION's memo backdates through `values_equal`/`PartialEq` and an input setter overwrites outright -- and this crate declares no tracked structs, only tracked functions. So this changes which impl the compiler selects and changes no runtime behavior at all today; it is a correctness-of- documentation fix plus forward-compatibility for the first tracked struct. A static assertion instantiates the bound at both marker states so the doc claim is a build failure if it stops holding, and a test names `UpdateDispatch::>::maybe_update` with `UpdateFallback` deliberately out of scope, so it can only resolve to the inherent `Update`-backed method -- direct evidence of which impl runs rather than an argument that it runs. --- src/simlin-engine/src/common.rs | 94 ++++++++++++++++++++++++++++++++- 1 file changed, 92 insertions(+), 2 deletions(-) diff --git a/src/simlin-engine/src/common.rs b/src/simlin-engine/src/common.rs index baf097042..cfff90128 100644 --- a/src/simlin-engine/src/common.rs +++ b/src/simlin-engine/src/common.rs @@ -206,6 +206,9 @@ impl Drop for Interned { /// string content. `Arc`/`str` are not covered by salsa's blanket /// `Update` impls, so this is provided manually here; the public newtypes /// keep `#[derive(salsa::Update)]`, whose per-field dispatch finds this impl. +/// The GENERIC newtype `Ident` needs one extra thing for that to hold +/// -- an `Update` impl on the `Canonical`/`Raw` markers, since salsa's derive +/// bounds every type parameter -- see [`Ident`] (GH #972). #[derive(Clone)] pub(crate) struct CanonicalStorage(std::sync::Arc); @@ -2111,6 +2114,49 @@ mod interned_identifier_tests { assert!(!unchanged, "equal values must report no change"); assert_eq!(slot.as_str(), "new_value"); } + + /// An `Ident` field inside a `#[derive(salsa::Update)]` container + /// must resolve through `Ident`'s OWN derived impl -- and hence through + /// `CanonicalStorage::maybe_update` -- rather than through salsa's + /// `'static + PartialEq` fallback (GH #972). + /// + /// The evidence is the resolution itself. `salsa::plumbing::UpdateDispatch` + /// is the derive's per-field entry point, and it carries exactly two + /// `maybe_update`s: an INHERENT one on `Dispatch where D: Update`, and a + /// `Fallback` TRAIT one. `UpdateFallback` is deliberately NOT imported in + /// this module, so naming `UpdateDispatch::>::maybe_update` + /// can only resolve to the inherent -- `Update`-backed -- method. Before the + /// markers gained `Update`, this line did not compile at all, which is the + /// same E0277 the issue's probe reported. + #[test] + fn an_ident_field_dispatches_through_its_own_update_impl() { + let dispatch: unsafe fn(*mut Ident, Ident) -> bool = + salsa::plumbing::UpdateDispatch::>::maybe_update; + + // And it behaves as the delegation implies: change reported iff the + // canonical string differs, which is `CanonicalStorage`'s rule. + let mut slot = Ident::::new("old_value"); + let changed = { + // SAFETY (test): `&mut slot` is a valid, owned `Ident`; we pass its + // pointer and a fresh owned value, matching the `maybe_update` + // contract. + #[allow(unsafe_code)] + unsafe { + dispatch(&mut slot as *mut _, Ident::::new("new_value")) + } + }; + assert!(changed, "a differing ident must report a change"); + assert_eq!(slot.as_str(), "new_value"); + + let unchanged = { + #[allow(unsafe_code)] + unsafe { + dispatch(&mut slot as *mut _, Ident::::new("new_value")) + } + }; + assert!(!unchanged, "an equal ident must report no change"); + assert_eq!(slot.as_str(), "new_value"); + } } // Implementations for identifier types @@ -2298,11 +2344,23 @@ impl AsRef for RawElementName { // canonicalization guarantees through the type system. /// Marker type for canonical identifiers -#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)] +/// +/// `salsa::Update` is derived even though the marker holds no data and is never +/// stored on its own. Salsa's derive adds an `Update` bound to EVERY type +/// parameter, so without it `Ident: salsa::Update` is unsatisfiable +/// and an `Ident` field inside a `#[derive(salsa::Update)]` container silently +/// resolves through salsa's `'static + PartialEq` fallback instead of through +/// [`CanonicalStorage`]'s manual impl (GH #972). The derive is the whole fix: on +/// a field-less struct it generates the "nothing can differ, report no change" +/// body -- exactly salsa's own `PhantomData` impl -- so the `unsafe` trait's +/// obligation is discharged vacuously and no `unsafe` is hand-written here. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord, salsa::Update)] pub struct Canonical; /// Marker type for raw (non-canonical) identifiers -#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)] +/// +/// `salsa::Update` for the same reason as [`Canonical`]; see its docs. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord, salsa::Update)] pub struct Raw; /// An owned identifier with state tracking (canonical or raw). @@ -2314,12 +2372,44 @@ pub struct Raw; /// `PartialOrd`/`salsa::Update` delegate to that handle's manual impls (value /// equality, value-based hash consistent with `Borrow`, lexicographic /// ordering), preserving the previous `String`-backed semantics. +/// +/// The `salsa::Update` half of that claim is only true because [`Canonical`] and +/// [`Raw`] derive `Update` themselves: salsa's derive adds an `Update` bound to +/// every type parameter, so a marker without one makes the generated +/// `unsafe impl Update for Ident` unsatisfiable, +/// and a container's per-field dispatch quietly resolves `Ident` +/// through salsa's `'static + PartialEq` fallback instead (GH #972). That +/// fallback happens to be value-correct here -- `Ident`'s derived `PartialEq` +/// bottoms out in the same interned comparison `CanonicalStorage::maybe_update` +/// makes -- so this was never a wrong answer, only a doc claiming a delegation +/// that could not occur. The assertion below is what keeps the claim honest: it +/// is a compile error the moment the bound stops being satisfiable. +/// +/// Scope of the fix, stated so nobody over-reads it: salsa calls a FIELD's +/// `maybe_update` only for tracked STRUCTS (`salsa-macro-rules`' +/// `setup_tracked_struct`), while a tracked FUNCTION's memo backdates through +/// `values_equal`/`PartialEq` (`salsa::function::backdate`) and an input setter +/// overwrites outright. This crate declares no tracked structs -- every +/// `#[salsa::tracked]` item here is a function -- so making the bound +/// satisfiable changes which impl the compiler SELECTS and changes no runtime +/// behavior at all today. It is a correctness-of-documentation fix plus +/// forward-compatibility for the first tracked struct, not a behavior change. #[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, salsa::Update)] pub struct Ident { inner: CanonicalStorage, _phantom: PhantomData, } +// The derived `Update` on `Ident` must be *reachable* at both marker +// states, not merely present. A prose claim about which `maybe_update` runs is +// unfalsifiable; instantiating the bound turns it into a build failure. +const _: fn() = || { + fn assert_update() {} + assert_update::>(); + assert_update::>(); + assert_update::(); +}; + /// A borrowed identifier reference with state tracking /// This is the key type that enables zero-copy substring operations #[cfg_attr(feature = "debug-derive", derive(Debug))] From 0225f73d08a3396dd0019e3524808431103bbb7b Mon Sep 17 00:00:00 2001 From: Bobby Powers Date: Mon, 27 Jul 2026 12:54:23 -0700 Subject: [PATCH 03/17] doc: record three rules the keyword-quoting work paid for Each of these cost real rework on the branch that added them, so they are written as rules rather than as advice. "Claims About Other Tools" is the expensive one. A premise about what Vensim does -- asserted, never checked -- carried a design decision through three rounds of adversarial review, and every round found genuine defects in the code built on top of it while none questioned the premise. That is the shape of the failure: review checks code against the stated claim, not the claim against the world, so a wrong premise about an external system survives exactly the process meant to catch it. The section names the primary sources, including the XMILE spec already checked in at docs/reference/xmile-v1.0.html, which settles a subscript-precedence question the same branch nearly answered by reasoning from our own compiler instead. The test rule generalizes a mistake that recurred three times in one change at three different altitudes: a test covering one arm of a two-arm gate, then one position of a two-position decision, then one of six call sites. Each read as though it pinned the decision, and each left the untested arm free to compute a wrong number. Deriving the rows from an enumeration -- the variant list, the call-site list -- makes "no gaps" checkable instead of assumed. The scope signal is the cheapest of the three to apply: when a second attempt at the same discovered issue produces a new defect, the understanding behind it has now failed twice, and further attempts are designed against it. Splitting at that point would have ended the episode above four rounds earlier. --- CLAUDE.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/CLAUDE.md b/CLAUDE.md index 7031fc16e..7d1da4de3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -94,6 +94,20 @@ IMPORTANT: Simple, general, testable, maintainable code is better than preservin IMPORTANT: If feedback seems non-actionable, it means you need comments explaining why the code looks that way. +**CRITICAL**: A test that pins one arm of an N-way decision reads exactly like a test that pins the decision. Derive the rows from the enumeration -- the variant list, the call-site list, the axis list -- and cover every arm, or state in the test which arms it does not cover and where they are covered instead. "It passes" and "it constrains the code" are different claims. + +## Claims About Other Tools + +Vensim, Stella, and the XMILE specification are external systems. What one of them does is a **fact to be checked, not a premise to reason from** -- and the sources are cheap: + +- **XMILE spec**: `docs/reference/xmile-v1.0.html`, in-repo and greppable (strip tags and search the prose; the footnotes carry the resolution rules). +- **Vensim function reference**: one page per function at vensim.com/documentation, named fn_ plus the lowercased function name with underscores. +- **Ground truth output**: real Vensim DSS runs checked into `test/` (e.g. `test/test-models/tests/vector_order/output.tab`, which is how VECTOR SORT ORDER's semantics were settled). + +When code encodes a claim about another tool, cite the source next to it. When you cannot check one, write that the claim is unverified rather than asserting it -- and never let an unverified claim carry a design decision. + +This is the most expensive class of error in this repo, because review does not catch it: reviewers check the code against the stated claim, not the claim against the world. A wrong premise about an external tool therefore survives every round of review that its own code passes, and the work built on top of it is wasted rather than merely wrong. + ## Comment and Rustdoc Standards - Preserve useful comments/docstrings when refactoring. Do not delete comments unless they are stale, wrong, or redundant with clearer replacement code. @@ -132,6 +146,8 @@ Two things are NOT covered by "fix it", and both are explicit conversations rath - **The fix is too large to fold in** -- it would swamp the branch's review surface, or it belongs to a different subsystem. Say so, with a cost estimate, and sequence it: its own commit, its own PR, or (if that is what the user wants) tracked for later. Do not make that call quietly. - **The fix is not yours to make** -- it needs a product decision, or access you do not have. Surface it. +**A fix that introduces a regression is a scope signal, not a bug to try harder at.** Once the second attempt at the same discovered issue produces a new defect, stop and re-scope: the problem is larger than the branch it was found in, and each further attempt is being designed against an understanding that has already failed twice. Split it out with what was learned, rather than continuing. Discovering something mid-task tells you it exists; it does not tell you it belongs in the change you are making. + When something genuinely does need tracking, spawn the `track-issue` agent (via the Task tool with `subagent_type: "track-issue"`) with a description of the problem. It checks for duplicates in GitHub issues and [docs/tech-debt.md](/docs/tech-debt.md) and files the item, keeping your context on the main task. ## Generated/Noise Paths From 5ed3bccd546e874f436e8526bddf23c2616ec49c Mon Sep 17 00:00:00 2001 From: Bobby Powers Date: Mon, 27 Jul 2026 14:41:25 -0700 Subject: [PATCH 04/17] engine: reject over-wide LTM targets at the front door LTM occurrence identity is a `SiteId`: a path of u16 child indices. Three of the walk's push AXES take a count the AST node's own shape does not bound -- an Ast::Arrayed target's equation slots, one builtin call's contents (Mean is the only variadic BuiltinFn), and one subscript's index list, which Expr2 lowering does NOT narrow to the subscripted variable's declared arity. Past 65,536 children, two references share an identity and the ceteris-paribus wrap holds or freezes the wrong one, emitting a plausible, wrong link score. Only the builtin axis was handled, by a reserved u16::MAX sentinel, a suppression depth counter with a deliberately asymmetric two-view contract, and a checked sentinel-mapping conversion in the consumer -- machinery that also left module_output_ref_in_document_order unable to distinguish a width-suppressed module-output reference from "the target reads no output of this module". Replace all of it with one check run per target equation before the walk pushes a single path component. A rejected equation records no occurrence at all, so no recorded SiteId can be a path two children share, and model_ltm_variables refuses the whole model with a Warning naming the variable. The refusal is model-level rather than per-edge because the trigger is a property of a target equation and because consumers read a MISSING per-edge entry as a real Bare classification -- so a per-site drop would misclassify rather than decline. The name-keyed, path-free per-edge view stays complete, so loop detection is unaffected: a refusal costs the scores, not the structure. One ordering consequence, benign and deliberate: the refusal sits after the stateless early return, so a stateless model that is also over-wide returns no variables and no width warning -- it emits no scores either way, so there is nothing to warn about. It is a separate pre-pass rather than a counter carried on the walk because its descent has to be a SUPERSET of the walk's. The walk pushes a path component for a LOOKUP table argument and then does not recurse, while the ceteris-paribus wrap and pin_only_source_refs both descend through it with child_path -- so a builtin nested under a table argument is reachable by consumer path indices and by no producer path at all. Subscript indices are the same shape of problem with a different skip predicate on each side. The check's arms are derived from the walk's own push sites and its Expr2 match is exhaustive with no catch-all, so a new variable-width variant is a compile error rather than an unchecked axis. Tests cover all eleven descent edges, the accept side of the boundary on each axis, and that a refusal leaves model_edge_shapes and model_detected_loops untouched; they use a #[cfg(test)] limit override and tiny fixtures. The 16 LTM char goldens and 12 fragment goldens are byte-identical, since the deleted machinery only activated past 65,535. --- src/simlin-engine/CLAUDE.md | 2 +- src/simlin-engine/src/db.rs | 34 +- src/simlin-engine/src/db/ltm/link_scores.rs | 6 + src/simlin-engine/src/db/ltm/mod.rs | 48 ++ src/simlin-engine/src/db/ltm_ir.rs | 447 +++++++++--- src/simlin-engine/src/db/ltm_ir_tests.rs | 714 +++++++++++++++++--- src/simlin-engine/src/ltm_augment.rs | 55 +- src/simlin-engine/src/ltm_augment_tests.rs | 55 +- 8 files changed, 1085 insertions(+), 276 deletions(-) diff --git a/src/simlin-engine/CLAUDE.md b/src/simlin-engine/CLAUDE.md index 6252d6ec9..3bbb9655e 100644 --- a/src/simlin-engine/CLAUDE.md +++ b/src/simlin-engine/CLAUDE.md @@ -112,7 +112,7 @@ The primary compilation path uses salsa tracked functions for fine-grained incre - **`src/db/fragment_compile.rs`** (a `db` submodule) - The *emission half* of per-variable compilation (the *lowering half* is the sibling `src/db/var_fragment.rs`). `compile_var_fragment` is the salsa-tracked per-variable fragment compiler; `compile_implicit_var_fragment`/`compile_implicit_var_phase_bytecodes` do the same for the implicit (SMOOTH/DELAY/TREND) helpers, sharing the `lower_implicit_var` parent->implicit->parse->lower prefix. The emission tail (`compile_phase_to_per_var_bytecodes`) lives in `src/db/assemble.rs` and is the **single fragment emission entry point** every emitter -- explicit, implicit, and both LTM ones -- goes through (GH #964). Neither half assigns offsets any more: lowering hands codegen a `Vec` whose variable references are names, so the private per-fragment layout and the reverse offset map that used to undo it are gone. Runlist membership is read through the `var_runlist_membership` projection, not the whole `ModelDepGraphResult`, so an unrelated variable add/delete/rename no longer invalidates every fragment in the model. - **`src/db/assemble.rs`** (a `db` submodule) - Module/simulation assembly: the table/metadata extraction helpers (`extract_tables_from_source_var`, `build_module_inputs`, `build_stub_variable`, `build_submodel_metadata`), the per-variable emission tail (`compile_phase_to_per_var_bytecodes`, the SINGLE fragment emission entry point since GH #964's stage 2; `fragment_emit_ctx`, which assembles the phase-invariant `compiler::ModuleCtx` its callers borrow into it; the `VarFragmentResult`/`PerVarSizes` values; and `temp_sizes_by_id`, the temp-id-ORDERED flattening of the temp-size map), the production element-graph source `var_phase_symbolic_fragment_prod` (the layout-independent `SymVarRef` substrate the cycle gate's element-order probe consumes), the resolved recurrence-SCC interleaver (`segment_member_by_element` / `combine_scc_fragment`, the per-element-granular generalization of `concatenate_fragments`), the salsa-tracked `assemble_module` / `assemble_simulation`, module-instance enumeration (`enumerate_module_instances`), and the flattened-offset map builder (`calc_flattened_offsets_incremental`). Until stage 2 there were THREE hand-copied emitters (this one plus two inline copies in `src/db/ltm/compile.rs`), each building a stand-in one-variable `compiler::Module` by struct literal and deep-cloning `offsets`/`tables`/`dimensions`/`dimensions_ctx`/`module_refs` per phase; the temp-size ordering fix below had to be applied to all three at once, which is what a mirror family costs. `module_refs` and `runlist_order` turned out to be read by no line of codegen at all, so collapsing the copies also deleted `build_caller_module_refs` -- a whole second per-variable dependency walk whose only consumer was those dead fields. `temp_sizes_by_id`'s ordering is load-bearing because `PerVarBytecodes` is a salsa-cached value with a derived `PartialEq`, so a `HashMap`-iteration-order flip made an identical fragment compare unequal, defeating backdating and making the compiled artifact irreproducible run to run, GH #595's class; `FragmentMerger::absorb` folds the entries order-independently, which is why the defect had no bytecode consequence and survived undetected. `assemble_module` injects each resolved SCC's combined per-element fragment at the first member's runlist slot (the dt fragment into flows, the init fragment as one synthetic-ident `SymbolicCompiledInitial`), preserving per-write `SymVarRef` identity so layout offsets are unchanged. `build_submodel_metadata` registers not only a sub-model's explicit/implicit source variables but -- when `ltm_enabled` -- its LTM synthetic vars (and their implicit helpers) at their `compute_layout` offsets, so a parent fragment's cross-module reference to a sub-model LTM var resolves the same way the full flattened-offset assembly does. This is what lets the exhaustive-mode input→macro link score (the composite-reference form `"{module}·$⁚ltm⁚composite⁚{port}"`) compile in the standalone fragment compiler (`compile_ltm_equation_fragment`) instead of stubbing to a constant 0 (GH #548): the composite link score through a SMOOTH/DELAY/TREND/NPV macro is the max-magnitude path score through the macro (LTM ref 6.3/6.4), computed by the sub-model's `$⁚ltm⁚composite⁚{port}` aux. - **`src/db/analysis.rs`** - Salsa-tracked causal graph analysis: `model_causal_edges`, `model_loop_circuits`, `model_cycle_partitions`, `model_detected_loops`. Element-level tracked functions: `model_element_causal_edges` (emits per-reference element edges via `emit_edges_for_reference`, driven by the `db::ltm_ir` classification IR: each reference's access shape -- `Bare`, `FixedIndex(elements)`, `PerElement { axes }` (the GH #525 iterated+literal mix, expanded as the diagonal-with-pinned-axes rows from the shared `read_slice_rows` derivation, broadcast over unshared target dims), `Wildcard`, or `DynamicIndex` -- and its `routing` (`Direct` or `ThroughAgg`) come from `model_ltm_reference_sites`; a `ThroughAgg` reference is routed through a synthetic `$⁚ltm⁚agg⁚{n}` aggregate node by `emit_agg_routed_edges` -- only the rows the reducer's `read_slice` reads: `source[,,] → agg[]` then `agg[] → to[e]` (`agg.result_dims` drives the fan-out, diagonal/broadcast like `Bare`; an `Iterated` axis's slot coordinate is remapped to the corresponding TARGET-dim element via `ltm_agg::iterated_axis_slot_elements` when the axis is a positionally-mapped pair -- GH #534, identity in the literal case), never the all-pairs N×M cross-product; a whole-extent reducer (all-`Reduced`) degenerates to "every source element → scalar agg → every target element". `emit_agg_routed_edges` derives the agg's result-axis dimensions from `AggNode::result_dims` (resolved against `from` ⌢ `to` dims), so it handles a *scalar* feeder of a (possibly arrayed) hoisted reducer (`scale` in `growth[D1] = SUM(matrix[D1,*] * scale)`): `from_dims.is_empty()` ⇒ emit `from → agg[]` (or the bare `from → agg` when the agg is scalar) rather than the malformed `from[]` node the row-layout machinery would mint. `Direct` `DynamicIndex` references (`arr[i+1]`, ranges, and the not-hoistable dynamic-index reducer carve-out `SUM(pop[idx,*])` -- reclassified from `Wildcard` by the IR) still expand to the conservative cross-product. A `Bare` edge between differently-named but MAPPED dimensions (`x[Region] → target[State]` with a `State→Region` mapping) projects the mapping's element-correspondence DIAGONAL via `expand_same_element` + `DimensionsContext::mapped_element_correspondence` -- one source element per target element, not the `Region × State` cross-product (GH #527) -- WHEN a usable correspondence exists (positional mappings only; explicit element maps are declined since execution resolves positionally, see the helper's gate); an unmapped or element-mapped disjoint-named pair keeps the broadcast), `model_element_loop_circuits` (Johnson's algorithm on element graph; legacy path retained for non-LTM consumers), `model_element_cycle_partitions` (stock-to-stock SCCs at element granularity). Tiered enumeration: `model_edge_shapes` (per-`(from, to)` `BTreeSet` projected from the IR's classified sites) and `model_loop_circuits_tiered` (variable-level Johnson + cycle classification + slow-path-subgraph Johnson, keeping synthetic agg nodes in the slow-path subgraph projection so cross-element loops through a hoisted reducer are recovered) replace the full-element Johnson run for LTM compilation. `classify_cycle` labels each variable-level cycle as `PureScalar`, `PureSameElementA2A`, or `CrossElementOrMixed`; pure cycles emit one Loop directly while cross-element / mixed cycles drive the slow-path subgraph (the induced element subgraph over their nodes). `TieredCircuitsResult.slow_path_largest_scc` exposes the cross-element subgraph's largest SCC for the LTM auto-flip gate. Produces `DetectedLoop` structs with polarity (each carrying an optional `name`: `None` for an enumerated loop, `Some(label)` for a modeler-pinned one, plus a `partition: Option` -- a RESULT-SCOPED dense index into `DetectedLoopsResult.partitions` (reusing the discovery surface's `ltm_finding::DiscoveredPartition`), `None` for a loop with no parent-level stock; same stability caveat as `FoundLoop.partition`, GH #685). `model_detected_loops` builds its exhaustive loop set with the SAME machinery the scored surface uses -- `model_loop_circuits_tiered` feeding `db::ltm::build_loops_from_tiered` (under the shared `cross_agg_loop_budget`) plus the shared `recover_agg_hop_polarities` pass -- so the detected and scored loop sets, polarities, and ids are identical BY CONSTRUCTION for every model shape (GH #746; the runtime id join `reclassify_loops_from_results` / pysimlin `get_relative_loop_score` reads `$⁚ltm⁚loop_score⁚{id}` keyed purely on the id, and the previous variable-level-Johnson + per-agg-splice path only bijected for all-scalar cycles -- arrayed cycles silently joined another loop's series). Cross-element loops therefore surface per element instance with element-subscripted variables, mirroring cross-element pins. It resolves cycle partitions over the ELEMENT-level graph (`model_element_cycle_partitions` -- the same granularity the scored `loop_partitions` and the discovery surface use), and `resolve_loop_partitions` mirrors `ltm_finding::attach_partition_metadata` exactly (first-appearance dense remap, `DiscoveredPartition { stocks, loop_count }`) so the two surfaces report partitions in the same shape AND at the same granularity: the partition stock SETS are a usable cross-surface key for scalar and arrayed models alike (an A2A loop's single `partition` index is its first resolving slot's partition; the scored surface's `loop_partitions[id]` carries the full per-slot vector). It appends the model's *pinned* loops (`db::ltm::model_pinned_loops`, deduped against the enumerated set by canonical variable-cycle rotation; a pin that duplicates an enumerated loop transfers its `name` onto the surviving enumerated `DetectedLoop` rather than being discarded), and pinned loops get a partition too. It consults the SHARED `db::ltm::model_ltm_mode` query for the exhaustive-vs-discovery decision -- the SAME gate `model_ltm_variables` uses -- so the two query surfaces never disagree: in discovery mode (whether the user forced `ltm_discovery_mode`, the variable-level SCC exceeded `MAX_LTM_SCC_NODES`, or the slow-path late-flip fired) it returns *only* the pinned loops, so a pinned loop surfaces through `simlin_analyze_get_loops` even in discovery mode (and is backed by the `loop_score` that mode actually emits). Before this unification `model_detected_loops` gated only on the `causal_graph_with_modules` SCC size and ignored both the user flag and the slow-path flip, so a small user-forced-discovery model dropped the pin entirely. `RefShape` and `emit_edges_for_reference` (plus the element-name expansion helpers) live here; the AST walker / agg-routing decision lives in `src/db/ltm_ir.rs`. -- **`src/db/ltm_ir.rs`** (a `db` submodule, a child of `db.rs` kept in its own file purely for the per-file line cap; like `ltm_agg`, it builds on `crate::db`) - The single salsa-tracked place a causal edge's access shape *and* aggregate-node routing are decided. `model_ltm_reference_sites(db, model, project) -> LtmReferenceSitesResult` walks each variable's `Expr2` AST once, consults `enumerate_agg_nodes` (the sole "hoistable maximal reducer" decider), and buckets every `Var`/`Subscript` reference by its `(from, to)` causal edge into a `Vec` (`shape` + `target_element` + `routing ∈ {Direct, ThroughAgg{agg}}`); the byte-identical `route_through_agg = !routed_aggs.is_empty() && in_reducer` decision and the `aggs_in_var(to).filter(is_synthetic && reads from)` filter exist here and nowhere else. `model_element_causal_edges`, `model_edge_shapes`, and `model_ltm_variables` are pure readers. Also hosts the AST-walker helpers moved out of `src/db/analysis.rs` (`collect_reference_sites` / `classify_subscript_shape` / `resolve_literal_index` / `collect_reference_shapes`); `classify_subscript_shape` carries the AC1.4 fix -- a subscript whose indices are *all* `Wildcard`/`StarRange` (the reducer-style whole-extent access, e.g. `SUM(x[*:Dim])`) classifies as `Wildcard` to agree with `enumerate_agg_nodes`'s read-slice hoisting test. `classify_iterated_dim_shape` consumes `ltm_agg::classify_axis_access` per axis (T6 of the shape-expressiveness design; a `Reduced` result is post-filtered to `None` -- a non-reducer reference never collapses an axis): an ALL-`Iterated` subscript -- indices that are exactly the target equation's iterated dimensions, position-matched to the source's declared dims by name or by a positional mapping in EITHER declaration direction (GH #511/#527/#757) -- classifies `Bare` (a same-element-on-shared-dims reference projected via `expand_same_element`); a MIXED `Iterated`+`Pinned` subscript (`pop[Region, young]` inside an A2A-over-`Region` equation, GH #525) classifies `RefShape::PerElement { axes }`, whose element edges are the diagonal-with-pinned-axes rows from the shared `read_slice_rows` derivation (never the cross-product whose phantom circuits carried silent confident loop scores) and whose link scores are per-(row, FULL-target-element) scalars `$⁚ltm⁚link_score⁚{from}[{row}]→{to}[{e}]` (the row a function of `e`: project `e` onto the Iterated axes, fill Pinned with literals -- including the BROADCAST case where the Iterated dims are a strict subset of the target's), emitted by `emit_per_element_link_scores`/`generate_per_element_link_equation` and routed per-circuit by the loop builder's `is_per_element_edge` predicate (keyed on the same `model_edge_shapes` IR projection); an all-`Pinned` subscript falls through to `classify_subscript_shape`'s `FixedIndex` (so every existing Bare/FixedIndex link-score name is untouched). A *partially*-iterated subscript that is a *reducer argument* (`SUM(matrix[D1,*])`, `SUM(pop[NYC,*])`, `SUM(matrix3d[D1,NYC,*])`) is hoisted into a synthetic agg by `enumerate_agg_nodes` (its read slice is statically describable), so the reference is `ThroughAgg`-routed and its (`Wildcard`) shape is ignored; the only `Direct` `Wildcard` reducer references remaining are a *whole-RHS* variable-backed reducer's argument (`total = SUM(population[*])`), which keeps `Wildcard`, a DE-HOISTED array-valued reducer's wildcard arg (`RANK(pop[*], 1)` -- GH #771: RANK never sets `in_reducer`, so the site keeps `Wildcard` and takes the conservative cross-product), and a not-hoistable reducer's argument -- a *dynamic index* (`SUM(pop[idx,*])`, `idx` non-literal) or an ELEMENT-mapped sliced reducer the correspondence declines (GH #756; the positionally-mapped case is hoisted since GH #534 -- both declaration directions since GH #757 -- with the `Iterated` axis carrying the (target, source) dim pair). `model_ltm_reference_sites` reclassifies such a not-hoisted `Direct` `Wildcard` `in_reducer` site as `DynamicIndex` (#514) so the `Wildcard`-shape conservative cross-product never fires from a `Direct` site that *could* have been hoisted (the de-hoisted RANK arg is not hoist-eligible, so it deliberately stays `Wildcard`). (`classify_iterated_dim_shape`'s mapped branch -- a *whole-equation*-iterated subscript like `x[State]` inside `target[State] = x[State] * c`, not a sliced reducer argument -- is a separate path and still classifies as `Bare`.) The mapped case needs a `DimensionsContext` (built from `project_datamodel_dims`), so the IR is recomputed when a dimension's mappings change. +- **`src/db/ltm_ir.rs`** (a `db` submodule, a child of `db.rs` kept in its own file purely for the per-file line cap; like `ltm_agg`, it builds on `crate::db`) - The single salsa-tracked place a causal edge's access shape *and* aggregate-node routing are decided. `model_ltm_reference_sites(db, model, project) -> LtmReferenceSitesResult` walks each variable's `Expr2` AST once, consults `enumerate_agg_nodes` (the sole "hoistable maximal reducer" decider), and buckets every `Var`/`Subscript` reference by its `(from, to)` causal edge into a `Vec` (`shape` + `target_element` + `routing ∈ {Direct, ThroughAgg{agg}}`); the byte-identical `route_through_agg = !routed_aggs.is_empty() && in_reducer` decision and the `aggs_in_var(to).filter(is_synthetic && reads from)` filter exist here and nowhere else. `model_element_causal_edges`, `model_edge_shapes`, and `model_ltm_variables` are pure readers. Also hosts the AST-walker helpers moved out of `src/db/analysis.rs` (`collect_reference_sites` / `classify_subscript_shape` / `resolve_literal_index` / `collect_reference_shapes`); `classify_subscript_shape` carries the AC1.4 fix -- a subscript whose indices are *all* `Wildcard`/`StarRange` (the reducer-style whole-extent access, e.g. `SUM(x[*:Dim])`) classifies as `Wildcard` to agree with `enumerate_agg_nodes`'s read-slice hoisting test. `classify_iterated_dim_shape` consumes `ltm_agg::classify_axis_access` per axis (T6 of the shape-expressiveness design; a `Reduced` result is post-filtered to `None` -- a non-reducer reference never collapses an axis): an ALL-`Iterated` subscript -- indices that are exactly the target equation's iterated dimensions, position-matched to the source's declared dims by name or by a positional mapping in EITHER declaration direction (GH #511/#527/#757) -- classifies `Bare` (a same-element-on-shared-dims reference projected via `expand_same_element`); a MIXED `Iterated`+`Pinned` subscript (`pop[Region, young]` inside an A2A-over-`Region` equation, GH #525) classifies `RefShape::PerElement { axes }`, whose element edges are the diagonal-with-pinned-axes rows from the shared `read_slice_rows` derivation (never the cross-product whose phantom circuits carried silent confident loop scores) and whose link scores are per-(row, FULL-target-element) scalars `$⁚ltm⁚link_score⁚{from}[{row}]→{to}[{e}]` (the row a function of `e`: project `e` onto the Iterated axes, fill Pinned with literals -- including the BROADCAST case where the Iterated dims are a strict subset of the target's), emitted by `emit_per_element_link_scores`/`generate_per_element_link_equation` and routed per-circuit by the loop builder's `is_per_element_edge` predicate (keyed on the same `model_edge_shapes` IR projection); an all-`Pinned` subscript falls through to `classify_subscript_shape`'s `FixedIndex` (so every existing Bare/FixedIndex link-score name is untouched). A *partially*-iterated subscript that is a *reducer argument* (`SUM(matrix[D1,*])`, `SUM(pop[NYC,*])`, `SUM(matrix3d[D1,NYC,*])`) is hoisted into a synthetic agg by `enumerate_agg_nodes` (its read slice is statically describable), so the reference is `ThroughAgg`-routed and its (`Wildcard`) shape is ignored; the only `Direct` `Wildcard` reducer references remaining are a *whole-RHS* variable-backed reducer's argument (`total = SUM(population[*])`), which keeps `Wildcard`, a DE-HOISTED array-valued reducer's wildcard arg (`RANK(pop[*], 1)` -- GH #771: RANK never sets `in_reducer`, so the site keeps `Wildcard` and takes the conservative cross-product), and a not-hoistable reducer's argument -- a *dynamic index* (`SUM(pop[idx,*])`, `idx` non-literal) or an ELEMENT-mapped sliced reducer the correspondence declines (GH #756; the positionally-mapped case is hoisted since GH #534 -- both declaration directions since GH #757 -- with the `Iterated` axis carrying the (target, source) dim pair). `model_ltm_reference_sites` reclassifies such a not-hoisted `Direct` `Wildcard` `in_reducer` site as `DynamicIndex` (#514) so the `Wildcard`-shape conservative cross-product never fires from a `Direct` site that *could* have been hoisted (the de-hoisted RANK arg is not hoist-eligible, so it deliberately stays `Wildcard`). (`classify_iterated_dim_shape`'s mapped branch -- a *whole-equation*-iterated subscript like `x[State]` inside `target[State] = x[State] * c`, not a sliced reducer argument -- is a separate path and still classifies as `Bare`.) The mapped case needs a `DimensionsContext` (built from `project_datamodel_dims`), so the IR is recomputed when a dimension's mappings change. This query is also the **LTM front door** (GH #978/#979): occurrence identity is a `SiteId`, a path of `u16` child indices, and exactly three of the walk's `push` sites take a count the AST node's own shape does not bound -- an `Ast::Arrayed` target's equation slots, one builtin call's contents (`BuiltinFn::Mean` is the only variadic variant), and one subscript's index list (`Expr2::from` does NOT narrow it to the subscripted variable's declared arity). Those three are the arms of `SiteWidthAxis`; every other push is a literal fixed by the node. `ast_site_width_rejection` checks each target equation against `MAX_SITE_CHILDREN` BEFORE the walk pushes a single component, a rejected equation records no occurrence at all (so no two references can share a `SiteId`), and `model_ltm_variables` refuses the whole model with a `Warning` naming the variable rather than scoring it from a stream that cannot name every reference. That replaced a reserved-sentinel mechanism threaded through the IR and the ceteris-paribus wrap; because the check is a front door, the walk's `as u16` conversions and `ltm_augment::child_path` are in range by construction, and `db::module_output_ref_in_document_order`'s `None` no longer conflates a width-suppressed reference with a genuine absence (one pre-existing, unrelated source of `None` remains and is documented there: the walker skips a `LOOKUP` table argument as static data, so a composite read there records no occurrence). It is a SEPARATE pre-pass rather than a counter carried on the walk because its descent must be a superset of the walk's: the walk pushes a path component for a `LOOKUP` table argument and then does not recurse, while `ltm_augment::wrap_non_matching_in_previous` and `post_transform::pin_only_source_refs` DO descend through it with `child_path` -- so a builtin nested under a table argument is reachable by consumer path indices and by no producer path at all, and only a check that goes there bounds it. The per-edge `sites` view is name-keyed and path-free, so it stays complete even for a refused model and loop DETECTION (`model_element_causal_edges` / `model_detected_loops`) is unaffected -- what a refusal costs is the scores, not the structure. `SiteChildrenLimitGuard` is the `#[cfg(test)]` limit override that lets the refusal be tested with tiny fixtures. - **`src/db/ltm_ir_tests.rs`** - Tests for the reference-site IR: the per-AST-site `(shape, in_reducer)` contract (the `ref_site_*` regression guards, ported from `src/db/analysis.rs`) plus the public `ClassifiedSite` contract -- `(shape, target_element, routing)` per site, the AC1.4 `StarRange` consistency, and the AC1.5 SIZE / scalar-source-reducer `Direct` routing, each cross-checked against `enumerate_agg_nodes`. - **`src/db/macro_registry.rs`** (a `db` submodule, like `db::ltm_ir`, only to keep `db.rs` under the per-file line cap) - The per-project macro-registry salsa query. `project_macro_registry(db, project) -> MacroRegistryResult` wraps the pure `module_functions::MacroRegistry` (resolver) and exposes the build error. Registry-build *validation* (recursion cycle, duplicate macro name, macro/model name collision, a module inside a macro body) can't be re-derived from the canonical-name-keyed `SourceProject.models` (it collapses duplicate / colliding model names), so the query is demand-driven from two salsa inputs: the ordered pre-dedup `SourceProject.macro_declarations` list (canonical name + `macro_spec`, one entry per project model in declaration order) drives Passes 1-2 (duplicate macro name, macro/model collision), and the macro-marked models' bodies -- read from `models` -- drive Pass 3 (recursion cycle, from the equations) and Pass 4 (a `Variable::Module` in a macro body, from the variable kinds). `build_error_from_inputs` reconstructs a `Vec` in declaration order (canonical name + spec + macro bodies; non-macro bodies empty, which Passes 1-2 don't read and Passes 3-4 skip) and calls the UNCHANGED `MacroRegistry::build` on it, so the produced typed `(ErrorCode, message)` is byte-identical to building over the original datamodel `Vec`. Because the query reads only macro_declarations + macro-marked models' bodies (it skips `macro_spec.is_none()` models' bodies), an ordinary equation edit does NOT invalidate it. **On a build failure the returned registry is EMPTY, and that is load-bearing for CYCLE SAFETY, not just for error quality**: `collect_all_diagnostics` runs every model's passes after emitting `build_error`, and `analysis::analyze_model` never reads `build_error` at all, so only the empty registry stops a rejected macro's call sites from re-synthesizing the implicit module edge `db::project_module_graph` cannot see (a process abort under `panic=abort`). A "keep a partial registry alongside the error" refactor reopens that hole; the argument and its pinning test are on the query's rustdoc. It does NOT accumulate: `build_error` is a plain memoized value that its two consumers read directly -- `compile_project_incremental` (to fail with a clear message) and `collect_all_diagnostics` (to emit the one project-level `Diagnostic`). Accumulating it from inside the query body was wrong twice over: every model's `model_all_diagnostics` subtree reaches this query through `model_module_ident_context`, so the DFS reported N identical copies for an N-model project, and after an unrelated revision bump the deep-verify path recomputed the pruning flags as `Empty` and the diagnostic vanished from the collection entirely. `enclosing_macro_for_var` resolves a macro-body variable's enclosing-macro name (#554, the renamed-intrinsic precedence). - **`src/db/dep_graph.rs`** (a `db` submodule, like `db::ltm_ir` / `db::macro_registry`, only to keep `db.rs` under the per-file line cap) - Owns the **model dependency-graph cycle gate** and its shared relations. The production gate `model_dependency_graph_impl` lives here (the transitive-closure DFS `compute_transitive`/`compute_inner`; the SCC-aware back-edge break `same_resolved_scc`; the SCC-as-collapsed-node transitive accumulation -- every member of a resolved recurrence SCC ends with the identical member-free union of the SCC's *external* successors so the topo sort never re-sees the intra-SCC cycle; and the SCC-contiguous topological runlist sort `topo_sort_str` so a resolved SCC's members are emitted as one byte-stable contiguous block at the SCC's slot for Phase-2-Task-6 combined-fragment injection). **Deterministic initials runlist**: the init set is a `HashSet`, so `model_dependency_graph_impl` sorts it (`init_list.sort_unstable()`) before `topo_sort_str` -- `topo_sort_str` breaks ties (variables with no ordering edge between them) by visit order, so without the sort the same model compiled twice produced different init orderings and therefore different initial values for any unordered pair (GH #595 tracks the deeper soundness gap; the Flows/Stocks runlists are already deterministic by filtering the pre-sorted `var_names`). **dt stock-submodel-output chain-break**: `build_var_info` filters a `submodel·subvar` dt dependency whose `subvar` is a Stock out of the dt phase (a stock breaks the chain, read from the prior timestep) for **every** reader -- not just module-kind variables (`model.rs::module_output_deps` used to state the same rule; GH #568 deleted that walk, so this is now the only statement of it); a non-module reader of a stock submodel output (e.g. `v = SMOOTH(...)·output` where the output is an INTEG stock) would otherwise gain a spurious same-step `reader -> module` edge that `topo_sort_str` breaks arbitrarily, sometimes emitting the module before its input (a stale read each flows step). The init phase keeps the edge (stocks do not break the chain there), so only `dt_deps` are filtered. The per-variable projection `var_runlist_membership` -> `RunlistMembership` lives here too: it returns only the three "is this variable in the initials / flows / stocks runlist" bits `compile_var_fragment` reads, so it backdates and an unrelated dep-graph change no longer invalidates every fragment (GH #964). The dep-graph result types (`SccPhase`, `ResolvedScc`, `ModelDepGraphResult`) and the thin `#[salsa::tracked]` wrapper `crate::db::model_dependency_graph` (keyed on an interned `ModuleInputSet`, empty = the no-inputs case) live here too -- the wrapper delegates straight to `model_dependency_graph_impl` -- and are re-exported at the `db.rs` root. Also holds the single shared **cycle relation** `walk_successors(var_info, name, phase: SccPhase) -> Vec<&Ident>` parameterized by phase: Module/absent ⇒ `[]` in both phases; a Stock ⇒ `[]` in `Dt` only (a stock is a dt sink but NOT an init sink -- the one per-phase difference); otherwise the phase's dep set (`dt_deps` for `Dt`, `initial_deps` for `Initial`) filtered to known targets, with stock-targeted deps dropped in `Dt` only, module-targets kept -- exactly the successor set `compute_inner` iterates per phase. (`SccPhase` is defined here; it already keys `ResolvedScc.phase`/`combine_scc_for_phase`, so the dt/init distinction is the same one this relation makes.) Plus the shared `VarInfo` map builder `build_var_info` (consumed verbatim by `model_dependency_graph_impl` and the `#[cfg(test)]` SCC accessor, so the accessor observes the *exact* `var_info` the engine builds -- never a reconstruction), and the recurrence-SCC element-acyclicity refinement `resolve_recurrence_sccs` / `refine_scc_to_element_verdict` / `symbolic_phase_element_order` (GH #575: the cross-member-comparable symbolic `SymVarRef` element graph; N=1 self-recurrence is just the 1-member case). Defining the relation once and using it in both the gate and the accessor makes the accessor's relation the engine's relation by construction. Also hosts the `#[cfg(test)]` SCC accessor: `dt_cycle_sccs` (uncapped `crate::ltm::scc_components` Tarjan over the `walk_successors(.., SccPhase::Dt)` adjacency → `DtCycleSccs { multi, self_loops }`, sorted/byte-stable), the pure `dt_cycle_sccs_consistency_violation` predicate (functional core), and `dt_cycle_sccs_engine_consistent` (imperative shell -- cross-checks the instrumented SCC set against the engine's real `CircularDependency` flagging on the same compiled model, panics on divergence), plus `array_producing_vars` / `var_noninitial_lowered_exprs` (the set of variables whose engine-lowered non-initial exprs contain an array-producing builtin, over the same `build_var_info` universe). diff --git a/src/simlin-engine/src/db.rs b/src/simlin-engine/src/db.rs index 2f7b85214..f86b27018 100644 --- a/src/simlin-engine/src/db.rs +++ b/src/simlin-engine/src/db.rs @@ -715,21 +715,27 @@ fn module_composite_ports( /// left-to-right order, so taking the first that names `module` is a /// reproducible choice over the SAME set the old scan considered. /// -/// One case this CANNOT distinguish, deliberately: a composite read from a -/// builtin child index no `SiteId` element can address (a variadic call with -/// 65,536+ arguments) has its occurrence suppressed by `walk_all_in_expr`, so it -/// looks identical here to "`to` reads no output of `module`". Both yield `None` -/// and [`module_link_score_equation`] falls through to the signed magnitude-1 -/// `black_box_unit_transfer_equation` instead of a real ceteris-paribus partial. +/// `None` no longer conflates a WIDTH-suppressed reference with a genuine +/// absence. The occurrence stream is complete in that respect for every model +/// that reaches here, because the LTM front door +/// (`db::ltm_ir::model_ltm_reference_sites`' `site_width_rejection`, GH +/// #978/#979) refuses a model holding an equation whose `SiteId` paths could not +/// name every child, and `model_ltm_variables` then emits no link score for it +/// at all. Until that check existed, an over-arity builtin child had its +/// occurrence suppressed and looked identical here to "reads no output", so the +/// caller silently approximated the first with the signed magnitude-1 +/// `black_box_unit_transfer_equation`. /// -/// That is accepted rather than fixed. The outcome is the SAME documented -/// fallback an unlocatable reference has always taken -- an approximation, not a -/// plausible-looking wrong number -- and distinguishing the two would mean -/// threading an "unaddressable" marker through the IR and this query to serve an -/// arity no real model reaches (a 65,536-argument builtin; note the arity is also -/// far past where LTM generation is tractable at all, see GH #977). If a future -/// change makes such a marker cheap, the honest behavior is to skip the edge -/// LOUDLY here rather than silently approximate it. +/// One PRE-EXISTING source of `None` survives, deliberately and unrelated to +/// width: the walker skips a `LOOKUP` table argument as static data without +/// recursing (`ltm_ir.rs`, `BuiltinContents::LookupTable(_) => {}`), and +/// `OccurrenceRef::ModuleOutput` is minted only in the arms it reaches by +/// recursion -- so a composite read as a table argument records no occurrence +/// and lands here as `None` too. That shape is not known to be reachable from a +/// compiling model (a table argument names a graphical function, not a value), +/// and the outcome is the same explicit approximation an unlocatable reference +/// has always taken; it is noted so the next reader does not take `None` for an +/// absolute. fn module_output_ref_in_document_order( db: &dyn Db, model: SourceModel, diff --git a/src/simlin-engine/src/db/ltm/link_scores.rs b/src/simlin-engine/src/db/ltm/link_scores.rs index c1cac98ef..5e840222a 100644 --- a/src/simlin-engine/src/db/ltm/link_scores.rs +++ b/src/simlin-engine/src/db/ltm/link_scores.rs @@ -42,6 +42,12 @@ use super::parse::{ /// (`try_scalar_to_arrayed_link_scores`, `emit_per_element_link_scores`, /// `emit_agg_to_target_link_scores`) so the slot map and `OccurrenceLookup::for_slot` /// cannot disagree. +/// +/// The `as u16` slot conversions below are in range by the LTM front door +/// (`db::ltm_ir::ast_site_width_rejection`), which refuses a target needing more +/// slots than `MAX_SITE_CHILDREN` can tell apart -- counting the trailing +/// default slot this type computes unconditionally, whether or not the target +/// declares a default equation. struct ArrayedSlotMap { /// `None` for a scalar / `ApplyToAll` target (one slot `0`); otherwise the /// canonical-element-key -> slot map plus the default slot (`keys.len()`, diff --git a/src/simlin-engine/src/db/ltm/mod.rs b/src/simlin-engine/src/db/ltm/mod.rs index 28a6ace3c..ba1027eef 100644 --- a/src/simlin-engine/src/db/ltm/mod.rs +++ b/src/simlin-engine/src/db/ltm/mod.rs @@ -1156,6 +1156,54 @@ pub fn model_ltm_variables( }; } + // The LTM front door (GH #978/#979). Occurrence identity is a `SiteId`: a + // path of `u16` child indices. An equation needing more children at one + // level than a `u16` can tell apart -- an `Ast::Arrayed` target with more + // slots, a variadic `MEAN` with more arguments, or a subscript with more + // indices -- would give two distinct references the SAME identity, and the + // ceteris-paribus wrap would then hold or freeze the wrong one and emit a + // plausible, wrong link score. `model_ltm_reference_sites` therefore records + // no occurrence for such an equation, which leaves the stream unable to name + // every reference; scoring the model from it would silently mis-attribute, + // and the per-edge fallbacks would read a MISSING entry as a real + // classification. So refuse the model outright, loudly. + // + // This is a whole-model refusal rather than a per-edge skip because the + // trigger is a property of a TARGET EQUATION, not of an edge: every edge + // into that target is affected, and there is no per-site decline that a + // consumer could tell apart from "no reference here". Loop DETECTION is + // untouched -- `model_element_causal_edges` / `model_detected_loops` read + // the name-keyed, path-free `sites` view, which stays complete -- so what is + // lost is the scores, not the structure. + let ref_sites = crate::db::ltm_ir::model_ltm_reference_sites(db, model, project); + if let Some(rejection) = &ref_sites.site_width_rejection { + let msg = format!( + "LTM analysis was skipped for this model: variable \"{}\" needs {} {} \ + at one level of its equation, more than the {} distinct children one \ + link-score occurrence path can address -- two references would share \ + an identity and be scored as one. No link, loop, or pathway scores \ + were generated for this model.", + rejection.variable, + rejection.count, + rejection.axis.describe(), + rejection.limit, + ); + CompilationDiagnostic(Diagnostic { + model: model.name(db).clone(), + variable: Some(rejection.variable.clone()), + error: DiagnosticError::Assembly(msg), + severity: DiagnosticSeverity::Warning, + }) + .accumulate(db); + return LtmVariablesResult { + vars: vec![], + loop_partitions: indexmap::IndexMap::new(), + agg_recovery_truncated: false, + pathways_truncated: false, + mode: model_ltm_mode(db, model, project), + }; + } + // GH #486's non-Euler rejection is NOT emitted here. The integration // method that the VM actually honors is a single, main-model-governed // property of the assembled simulation (`assemble_simulation`'s `Specs` diff --git a/src/simlin-engine/src/db/ltm_ir.rs b/src/simlin-engine/src/db/ltm_ir.rs index 73fe2a247..264261b46 100644 --- a/src/simlin-engine/src/db/ltm_ir.rs +++ b/src/simlin-engine/src/db/ltm_ir.rs @@ -334,6 +334,252 @@ pub(crate) fn collect_all_reference_sites( collect_all_reference_sites_and_occurrences(target_var, variables, dim_ctx, lookup_dims).0 } +// ── The LTM front door: `SiteId` addressability ──────────────────────────── + +/// The number of children one [`SiteId`] path component can tell apart. +/// +/// A component is a `u16`, so an equation needing more than this many distinct +/// child positions at one level cannot be addressed: two occurrences would share +/// a path, and the ceteris-paribus wrap would then hold or freeze the WRONG +/// reference and emit a plausible, wrong link score -- silent corruption, the +/// failure class this IR exists to eliminate. +/// +/// Rather than making that case safe by threading a reserved sentinel through +/// the IR and the wrap, LTM refuses such a model at the front door: the walk +/// records no occurrence for the offending equation ([`WalkAccum::record_occurrences`]) +/// and `model_ltm_variables` emits no LTM variable for the model at all, with a +/// `Warning` naming the variable. Every `SiteId` component the walk pushes is +/// therefore in range by construction -- which is what lets the pushes be plain +/// conversions and the consumer's `ltm_augment::child_path` be total. +/// +/// Widening the component to `u32` was the alternative. It is rejected: the +/// occurrence IR is salsa-cached per model and every occurrence carries a boxed +/// path, so widening doubles that footprint on every model to serve a width no +/// real model reaches -- and it would not remove the case, only move it. +pub(crate) const MAX_SITE_CHILDREN: usize = u16::MAX as usize + 1; + +#[cfg(test)] +thread_local! { + /// Test-only override of [`site_children_limit`], installed by + /// [`SiteChildrenLimitGuard`]. Lets a test trip the front door with a tiny + /// fixture instead of building an equation wide enough to trip the + /// production constant (per docs/dev/rust.md#test-time-budgets). + static SITE_CHILDREN_LIMIT_OVERRIDE: std::cell::Cell> = + const { std::cell::Cell::new(None) }; +} + +/// The `SiteId` child-count limit in force. [`MAX_SITE_CHILDREN`] in production +/// builds; in `#[cfg(test)]` builds an active [`SiteChildrenLimitGuard`] +/// override takes precedence. +pub(crate) fn site_children_limit() -> usize { + #[cfg(test)] + { + if let Some(limit) = SITE_CHILDREN_LIMIT_OVERRIDE.with(|c| c.get()) { + return limit; + } + } + MAX_SITE_CHILDREN +} + +/// RAII guard (test-only) that lowers [`site_children_limit`] for the current +/// thread for the guard's lifetime, restoring the previous value on drop -- so a +/// panicking test does not leak the override to the next test reusing the +/// thread. +/// +/// `model_ltm_reference_sites` and `model_ltm_variables` are salsa-memoized, so +/// the guard must outlive every call in the test whose limit it controls (a +/// later call on the same `db` would otherwise return the memoized +/// tiny-limit result regardless of the override state). +#[cfg(test)] +pub(crate) struct SiteChildrenLimitGuard { + prev: Option, +} + +#[cfg(test)] +impl SiteChildrenLimitGuard { + pub(crate) fn new(limit: usize) -> Self { + let prev = SITE_CHILDREN_LIMIT_OVERRIDE.with(|c| c.replace(Some(limit))); + Self { prev } + } +} + +#[cfg(test)] +impl Drop for SiteChildrenLimitGuard { + fn drop(&mut self) { + SITE_CHILDREN_LIMIT_OVERRIDE.with(|c| c.set(self.prev)); + } +} + +/// Which child axis of a [`SiteId`] path an equation overran. +/// +/// The rows are exactly the walk's own variable-width `push` sites. Every OTHER +/// push is a literal bounded by the node's shape -- an `Ast::Scalar` / +/// `Ast::ApplyToAll` target's single slot `0`, `Op1`'s one operand, `Op2`'s two, +/// `Expr2::If`'s three, and an `IndexExpr2::Range`'s two halves -- so those axes +/// need no check and have no arm here. +#[derive(Debug, Clone, Copy, PartialEq, Eq, salsa::Update)] +pub(crate) enum SiteWidthAxis { + /// An `Ast::Arrayed` target's equation slots: one per `` equation, + /// plus the trailing default slot. + ArrayedSlots, + /// One builtin call's ordered contents. `BuiltinFn::Mean` is the only + /// variadic variant (every other holds at most five fixed children), so + /// `MEAN(a, b, ...)` is the only equation shape that can reach this. + BuiltinContents, + /// One `Expr2::Subscript`'s index list. The parser accepts any number of + /// comma-separated indices and `Expr2::from` does NOT narrow it to the + /// subscripted variable's declared arity (it simply ignores indices past + /// `dims.len()`), so this axis is bounded by the AST alone. + SubscriptIndices, +} + +impl SiteWidthAxis { + /// Human-readable plural for the diagnostic message. + pub(crate) fn describe(self) -> &'static str { + match self { + SiteWidthAxis::ArrayedSlots => "equation slots (per-element equations plus a default)", + SiteWidthAxis::BuiltinContents => "arguments to one builtin call", + SiteWidthAxis::SubscriptIndices => "indices in one subscript", + } + } +} + +/// Why a model's equations cannot be addressed by a [`SiteId`], as reported by +/// the LTM front door. +/// +/// Carries the first offending variable in the walk's own deterministic order +/// (variables canonical-sorted, then left-to-right DFS), so the emitted +/// diagnostic is reproducible across processes. +#[derive(Debug, Clone, PartialEq, Eq, salsa::Update)] +pub(crate) struct SiteWidthRejection { + /// Canonical name of the target variable whose equation is too wide. + pub variable: String, + /// Which axis overran. + pub axis: SiteWidthAxis, + /// How many children that axis needed. + pub count: usize, + /// The limit in force when the rejection was recorded. + pub limit: usize, +} + +/// The first place `ast` needs more `SiteId` child positions at one level than +/// `limit` can tell apart, in the walk's own left-to-right order. +/// +/// Path-free by construction: it counts children and never builds a path. That +/// is what makes it a FRONT door -- a truncated path the walk had already +/// recorded would be the aliasing bug itself, not a check against it. +/// +/// Its descent is a strict SUPERSET of [`walk_all_in_expr`]'s, and that is the +/// reason this is a separate pre-pass rather than a counter carried on the walk +/// itself: the CONSUMER's path cursor (`ltm_augment::child_path`) descends +/// through nodes the producer never enters, so a check that saw only what the +/// producer sees could not bound the consumer's paths. Two places: +/// +/// - a `LOOKUP` **table argument**. The walk pushes a path component for it and +/// then records nothing (`BuiltinContents::LookupTable(_) => {}`) without +/// recursing -- static table data is not a causal edge. The wrap DOES recurse: +/// `ltm_augment::wrap_non_matching_in_previous`'s LOOKUP arm hands the table +/// argument to `post_transform::pin_only_source_refs` at `child_path(path, i)`, +/// and that function's own `App` arm then descends into every argument at +/// `child_path(path, i)` again. So a builtin nested under a table argument is +/// reachable by consumer path indices and by no producer path at all. +/// - every **subscript index**. The walk skips an index that resolves to a +/// literal element of the subscripted variable's axis; the wrap's skip is a +/// different predicate (the occurrence's `Pinned` axes, or what +/// `pin_source_subscript_indices` leaves unresolved), so the two sets are not +/// the same and only the union bounds both. +/// +/// Descending in both is therefore load-bearing, not conservatism. It is also +/// sound in the other direction: descending where NEITHER side goes could only +/// over-reject a node no path names, never miss one. +fn ast_site_width_rejection( + ast: &crate::ast::Ast, + limit: usize, +) -> Option<(SiteWidthAxis, usize)> { + use crate::ast::Ast; + + match ast { + Ast::Scalar(expr) | Ast::ApplyToAll(_, expr) => expr_site_width_rejection(expr, limit), + Ast::Arrayed(_, per_elem, default_expr, _) => { + // Slots are numbered in canonical element-key-sorted order with the + // default equation the slot AFTER the last element. Reserve that + // trailing slot unconditionally: `db::ltm::link_scores`' + // `ArrayedSlotMap` computes it as `keys.len()` whether or not a + // default equation exists, and routes an unlisted element there. + let slots = per_elem.len() + 1; + if slots > limit { + return Some((SiteWidthAxis::ArrayedSlots, slots)); + } + // Visit slots in the walk's own order so the reported rejection is + // deterministic (a `HashMap` iteration order is not). + let mut elem_keys: Vec<_> = per_elem.keys().collect(); + elem_keys.sort(); + for k in elem_keys { + if let Some(rejection) = expr_site_width_rejection(&per_elem[k], limit) { + return Some(rejection); + } + } + default_expr + .as_ref() + .and_then(|expr| expr_site_width_rejection(expr, limit)) + } + } +} + +/// [`ast_site_width_rejection`]'s recursive half over one expression tree. +/// +/// The `match` is exhaustive with no catch-all arm, so a new `Expr2` variant is +/// a compile error here rather than a silently unchecked axis. +fn expr_site_width_rejection( + expr: &crate::ast::Expr2, + limit: usize, +) -> Option<(SiteWidthAxis, usize)> { + use crate::ast::{Expr2, IndexExpr2}; + use crate::builtins::{BuiltinContents, walk_builtin_expr}; + + match expr { + Expr2::Const(..) | Expr2::Var(..) => None, + Expr2::Subscript(_, indices, _, _) => { + if indices.len() > limit { + return Some((SiteWidthAxis::SubscriptIndices, indices.len())); + } + indices.iter().find_map(|idx| match idx { + IndexExpr2::Expr(e) => expr_site_width_rejection(e, limit), + IndexExpr2::Range(l, r, _) => expr_site_width_rejection(l, limit) + .or_else(|| expr_site_width_rejection(r, limit)), + IndexExpr2::Wildcard(_) + | IndexExpr2::StarRange(_, _) + | IndexExpr2::DimPosition(_, _) => None, + }) + } + Expr2::App(builtin, _, _) => { + // Count with the walker's OWN enumerator, so the front door and the + // walk cannot disagree about how many children a builtin has. + let mut children: Vec<&Expr2> = Vec::new(); + let mut n: usize = 0; + walk_builtin_expr(builtin, |contents| { + n += 1; + match contents { + BuiltinContents::Expr(e) | BuiltinContents::LookupTable(e) => children.push(e), + BuiltinContents::Ident(_, _) => {} + } + }); + if n > limit { + return Some((SiteWidthAxis::BuiltinContents, n)); + } + children + .into_iter() + .find_map(|e| expr_site_width_rejection(e, limit)) + } + Expr2::Op1(_, operand, _, _) => expr_site_width_rejection(operand, limit), + Expr2::Op2(_, left, right, _, _) => expr_site_width_rejection(left, limit) + .or_else(|| expr_site_width_rejection(right, limit)), + Expr2::If(cond, then_e, else_e, _, _) => expr_site_width_rejection(cond, limit) + .or_else(|| expr_site_width_rejection(then_e, limit)) + .or_else(|| expr_site_width_rejection(else_e, limit)), + } +} + /// The bundled accumulators the single walk feeds: the per-source /// [`ReferenceSite`] buckets (the existing per-edge view) and the flat, /// document-ordered [`RawOccurrence`] stream (the per-occurrence view). The @@ -344,19 +590,21 @@ struct WalkAccum<'a> { sites: &'a mut HashMap>, occurrences: &'a mut Vec, path: Vec, - /// `> 0` while the walk is inside a builtin child whose index a [`SiteId`] - /// element cannot address (see [`site_child_index`]). + /// `false` when the LTM front door rejected this target's equation as too + /// wide for a [`SiteId`] to address (see [`ast_site_width_rejection`]). /// - /// Only the OCCURRENCE view is suppressed there; `push_ref_site` keeps - /// running, so the per-edge view -- and therefore `model_edge_shapes` and - /// the element causal graph -- still sees every reference with its real - /// shape. Suppressing the ref site too would leave the IR with no entry for - /// the edge, and consumers default a missing entry to a single `Bare` site, - /// which would MISCLASSIFY a `FixedIndex`/`DynamicIndex` reference and emit - /// wrong element edges. It is a depth COUNTER rather than a flag because the - /// suppression must cover the unaddressable child's whole subtree: emitting - /// a nested occurrence at a truncated path would alias a sibling's `SiteId`. - suppress_occurrences: usize, + /// Only the OCCURRENCE view is dropped; `push_ref_site` keeps running, so + /// the per-edge view -- and therefore `model_edge_shapes` and the element + /// causal graph -- still sees every reference with its real shape. That view + /// is name-keyed and carries no path, so it is unaffected by the width; + /// dropping it too would leave the IR with no entry for the edge, and + /// consumers default a missing entry to a single `Bare` site, which would + /// MISCLASSIFY a `FixedIndex`/`DynamicIndex` reference and emit wrong + /// element edges. The occurrence view instead records NOTHING, so no two + /// occurrences of a rejected equation can share a `SiteId` -- and + /// `model_ltm_variables` refuses the whole model, so no consumer ever reads + /// an occurrence stream that is missing entries. + record_occurrences: bool, } impl WalkAccum<'_> { @@ -368,9 +616,9 @@ impl WalkAccum<'_> { /// does NOT mean "no reference": consumers that find no IR entry for an edge /// fall back to a single `Bare` site, so skipping a `FixedIndex`/`DynamicIndex` /// reference MISCLASSIFIES it and emits wrong element edges and link scores. - /// In the occurrence view, absence is safe and sometimes required -- a - /// SiteId-unaddressable child must record nothing, or the wrap's path lookup - /// aliases a sibling. That is why `suppress_occurrences` gates only + /// In the occurrence view, absence is safe -- the wrap treats a miss as "not + /// a recorded causal reference" and a front-door-rejected equation records + /// none at all. That is why `record_occurrences` gates only /// `push_occurrence`. fn push_ref_site( &mut self, @@ -407,10 +655,11 @@ impl WalkAccum<'_> { reducer_keys: &[String], index_nested: bool, ) { - // Inside an unaddressable builtin child, record NO occurrence: its path - // could not be reproduced by a consumer, and emitting it at a truncated - // path would alias a sibling. See `suppress_occurrences`. - if self.suppress_occurrences > 0 { + // A front-door-rejected equation records NO occurrence: its paths could + // not tell every child apart, and two occurrences sharing a `SiteId` is + // exactly the aliasing the identity exists to prevent. See + // `record_occurrences`. + if !self.record_occurrences { return; } self.occurrences.push(RawOccurrence { @@ -424,6 +673,17 @@ impl WalkAccum<'_> { } } +/// The three views one target-equation walk produces: the per-EDGE +/// [`ReferenceSite`] buckets (name-keyed, path-free), the flat document-ordered +/// [`RawOccurrence`] stream (`SiteId`-keyed), and the LTM front door's verdict on +/// that equation (`Some((axis, count))` when it needs more children at one level +/// than a `SiteId` component can tell apart). +type WalkedTarget = ( + HashMap>, + Vec, + Option<(SiteWidthAxis, usize)>, +); + /// Walk a target's AST once, producing BOTH the per-source [`ReferenceSite`] /// map (the per-edge view current consumers read) AND the flat, document-order /// [`RawOccurrence`] stream (the per-occurrence view the ceteris-paribus @@ -433,17 +693,25 @@ impl WalkAccum<'_> { /// composites (which are not model-variable keys) -- and it OMITS an index /// token that is a literal element selector (the A2a bug fix; see the /// `Subscript` arm of [`walk_all_in_expr`]). +/// +/// The third return is the LTM front door's verdict on THIS equation +/// ([`ast_site_width_rejection`], run before the walk starts). When it is +/// `Some`, the occurrence stream is empty by construction -- no `SiteId` is +/// minted for an equation whose paths could not tell every child apart -- while +/// the per-edge map is complete as always. fn collect_all_reference_sites_and_occurrences( target_var: &crate::variable::Variable, variables: &HashMap, crate::variable::Variable>, dim_ctx: &crate::dimensions::DimensionsContext, lookup_dims: &mut impl FnMut(&str) -> Vec, -) -> (HashMap>, Vec) { +) -> WalkedTarget { let mut sites: HashMap> = HashMap::new(); let mut occurrences: Vec = Vec::new(); let Some(ast) = target_var.ast() else { - return (sites, occurrences); + return (sites, occurrences, None); }; + // The front door, before a single path component is pushed. + let width_rejection = ast_site_width_rejection(ast, site_children_limit()); // The target equation's iterated dimensions drive the #511 iterated- // subscript recognition; `Ast::Scalar` has none. let target_iterated_dims: Vec = match ast { @@ -464,7 +732,7 @@ fn collect_all_reference_sites_and_occurrences( sites: &mut sites, occurrences: &mut occurrences, path: Vec::new(), - suppress_occurrences: 0, + record_occurrences: width_rejection.is_none(), }; match ast { crate::ast::Ast::Scalar(expr) | crate::ast::Ast::ApplyToAll(_, expr) => { @@ -486,7 +754,9 @@ fn collect_all_reference_sites_and_occurrences( // Per-element expressions: visit slots in canonical element-key // order so the per-source site Vecs (and the occurrence stream / // its `SiteId`s) are deterministic. The slot index is the first - // `SiteId` path element. + // `SiteId` path element; the front door above refused this + // equation if it needs more slots than `MAX_SITE_CHILDREN` can + // tell apart, so both conversions are in range by construction. let mut elem_keys: Vec<_> = subscript_map.keys().collect(); elem_keys.sort(); for (slot, k) in elem_keys.iter().enumerate() { @@ -521,7 +791,7 @@ fn collect_all_reference_sites_and_occurrences( } } } - (sites, occurrences) + (sites, occurrences, width_rejection) } /// If `ident` is a module-qualified output composite (`module·port`, e.g. @@ -597,50 +867,6 @@ fn classify_occurrence_axes( .collect() } -/// The reserved [`SiteId`] path component meaning "this child is not -/// addressable". [`site_child_index`] NEVER returns it, so a path containing it -/// cannot equal any recorded occurrence's `SiteId` -- which is exactly what lets -/// the ceteris-paribus wrap append it for an over-arity child and be *certain* -/// the lookup misses instead of aliasing an unrelated sibling -/// (`ltm_augment::child_path`). -pub(crate) const UNADDRESSABLE_CHILD: u16 = u16::MAX; - -/// The [`SiteId`] child index for a builtin's `n`-th content, or `None` when the -/// arity exceeds what one path element can address. -/// -/// A `SiteId` element is a `u16`, which every AST-shaped child count fits by -/// construction (`Op1`/`Op2`/`If`/`Subscript` arity is bounded by the node). -/// Builtin arity is NOT: `MEAN`/`SUM` and friends are variadic, so a call with -/// 65,536+ arguments is representable in the AST. -/// -/// Returning `None` -- rather than wrapping -- is the whole point. A wrapped -/// index would RE-USE an earlier child's `SiteId`, so the wrap's path lookup -/// would silently return a DIFFERENT occurrence: if the two children carry -/// different subscript shapes, the wrap holds or freezes the wrong reference and -/// emits a plausible, wrong link score. That is silent corruption, the failure -/// class this IR exists to eliminate. -/// -/// The caller records NO occurrence for an unaddressable child. That is the loud -/// outcome, because it routes into machinery that already exists: a live-source -/// subscript with no occurrence at its tracked path (on a non-empty stream) sets -/// `WrapOutcome::missing_occurrence`, so the partial is abandoned and the -/// db-bearing emitter warns and skips. The per-EDGE `ReferenceSite` view is -/// unaffected (the caller still pushes those), so the causal graph keeps every -/// edge -- only the SiteId-addressable occurrence view declines to name a child -/// it cannot address. -/// -/// Widening the path element to `u32` was the alternative. It is rejected: the -/// occurrence IR is salsa-cached per model and every occurrence carries a boxed -/// path, so widening doubles that footprint on every model to serve an arity no -/// real model reaches -- and it would not remove the case, only move it. -/// -/// [`UNADDRESSABLE_CHILD`] is excluded from the addressable range so it remains a -/// value no recorded `SiteId` can contain; that reserves one child index out of -/// 65,536 and buys the consumer a provably-unmatchable path to descend under. -pub(crate) fn site_child_index(n: usize) -> Option { - u16::try_from(n).ok().filter(|&i| i != UNADDRESSABLE_CHILD) -} - /// Recursive helper for [`collect_all_reference_sites_and_occurrences`]: /// left-to-right DFS over an `Expr2` tree, pushing one [`ReferenceSite`] per /// model-variable reference (bucketed by source name) AND one @@ -773,6 +999,11 @@ fn walk_all_in_expr( // Everything else is genuine index content: recurse with // `index_nested = true`, so a model-variable dynamic index // (`arr[from]`) is marked reachable only through a subscript index. + // + // An index list is NOT narrowed to the subscripted variable's + // declared arity by `Expr2::from` (it ignores indices past + // `dims.len()`), so this is a third variable-width axis -- covered + // by the same front door, which makes the conversion in range. for (i, idx) in indices.iter().enumerate() { if resolve_literal_index(idx, &from_dims).is_some() { continue; @@ -824,23 +1055,15 @@ fn walk_all_in_expr( if pushed_reducer_key { reducer_keys.push(crate::patch::expr2_to_string(expr)); } - // Builtin arity is the one child count not bounded by the AST shape - // (`MEAN(a, b, ...)` is variadic), so it is the only place a `SiteId` - // element can run out of range. `site_child_index` returns `None` - // past the addressable range and we then record NO occurrence for - // that child -- see its rustdoc for why that is the loud outcome. + // Builtin arity is one of the three child counts not bounded by the + // AST node's own shape (`BuiltinFn::Mean` is the only variadic + // variant). The LTM front door -- `ast_site_width_rejection`, run + // before this walk -- refuses an equation whose arity exceeds + // `MAX_SITE_CHILDREN`, so the conversion below is in range by + // construction and no two children can share a path component. let mut child_n: usize = 0; walk_builtin_expr(builtin, |contents| { - // An unaddressable child still gets its REFERENCE SITES walked - // (edges and their shapes must stay complete); only the - // SiteId-keyed occurrence view is suppressed, for it and its - // whole subtree. - let addressable = site_child_index(child_n); - let suppressed = addressable.is_none(); - if suppressed { - acc.suppress_occurrences += 1; - } - acc.path.push(addressable.unwrap_or(UNADDRESSABLE_CHILD)); + acc.path.push(child_n as u16); match contents { BuiltinContents::Ident(id, _) => { let canonical = Ident::::new(id); @@ -883,9 +1106,6 @@ fn walk_all_in_expr( BuiltinContents::LookupTable(_) => {} } acc.path.pop(); - if suppressed { - acc.suppress_occurrences -= 1; - } child_n += 1; }); if pushed_reducer_key { @@ -1318,10 +1538,20 @@ struct RawOccurrence { /// consumer reads it (A2b is the first). Like `sites`, the HashMap *key* /// order is irrelevant (consumers sort keys themselves); only each value /// `Vec`'s order is load-bearing for salsa determinism. +/// +/// `site_width_rejection` is the LTM front door's verdict for the whole model +/// (GH #978/#979). When it is `Some`, at least one target equation needs more +/// children at one path level than a `SiteId` component can tell apart, and that +/// equation contributed NO occurrences -- so `occurrences` is incomplete and +/// `model_ltm_variables` refuses the model outright rather than scoring it from +/// a stream that cannot name every reference. `sites` is unaffected either way: +/// the per-edge view is name-keyed, carries no path, and feeds the element +/// causal graph and `model_detected_loops`, which stay correct. #[derive(Debug, Clone, Default, PartialEq, Eq, salsa::Update)] pub(crate) struct LtmReferenceSitesResult { pub sites: HashMap<(String, String), Vec>, pub occurrences: HashMap>, + pub site_width_rejection: Option, } /// Classify every causal-edge reference site in `model` exactly once. @@ -1339,6 +1569,13 @@ pub(crate) struct LtmReferenceSitesResult { /// text matches one of the site's enclosing reducer keys. That site-precise /// key check prevents a hoisted sibling reducer from claiming a declined /// sibling read on the same `(from, to)` edge (GH #793). +/// +/// This is also the LTM front door (GH #978/#979): each target equation is +/// checked for `SiteId` addressability BEFORE its walk pushes a single path +/// component, and the first failure is reported on +/// [`LtmReferenceSitesResult::site_width_rejection`]. A rejected equation +/// contributes no occurrence at all, so no recorded `SiteId` can be a path two +/// children share, and `model_ltm_variables` refuses to score the model. #[salsa::tracked(returns(ref))] pub(crate) fn model_ltm_reference_sites( db: &dyn Db, @@ -1385,6 +1622,10 @@ pub(crate) fn model_ltm_reference_sites( let mut sites: HashMap<(String, String), Vec> = HashMap::new(); let mut occurrences: HashMap> = HashMap::new(); + // The LTM front door's model-level verdict: the FIRST target equation (in + // the canonical-sorted visit order above) too wide for a `SiteId` to + // address. `model_ltm_variables` reads it and refuses the model. + let mut site_width_rejection: Option = None; for to_name in to_names { let to_var = &variables[to_name]; @@ -1393,12 +1634,28 @@ pub(crate) fn model_ltm_reference_sites( // One walk feeds BOTH views: the per-source `ReferenceSite` buckets // (per-edge, existing consumers) and the flat, document-ordered // `RawOccurrence` stream (per-occurrence, the A2b transform). - let (raw_by_source, raw_occurrences) = collect_all_reference_sites_and_occurrences( - to_var, - &variables, - dim_ctx, - &mut lookup_dims, - ); + let (raw_by_source, raw_occurrences, width_rejection) = + collect_all_reference_sites_and_occurrences( + to_var, + &variables, + dim_ctx, + &mut lookup_dims, + ); + // Record the rejection BEFORE the reference-free skip below: an + // over-wide equation need not reference any model variable at all + // (`MEAN` of 65,536 constants), and losing its verdict there would let + // the model be scored from an occurrence stream that silently dropped + // the whole equation. + if site_width_rejection.is_none() + && let Some((axis, count)) = width_rejection + { + site_width_rejection = Some(SiteWidthRejection { + variable: to_name_str.to_string(), + axis, + count, + limit: site_children_limit(), + }); + } // A target that references ONLY module-qualified outputs has no // `ReferenceSite` (module·port is not a model-variable key) but does // carry occurrences, so gate the skip on both being empty. @@ -1540,7 +1797,11 @@ pub(crate) fn model_ltm_reference_sites( } } - LtmReferenceSitesResult { sites, occurrences } + LtmReferenceSitesResult { + sites, + occurrences, + site_width_rejection, + } } #[cfg(test)] diff --git a/src/simlin-engine/src/db/ltm_ir_tests.rs b/src/simlin-engine/src/db/ltm_ir_tests.rs index 91490c0d2..b2c5a8aaa 100644 --- a/src/simlin-engine/src/db/ltm_ir_tests.rs +++ b/src/simlin-engine/src/db/ltm_ir_tests.rs @@ -1647,118 +1647,618 @@ mod occurrence_ir_tests { } } -/// F3: a `SiteId` child index must never WRAP. The addressable range is pinned -/// at its exact boundary, so a variadic builtin's 65,536th content declines to -/// be addressed rather than re-using child 0's path. -/// -/// Pinned on the pure boundary function rather than by building a 65,536-argument -/// `MEAN` call: the fixture would be enormous and slow (the 3-minute suite cap), -/// while the boundary is the entire property. The consequence of getting this -/// wrong is silent -- a wrapped index makes the wrap's lookup return a different -/// occurrence, so it holds or freezes the wrong reference and emits a plausible -/// wrong score -- which is why the guard returns `None` instead of a wrapped -/// value, and why the walker records no occurrence for such a child. -#[test] -fn site_child_index_declines_rather_than_wrapping() { - use super::site_child_index; - - assert_eq!(site_child_index(0), Some(0)); - assert_eq!(site_child_index(1), Some(1)); - // The last addressable child: one below `UNADDRESSABLE_CHILD`, which is - // reserved as the consumer's provably-unmatchable sentinel (see the sibling - // test `unaddressable_child_sentinel_is_never_a_real_child_index`). - assert_eq!(site_child_index(65_534), Some(65_534)); - assert_eq!(site_child_index(65_535), None); - // Past the range must DECLINE, not wrap to 0 (which would collide with the - // first child's SiteId and silently mis-address the occurrence). - assert_eq!(site_child_index(65_536), None); - assert_eq!(site_child_index(65_537), None); - assert_eq!(site_child_index(usize::MAX), None); -} +// ── Layer 4: the LTM front door (GH #978 / #979) ─────────────────────────── +// +// `SiteId` occurrence identity is a path of `u16` child indices. Three of the +// walk's `push` sites take a count that the AST node's own shape does NOT bound +// -- and they are exactly the three arms of `SiteWidthAxis`: +// +// | axis | what varies | reachable via | +// |---------------------|------------------------------------|-----------------------| +// | `ArrayedSlots` | `Ast::Arrayed` per-element slots | any `` list | +// | `BuiltinContents` | one builtin call's contents | `MEAN` (the only | +// | | | variadic `BuiltinFn`) | +// | `SubscriptIndices` | one subscript's index list | any `x[a, b, ...]` | +// +// Every OTHER push is a literal fixed by the node (`Ast::Scalar`/`ApplyToAll`'s +// single slot `0`, `Op1`'s one operand, `Op2`'s two, `Expr2::If`'s three, an +// `IndexExpr2::Range`'s two halves), which is why they have no arm and no test +// row: `expr_site_width_rejection`'s `match` is exhaustive with no catch-all, so +// a new variant with a variable child count is a compile error there. +// +// Each axis gets ONE fixture asserted in BOTH directions -- admitted at the +// production limit, refused when the limit is lowered below its width -- so the +// accept half cannot pass vacuously. +mod front_door_tests { + use super::*; + use crate::db::ltm_ir::{ + MAX_SITE_CHILDREN, SiteChildrenLimitGuard, SiteWidthAxis, model_ltm_reference_sites, + }; + + /// Sync `project` onto a FRESH db and hand `model_ltm_reference_sites`' full + /// result for `main` to `body`. A fresh db per call is required: the query is + /// salsa-memoized, so re-running it on one db under a different + /// `SiteChildrenLimitGuard` would return the first limit's answer. + fn with_ir(project: &TestProject, body: impl FnOnce(&LtmReferenceSitesResult)) { + let datamodel = project.build_datamodel(); + let db = SimlinDb::default(); + let sync = sync_from_datamodel(&db, &datamodel); + body(model_ltm_reference_sites( + &db, + sync.models["main"].source, + sync.project, + )); + } -/// The unaddressable-child sentinel must be a value `site_child_index` NEVER -/// emits. That reservation is what makes the consumer's fallback provable: the -/// ceteris-paribus wrap appends `UNADDRESSABLE_CHILD` for an over-arity child, so -/// if the producer could also emit it, a recorded occurrence would share the -/// path and the lookup would alias exactly the sibling it is trying to avoid. -#[test] -fn unaddressable_child_sentinel_is_never_a_real_child_index() { - use super::{UNADDRESSABLE_CHILD, site_child_index}; - - // The last ADDRESSABLE index is one below the sentinel. - assert_eq!(site_child_index(65_534), Some(65_534)); - assert_eq!(UNADDRESSABLE_CHILD, u16::MAX); - // The sentinel's own numeric position is declined, not returned. - assert_eq!(site_child_index(UNADDRESSABLE_CHILD as usize), None); - // ...so no `n` whatsoever maps to it. - for n in [0usize, 1, 65_534, 65_535, 65_536, usize::MAX] { - assert_ne!( - site_child_index(n), - Some(UNADDRESSABLE_CHILD), - "n = {n} must not map to the reserved sentinel" - ); + /// An `Ast::Arrayed` target with three per-element equations, so the walk + /// numbers four slots (three elements plus the trailing default slot + /// `ArrayedSlotMap` addresses unconditionally). + fn arrayed_slots_fixture() -> TestProject { + TestProject::new("main") + .named_dimension("Region", &["a", "b", "c"]) + .array_aux("population[Region]", "100") + .array_flow_with_ranges( + "births[Region]", + vec![ + ("a", "population[a] * 0.1"), + ("b", "population[b] * 0.2"), + ("c", "population[c] * 0.3"), + ], + ) + } + + /// A three-argument `MEAN`, the only variadic `BuiltinFn`. + fn builtin_contents_fixture() -> TestProject { + TestProject::new("main") + .scalar_aux("a", "1") + .scalar_aux("b", "2") + .scalar_aux("c", "3") + .scalar_aux("avg", "MEAN(a, b, c)") + } + + /// A two-index subscript. The index count is bounded by the AST alone: + /// `Expr2::from` does not narrow it to the subscripted variable's declared + /// arity (it simply ignores indices past `dims.len()`), which is why this + /// axis needs the front door despite GH #979 claiming it was bounded. + fn subscript_indices_fixture() -> TestProject { + TestProject::new("main") + .named_dimension("D1", &["x", "y"]) + .named_dimension("D2", &["p", "q"]) + .array_aux("matrix[D1,D2]", "1") + .array_aux("total[D1,D2]", "matrix[D1, D2] * 2") + } + + /// Every axis is admitted at the production limit: an ordinary model records + /// its occurrences and carries no rejection. This is the control for the + /// three refusal tests below -- each uses the SAME fixture with the limit + /// lowered, so the refusal cannot be an artifact of the fixture. + #[test] + fn production_limit_admits_every_axis() { + for (name, project, target) in [ + ("arrayed slots", arrayed_slots_fixture(), "births"), + ("builtin contents", builtin_contents_fixture(), "avg"), + ("subscript indices", subscript_indices_fixture(), "total"), + ] { + with_ir(&project, |ir| { + assert_eq!( + ir.site_width_rejection, None, + "{name}: an ordinary model must pass the front door" + ); + assert!( + ir.occurrences.contains_key(target), + "{name}: `{target}` must record its occurrences; got {:?}", + ir.occurrences.keys().collect::>() + ); + }); + } + } + + /// The boundary itself, on all three axes: an equation needing EXACTLY + /// `limit` children is fully addressable (indices `0 ..= limit - 1`) and must + /// be ADMITTED. The comparison is `>`, and off-by-one to `>=` is a real + /// hazard in the direction this session says to measure rather than reason + /// about: it would refuse a legal equation and cost the whole model every + /// link, loop and pathway score, silently. + /// + /// These rows reuse the refusal tests' own fixtures, one limit higher, so + /// each axis is asserted at `n == limit` (accept) and `n == limit + 1` + /// (refuse) against the same equation. Without them, `>` -> `>=` on the + /// builtin and subscript axes left the entire suite green. + #[test] + fn a_width_exactly_at_the_limit_is_admitted_on_every_axis() { + // ArrayedSlots: 3 element equations + the reserved default slot = 4. + { + let _guard = SiteChildrenLimitGuard::new(4); + with_ir(&arrayed_slots_fixture(), |ir| { + assert_eq!( + ir.site_width_rejection, None, + "4 slots at limit 4 are all addressable (0..=3)" + ); + assert!(ir.occurrences.contains_key("births")); + }); + } + // BuiltinContents: `MEAN(a, b, c)` yields exactly 3 contents. + { + let _guard = SiteChildrenLimitGuard::new(3); + with_ir(&builtin_contents_fixture(), |ir| { + assert_eq!( + ir.site_width_rejection, None, + "3 builtin contents at limit 3 are all addressable (0..=2)" + ); + assert!(ir.occurrences.contains_key("avg")); + }); + } + // SubscriptIndices: `matrix[D1, D2]` carries exactly 2 indices. + { + let _guard = SiteChildrenLimitGuard::new(2); + with_ir(&subscript_indices_fixture(), |ir| { + assert_eq!( + ir.site_width_rejection, None, + "2 subscript indices at limit 2 are all addressable (0..=1)" + ); + assert!(ir.occurrences.contains_key("total")); + }); + } + } + + #[test] + fn lowered_limit_refuses_an_over_wide_arrayed_slot_count() { + // Three element equations need four slots; a limit of three cannot tell + // the fourth apart from the first. + let _guard = SiteChildrenLimitGuard::new(3); + with_ir(&arrayed_slots_fixture(), |ir| { + let rejection = ir + .site_width_rejection + .as_ref() + .expect("a 4-slot target must be refused at limit 3"); + assert_eq!(rejection.variable, "births"); + assert_eq!(rejection.axis, SiteWidthAxis::ArrayedSlots); + assert_eq!(rejection.count, 4, "3 element equations plus the default"); + assert_eq!(rejection.limit, 3); + + // The occurrence view records NOTHING for the refused equation, so + // no two of its references can share a `SiteId`... + assert!( + !ir.occurrences.contains_key("births"), + "a refused equation must mint no SiteId; got {:?}", + ir.occurrences.get("births") + ); + // ...while the per-edge view stays complete AND keeps its real + // shapes. A missing entry there is read as a single `Bare` site by + // `model_element_causal_edges`, which would MISCLASSIFY these + // literal-element reads and emit wrong element edges. + let edge = ir + .sites + .get(&("population".to_string(), "births".to_string())) + .expect("the per-edge view is path-free and must survive refusal"); + assert_eq!(edge.len(), 3, "one site per element equation: {edge:?}"); + for site in edge { + assert!( + matches!(site.shape, RefShape::FixedIndex(_)), + "the edge must keep its real shape, not degrade to Bare: {site:?}" + ); + } + }); + } + + #[test] + fn lowered_limit_refuses_an_over_arity_builtin() { + let _guard = SiteChildrenLimitGuard::new(2); + with_ir(&builtin_contents_fixture(), |ir| { + let rejection = ir + .site_width_rejection + .as_ref() + .expect("a 3-argument MEAN must be refused at limit 2"); + assert_eq!(rejection.variable, "avg"); + assert_eq!(rejection.axis, SiteWidthAxis::BuiltinContents); + assert_eq!(rejection.count, 3); + assert!(!ir.occurrences.contains_key("avg")); + // The edges into `avg` survive with their real shapes. + for from in ["a", "b", "c"] { + let edge = ir + .sites + .get(&(from.to_string(), "avg".to_string())) + .unwrap_or_else(|| panic!("edge {from}->avg must survive refusal")); + assert_eq!(edge.len(), 1, "{from}: {edge:?}"); + } + }); + } + + #[test] + fn lowered_limit_refuses_an_over_wide_subscript() { + let _guard = SiteChildrenLimitGuard::new(1); + with_ir(&subscript_indices_fixture(), |ir| { + let rejection = ir + .site_width_rejection + .as_ref() + .expect("a 2-index subscript must be refused at limit 1"); + assert_eq!(rejection.variable, "total"); + assert_eq!(rejection.axis, SiteWidthAxis::SubscriptIndices); + assert_eq!(rejection.count, 2); + assert!(!ir.occurrences.contains_key("total")); + let edge = ir + .sites + .get(&("matrix".to_string(), "total".to_string())) + .expect("edge matrix->total must survive refusal"); + assert_eq!(edge.len(), 1, "{edge:?}"); + assert_eq!( + edge[0].shape, + RefShape::Bare, + "`matrix[D1,D2]` in an A2A-over-[D1,D2] body is the same-element read" + ); + }); + } + + /// The premise that makes `SubscriptIndices` a genuinely unbounded axis -- + /// verified by running the lowering, not by reading it. + /// + /// GH #979 asserts this axis is "bounded by the reference's declared + /// subscript arity". It is not: `Expr2::from`'s `Expr1::Subscript` arm + /// collects EVERY index and only consults `dims[i]` while `i < dims.len()`, + /// so a subscript with more indices than the variable declares reaches the + /// IR with all of them. (`classify_occurrence_axes` already accounts for + /// that shape, calling it an arity mismatch.) If a future change narrows the + /// index list during lowering, this reds -- and the axis, with its front-door + /// arm, can be retired. + #[test] + fn a_subscript_keeps_more_indices_than_the_variable_declares() { + let _guard = SiteChildrenLimitGuard::new(1); + let project = TestProject::new("main") + .indexed_dimension("D", 3) + .array_aux("arr[D]", "1") + // `arr` declares ONE dimension; this reference carries two indices. + .scalar_aux("target", "arr[1, 2]"); + with_ir(&project, |ir| { + let rejection = ir + .site_width_rejection + .as_ref() + .expect("an over-arity subscript must reach the IR with both indices"); + assert_eq!(rejection.variable, "target"); + assert_eq!(rejection.axis, SiteWidthAxis::SubscriptIndices); + assert_eq!( + rejection.count, 2, + "both indices survive lowering, though `arr` declares one dimension" + ); + }); + } + + /// The check must find an over-wide node wherever it sits, so it has to + /// descend every child edge it can descend. The rows below are ELEVEN, one + /// per recursive call in `expr_site_width_rejection` -- `Op1`'s operand, + /// `Op2`'s two, `If`'s three, an `App`'s `Expr` content and its + /// `LookupTable` content, a `Subscript`'s `IndexExpr2::Expr` and its + /// `Range`'s two halves. (`Const`/`Var` are leaves; `Wildcard`, `StarRange` + /// and `DimPosition` carry no `Expr2`. Neither has a row because neither has + /// a call.) Deleting any one descent reds exactly its row. + /// + /// The `LookupTable` and `Range` rows exist because a first version of this + /// test had 8 rows while the checker had 11 calls, and deleting either + /// descent left the whole suite green. The `LookupTable` row is the one that + /// matters most: that descent is not conservatism, it is the only coverage of + /// a path the CONSUMER builds and the producer walk never does (see + /// `ast_site_width_rejection`'s rustdoc). + #[test] + fn the_check_descends_every_expression_edge() { + // A 3-argument `MEAN` is over-wide at limit 2; every other node in these + // equations has at most 2 children, so a row can only fail by failing to + // reach the `MEAN`. (`LOOKUP`'s two contents and a `Range`'s two halves + // are each 2, exactly at the limit.) + let rows: &[(&str, &str)] = &[ + ("Op1 operand", "-MEAN(a, b, c)"), + ("Op2 left", "MEAN(a, b, c) + 1"), + ("Op2 right", "1 + MEAN(a, b, c)"), + ("If cond", "IF MEAN(a, b, c) > 0 THEN 1 ELSE 2"), + ("If then", "IF a > 0 THEN MEAN(a, b, c) ELSE 2"), + ("If else", "IF a > 0 THEN 1 ELSE MEAN(a, b, c)"), + ("App Expr content", "ABS(MEAN(a, b, c))"), + ("App LookupTable content", "LOOKUP(MEAN(a, b, c), 1)"), + ("Subscript IndexExpr2::Expr", "arr[MEAN(a, b, c)]"), + ("Subscript Range low", "SUM(arr[MEAN(a, b, c):3])"), + ("Subscript Range high", "SUM(arr[1:MEAN(a, b, c)])"), + ]; + for (edge, equation) in rows { + let _guard = SiteChildrenLimitGuard::new(2); + let project = TestProject::new("main") + .indexed_dimension("D", 3) + .array_aux("arr[D]", "1") + .scalar_aux("a", "1") + .scalar_aux("b", "2") + .scalar_aux("c", "3") + .scalar_aux("target", equation); + with_ir(&project, |ir| { + let rejection = ir + .site_width_rejection + .as_ref() + .unwrap_or_else(|| panic!("{edge}: `{equation}` must be refused at limit 2")); + assert_eq!(rejection.variable, "target", "{edge}"); + assert_eq!(rejection.axis, SiteWidthAxis::BuiltinContents, "{edge}"); + assert_eq!(rejection.count, 3, "{edge}"); + }); + } } -} -/// Occurrence suppression inside an unaddressable builtin child must NOT -/// suppress the per-EDGE reference site. -/// -/// The two views serve different consumers. The occurrence view is -/// SiteId-keyed, so a child the path cannot address must record nothing (else -/// the wrap aliases a sibling). The per-edge view is name-keyed and feeds -/// `model_edge_shapes` and the element causal graph -- and a MISSING IR entry -/// there does not mean "no reference": consumers default it to a single `Bare` -/// site, which would misclassify a `FixedIndex`/`DynamicIndex` reference and -/// emit wrong element edges and link scores. So the edge must keep its real -/// shape even when its occurrence is dropped. -/// -/// Pinned on the accumulator rather than through a 65,536-argument builtin: the -/// suppression counter is only ever raised at that arity, so a fixture-driven -/// test would be enormous while this asserts the exact invariant. -#[test] -fn suppressed_occurrences_still_record_edge_sites() { - use super::{RefShape, ReferenceSite, WalkAccum}; - use std::collections::HashMap; - - let mut sites: HashMap> = HashMap::new(); - let mut occurrences = Vec::new(); - { - let mut acc = WalkAccum { - sites: &mut sites, - occurrences: &mut occurrences, - path: vec![0], - // As if inside a builtin child whose index cannot be addressed. - suppress_occurrences: 1, + /// The same, one row per `Ast` shape: the check runs on whichever equation + /// form the target carries, including an `Ast::Arrayed`'s per-element AND + /// default expressions (which `ast_site_width_rejection` walks separately + /// from the slot count itself). + #[test] + fn the_check_descends_every_equation_shape() { + // Limit 3 with a FOUR-argument `MEAN`, so the arrayed rows' own slot + // counts (at most 2 elements plus the default = 3) stay inside the limit + // and the rejection can only come from descending into the equation. + const OVER_WIDE: &str = "MEAN(a, b, c, d)"; + let leaves = |p: TestProject| { + p.scalar_aux("a", "1") + .scalar_aux("b", "2") + .scalar_aux("c", "3") + .scalar_aux("d", "4") }; - acc.push_ref_site( - "pop", - RefShape::FixedIndex(vec!["nyc".to_string()]), - None, - &[], + let scalar = leaves(TestProject::new("main")).scalar_aux("target", OVER_WIDE); + let apply_to_all = leaves(TestProject::new("main").named_dimension("Region", &["x", "y"])) + .array_aux("target[Region]", OVER_WIDE); + // A per-element equation, in the slot the walk numbers first. + let arrayed_element = + leaves(TestProject::new("main").named_dimension("Region", &["x", "y"])) + .array_flow_with_ranges("target[Region]", vec![("x", OVER_WIDE), ("y", "1")]); + // The default (EXCEPT) equation, the slot AFTER the last element. + let arrayed_default = + leaves(TestProject::new("main").named_dimension("Region", &["x", "y"])) + .array_with_default_and_overrides("target[Region]", OVER_WIDE, vec![("x", "1")]); + + for (shape, project) in [ + ("Ast::Scalar", scalar), + ("Ast::ApplyToAll", apply_to_all), + ("Ast::Arrayed element", arrayed_element), + ("Ast::Arrayed default", arrayed_default), + ] { + let _guard = SiteChildrenLimitGuard::new(3); + with_ir(&project, |ir| { + let rejection = ir + .site_width_rejection + .as_ref() + .unwrap_or_else(|| panic!("{shape}: must be refused at limit 3")); + assert_eq!(rejection.variable, "target", "{shape}"); + assert_eq!(rejection.axis, SiteWidthAxis::BuiltinContents, "{shape}"); + assert_eq!(rejection.count, 4, "{shape}"); + }); + } + } + + /// An over-wide equation that references NO model variable must still be + /// refused, and the refusal must name the canonically-FIRST offender. + /// + /// Both halves guard the driver rather than the checker. The walk skips a + /// target with neither reference sites nor occurrences, so capturing the + /// verdict after that skip would silently admit a model whose widest + /// equation happens to be constant-only; and the first-offender pick is what + /// makes the emitted diagnostic reproducible across processes, which a + /// `HashMap`-ordered scan would not be. + #[test] + fn a_reference_free_over_wide_equation_is_refused_and_names_the_first_offender() { + let _guard = SiteChildrenLimitGuard::new(2); + let project = TestProject::new("main") + // Canonically after `avg_early`, and also over-wide. + .scalar_aux("zz_avg_late", "MEAN(1, 2, 3)") + .scalar_aux("avg_early", "MEAN(4, 5, 6)"); + with_ir(&project, |ir| { + let rejection = ir + .site_width_rejection + .as_ref() + .expect("a constant-only MEAN(1, 2, 3) is still an over-wide equation"); + assert_eq!( + rejection.variable, "avg_early", + "the canonically-first offender must be reported" + ); + assert_eq!(rejection.axis, SiteWidthAxis::BuiltinContents); + }); + } + + /// The production limit is the whole `u16` range: with no value reserved as + /// a sentinel, child `u16::MAX` is an ordinary, addressable child. + #[test] + fn the_production_limit_is_the_whole_u16_range() { + assert_eq!(MAX_SITE_CHILDREN, u16::MAX as usize + 1); + assert_eq!(u16::try_from(MAX_SITE_CHILDREN - 1), Ok(u16::MAX)); + } + + /// A refusal must reach the user AND stop LTM generation for the model: the + /// two halves of "refuse LTM for this model with a diagnostic". Asserted in + /// both directions on one fixture, so neither half can pass vacuously. + #[test] + fn refusal_emits_no_ltm_vars_and_warns_through_the_diagnostic_surface() { + use crate::db::{ + DiagnosticError, DiagnosticSeverity, collect_model_diagnostics, model_ltm_variables, + }; + use salsa::Setter; + + // A stock/flow feedback loop whose flow equation is a 3-argument `MEAN`: + // scoreable at the production limit, over-wide at limit 2. + let fixture = || { + TestProject::new("main") + .stock("level", "100", &["inflow"], &[], None) + .flow("inflow", "MEAN(level, drain, boost)", None) + .scalar_aux("drain", "1") + .scalar_aux("boost", "2") + }; + + let width_warnings = |db: &SimlinDb, model, project| -> Vec { + collect_model_diagnostics(db, model, project) + .into_iter() + .filter(|d| d.severity == DiagnosticSeverity::Warning) + .filter_map(|d| match d.error { + DiagnosticError::Assembly(msg) + if msg.contains("LTM analysis was skipped for this model") => + { + Some(msg) + } + _ => None, + }) + .collect() + }; + + // Control: at the production limit the model IS scored and no width + // warning is emitted. + { + let mut db = SimlinDb::default(); + let (project, model) = { + let sync = sync_from_datamodel(&db, &fixture().build_datamodel()); + (sync.project, sync.models["main"].source) + }; + project.set_ltm_enabled(&mut db).to(true); + assert!( + !model_ltm_variables(&db, model, project).vars.is_empty(), + "the control fixture must be scoreable, or the refusal below proves nothing" + ); + assert!( + width_warnings(&db, model, project).is_empty(), + "no width warning at the production limit" + ); + } + + // Refusal: no LTM variable at all, plus a Warning naming the variable. + { + let _guard = SiteChildrenLimitGuard::new(2); + let mut db = SimlinDb::default(); + let (project, model) = { + let sync = sync_from_datamodel(&db, &fixture().build_datamodel()); + (sync.project, sync.models["main"].source) + }; + project.set_ltm_enabled(&mut db).to(true); + assert!( + model_ltm_variables(&db, model, project).vars.is_empty(), + "a refused model must emit no link, loop, or pathway score" + ); + let warnings = width_warnings(&db, model, project); + assert_eq!( + warnings.len(), + 1, + "exactly one width refusal must reach collect_model_diagnostics; got {warnings:?}" + ); + assert!( + warnings[0].contains("inflow"), + "the warning must name the offending variable: {}", + warnings[0] + ); + // ...and the noun phrase must be the FIRED axis's, not another + // arm's. Spelled literally rather than as `describe()` so a swap of + // two arms' strings cannot move both sides of the comparison. + assert!( + warnings[0].contains("arguments to one builtin call"), + "the warning must describe the axis that actually fired: {}", + warnings[0] + ); + } + } + + /// `SiteWidthAxis::describe()` is a three-arm decision whose strings are the + /// user-facing noun phrase in the refusal `Warning`, so it gets one row per + /// arm with the text spelled literally. Nothing else in the suite would + /// notice two arms being swapped: the end-to-end assertion above pins the + /// arm-to-message wiring, and these pin the strings themselves. + #[test] + fn every_axis_describes_itself() { + let rows = [ + ( + SiteWidthAxis::ArrayedSlots, + "equation slots (per-element equations plus a default)", + ), + ( + SiteWidthAxis::BuiltinContents, + "arguments to one builtin call", + ), + (SiteWidthAxis::SubscriptIndices, "indices in one subscript"), + ]; + for (axis, expected) in rows { + assert_eq!(axis.describe(), expected, "{axis:?}"); + } + // Distinct, so a message cannot describe two axes the same way. + let described: std::collections::HashSet<&str> = + rows.iter().map(|(a, _)| a.describe()).collect(); + assert_eq!(described.len(), rows.len()); + } + + /// A refusal costs the SCORES, not the STRUCTURE. + /// + /// This is the batch's central claim about blast radius, and it is what makes + /// a whole-model refusal an acceptable trade: the per-edge `sites` view is + /// name-keyed and path-free, so it is untouched by a width refusal, and the + /// two surfaces that read it -- `model_edge_shapes` and the loop enumeration + /// behind `model_detected_loops` -- keep answering exactly as before. If a + /// later change routes either surface through the occurrence stream, or makes + /// the refusal empty `sites`, this reds and the trade has to be re-argued. + /// + /// The fixture is ARRAYED with literal-element reads on purpose. Both + /// consumers default a MISSING per-edge entry to a single `Bare` site, so on + /// a scalar model "the view survived" and "the view was emptied and + /// defaulted" are indistinguishable and the test would pass under a + /// `sites.clear()` -- verified: an earlier scalar version of this test did. + /// `FixedIndex` shapes are what make the difference observable. + #[test] + fn a_refusal_leaves_edge_shapes_and_detected_loops_unchanged() { + use crate::db::{model_detected_loops, model_edge_shapes}; + + // `level[Region]` -> `growth[Region]` -> `level[Region]`: a feedback loop + // whose per-element flow equations read the stock by LITERAL element, so + // the edge carries `FixedIndex` sites rather than `Bare`. Three element + // equations need four slots, so it is refused at limit 3. + let fixture = || { + TestProject::new("main") + .named_dimension("Region", &["a", "b", "c"]) + .array_stock("level[Region]", "100", &["growth"], &[], None) + .array_flow_with_ranges( + "growth[Region]", + vec![ + ("a", "level[a] * 0.1"), + ("b", "level[b] * 0.1"), + ("c", "level[c] * 0.1"), + ], + ) + }; + let structure = || { + let db = SimlinDb::default(); + let sync = sync_from_datamodel(&db, &fixture().build_datamodel()); + let (model, project) = (sync.models["main"].source, sync.project); + let loops: Vec<(String, Vec)> = model_detected_loops(&db, model, project) + .loops + .iter() + .map(|l| (l.id.clone(), l.variables.clone())) + .collect(); + (loops, model_edge_shapes(&db, model, project).clone()) + }; + + let (loops_ok, shapes_ok) = structure(); + assert!( + !loops_ok.is_empty(), + "the fixture must detect a loop, or the comparison below is vacuous" ); - acc.push_occurrence( - super::OccurrenceRef::Variable("pop".to_string()), - RefShape::FixedIndex(vec!["nyc".to_string()]), - Vec::new(), - &[], - false, + // The guard that makes the comparison able to fail: at least one edge + // must carry a NON-`Bare` shape, since `Bare` is exactly what a cleared + // view degrades to. + assert!( + shapes_ok + .edge_shapes + .values() + .any(|s| s.iter().any(|shape| !matches!(shape, RefShape::Bare))), + "the fixture must carry a non-Bare edge shape: {:?}", + shapes_ok.edge_shapes ); - } - // The occurrence is dropped (it could not be addressed)... - assert!( - occurrences.is_empty(), - "an unaddressable child must record no occurrence" - ); - // ...but the edge site survives, WITH its real shape. - let pop_sites = sites.get("pop").expect( - "the per-edge reference site must survive suppression -- a missing IR \ - entry is defaulted to `Bare` by consumers, misclassifying the reference", - ); - assert_eq!(pop_sites.len(), 1); - assert_eq!( - pop_sites[0].shape, - RefShape::FixedIndex(vec!["nyc".to_string()]), - "the edge must keep its real shape, not degrade to Bare" - ); + let (loops_refused, shapes_refused) = { + let _guard = SiteChildrenLimitGuard::new(3); + // Confirm the refusal really fires for this fixture at this limit. + with_ir(&fixture(), |ir| { + assert!(ir.site_width_rejection.is_some(), "refusal must fire"); + }); + structure() + }; + + assert_eq!( + loops_refused, loops_ok, + "loop DETECTION reads the path-free per-edge view and must be unaffected" + ); + assert_eq!( + shapes_refused, shapes_ok, + "`model_edge_shapes` reads the same view and must be byte-equal" + ); + } } diff --git a/src/simlin-engine/src/ltm_augment.rs b/src/simlin-engine/src/ltm_augment.rs index ee8777715..0f68015d9 100644 --- a/src/simlin-engine/src/ltm_augment.rs +++ b/src/simlin-engine/src/ltm_augment.rs @@ -252,31 +252,34 @@ struct WrapCtx<'a> { /// `classifier_agreement_tests::assert_occurrence_stream_aligns` proves /// corpus-wide. Cloning per descent is cheap: LTM equations are short. /// -/// `i` is a `usize` and the conversion is CHECKED, mapping an over-arity child -/// (a variadic builtin with 65,536+ arguments) to -/// [`db::ltm_ir::UNADDRESSABLE_CHILD`] rather than letting `as u16` wrap. That -/// matters because wrapping produced an EARLIER sibling's path: the lookup would -/// return an unrelated occurrence, `missing_occurrence` would never fire, and -/// the wrap would freeze the wrong reference and emit a plausible wrong score. -/// The sentinel is a value `site_child_index` never emits, so every lookup at or -/// below such a child provably MISSES -- which turns the case into the existing -/// loud skip-and-warn. Callers additionally flag it directly (see the `App` arms) -/// rather than relying only on the downstream miss. +/// `i` fits a `u16` by the LTM front door, not by luck: `model_ltm_reference_sites` +/// refuses a target equation needing more than +/// [`MAX_SITE_CHILDREN`](crate::db::ltm_ir::MAX_SITE_CHILDREN) children at one +/// level, and `model_ltm_variables` then emits no LTM variable for that model at +/// all -- so the wrap never runs on an equation whose child indices could +/// overflow. **The front door is the whole of the soundness argument here.** +/// +/// The conversion saturates rather than wrapping, which is strictly better -- +/// wrapping maps child 65,536 onto child 0, so it can alias an ARBITRARY earlier +/// sibling. But saturating is not a safety net: this change deleted the reserved +/// unaddressable-child sentinel that used to hold `u16::MAX` back, so `u16::MAX` +/// is now an ordinary, addressable child index (pinned by `db::ltm_ir::ltm_ir_tests`' +/// `the_production_limit_is_the_whole_u16_range`). A violated precondition would +/// therefore land on sibling 65,535's real recorded `SiteId` -- exactly the alias +/// the sentinel used to make impossible. Do not read the saturation as +/// protection; read it as "the failure mode is one specific collision instead of +/// an arbitrary one, and the front door is what keeps it unreachable". +/// +/// It is not a `panic`/`expect` because release builds use `panic = abort`: +/// aborting the host process is a worse answer than an unreachable collision on +/// a model for which no link score is generated at all. fn child_path(path: &[u16], i: usize) -> Vec { let mut v = Vec::with_capacity(path.len() + 1); v.extend_from_slice(path); - v.push(u16::try_from(i).unwrap_or(crate::db::ltm_ir::UNADDRESSABLE_CHILD)); + v.push(u16::try_from(i).unwrap_or(u16::MAX)); v } -/// Whether builtin child index `i` can be addressed by a `SiteId` element. The -/// wrap's `App` arms set [`WrapOutcome::missing_occurrence`] when this is false, -/// so the partial is abandoned LOUDLY at the overflow point instead of depending -/// on a lookup miss further down. -fn child_is_addressable(i: usize) -> bool { - crate::db::ltm_ir::site_child_index(i).is_some() -} - /// The `Collapse` / `Mismatch` / `NotIterated` verdict for an iterated-dimension /// subscript on a NON-live-source dependency, derived from the occurrence IR's /// per-axis classification (`node_occ.axes`) plus the two arity facts the @@ -726,9 +729,6 @@ fn wrap_non_matching_in_previous( None => a, } } else { - if !child_is_addressable(i) { - out.missing_occurrence = true; - } wrap_non_matching_in_previous(a, ctx, out, &child_path(path, i), frozen) } }) @@ -791,12 +791,6 @@ fn wrap_non_matching_in_previous( .into_iter() .enumerate() .map(|(i, a)| { - // An over-arity child cannot be addressed by a `SiteId`, so - // every occurrence decision inside it is unavailable: abandon - // the partial loudly rather than wrap on a guessed path. - if !child_is_addressable(i) { - out.missing_occurrence = true; - } wrap_non_matching_in_previous(a, ctx, out, &child_path(path, i), frozen) }) .collect(); @@ -3377,7 +3371,10 @@ fn build_arrayed_link_score_equation( // `slot` is the per-element occurrence-stream slot: `walk_all_in_expr` // numbers `Ast::Arrayed` slots in canonical element-key-sorted order (then // the default after the last element), so the wrap for element `slot` - // consumes exactly that slot's occurrences. + // consumes exactly that slot's occurrences. Both `slot as u16` conversions + // below are in range by the LTM front door, which refuses a target needing + // more slots than `db::ltm_ir::MAX_SITE_CHILDREN` can tell apart -- so this + // model would have emitted no link score to reach here. let slot_equation = |expr: &crate::ast::Expr2, gf_table_ref: Option<&str>, slot: u16| diff --git a/src/simlin-engine/src/ltm_augment_tests.rs b/src/simlin-engine/src/ltm_augment_tests.rs index 2b286d1c2..72599f0e4 100644 --- a/src/simlin-engine/src/ltm_augment_tests.rs +++ b/src/simlin-engine/src/ltm_augment_tests.rs @@ -5428,46 +5428,37 @@ fn slot_occurrence_index_groups_every_slot() { assert!(index.for_slot(99).is_empty()); } -/// `child_path` must map an over-arity builtin child to the reserved -/// unaddressable sentinel rather than letting `as u16` WRAP onto an earlier -/// sibling's path. +/// `child_path` must append each child's own index over the WHOLE range a +/// `SiteId` component addresses, and must never WRAP onto an earlier sibling's +/// path past it. /// -/// This is the consumer half of the F3 fix, and it needs its own pin: the -/// producer-side boundary tests (`db::ltm_ir_tests`) pass whether or not this -/// conversion is checked, because they only exercise the pure index function. -/// With a wrapping cast, child 65,536 produces child 0's path, so the lookup -/// returns an UNRELATED occurrence, `missing_occurrence` never fires, and the -/// wrap freezes the wrong reference -- a plausible, silent, wrong score. -#[test] -fn child_path_maps_over_arity_child_to_the_unaddressable_sentinel() { - use crate::db::ltm_ir::UNADDRESSABLE_CHILD; +/// The consumer half of the front-door contract (GH #978/#979). Every index the +/// LTM front door admits (`0 ..= MAX_SITE_CHILDREN - 1`) must round-trip +/// verbatim, because the wrap looks occurrences up by exactly this path; and the +/// out-of-range conversion must saturate rather than wrap, since `65_536 as u16` +/// is 0 -- child 0's path -- which would make the lookup return an UNRELATED +/// occurrence, never fire `missing_occurrence`, and freeze the wrong reference. +/// (Out of range is unreachable in production: the front door refuses such a +/// model and no link score is generated for it.) +#[test] +fn child_path_appends_every_addressable_index_and_never_wraps() { + use crate::db::ltm_ir::MAX_SITE_CHILDREN; // Ordinary children append their own index. assert_eq!(child_path(&[3], 0), vec![3, 0]); assert_eq!(child_path(&[3], 7), vec![3, 7]); + // The whole addressable range round-trips, boundary included. `u16::MAX` is + // an ordinary child index now that no value is reserved as a sentinel. assert_eq!(child_path(&[], 65_534), vec![65_534]); - - // Over-arity children get the sentinel, NOT a wrapped index. `65_536 as u16` - // is 0, which is precisely the collision this must avoid. - assert_eq!(child_path(&[3], 65_536), vec![3, UNADDRESSABLE_CHILD]); - assert_ne!(child_path(&[3], 65_536), child_path(&[3], 0)); - assert_eq!(child_path(&[3], 131_072), vec![3, UNADDRESSABLE_CHILD]); - assert_ne!(child_path(&[3], 131_072), child_path(&[3], 0)); - - // `child_is_addressable` agrees with the producer's own predicate at the - // boundary, so the wrap flags exactly the children the IR omits. - assert!(child_is_addressable(0)); - assert!(child_is_addressable(65_534)); - assert!(!child_is_addressable(65_535)); - assert!(!child_is_addressable(65_536)); - - // A path containing the sentinel can never be produced by the IR walker, so - // a lookup under it provably misses rather than aliasing. assert_eq!( - crate::db::ltm_ir::site_child_index(65_536), - None, - "the producer must decline the same child the consumer sentinels" + child_path(&[3], MAX_SITE_CHILDREN - 1), + vec![3, u16::MAX], + "the last child the front door admits must address itself" ); + + // Past the range: saturate, never wrap onto an earlier sibling. + assert_ne!(child_path(&[3], MAX_SITE_CHILDREN), child_path(&[3], 0)); + assert_ne!(child_path(&[3], 131_072), child_path(&[3], 0)); } #[cfg(test)] From 79185fb7380de62929a90a9f86f2dba4844f7bf5 Mon Sep 17 00:00:00 2001 From: Bobby Powers Date: Mon, 27 Jul 2026 16:29:13 -0700 Subject: [PATCH 05/17] engine: carry the reducer classification on the agg node emit_source_to_agg_link_scores and source_to_agg_hop_polarity recovered the reducer kind, name, body and polarity by printing an AggNode's reducer subexpression, re-parsing it, re-lowering it against a freshly built scope and re-classifying the result -- a closed round trip through our own printer and parser, run once per (agg, source) pair and once per agg-touching loop link. Each failed silently in its own way: a failed reconstruction made the emitter skip the edge's link scores entirely, zeroing the synthetic agg's loop score, and made the polarity hop return Unknown, degrading every loop through the agg to Undetermined. AggNode now carries the BuiltinFn the enumerator classified when it decided the hoist, and reconstruct_ltm_var_lowered is deleted along with its last helper; nothing on the emission path parses. The equation-shape reconstruction both call sites documented as a hazard (an arrayed agg had to be rebuilt as an ApplyToAll over result_dims) is removed rather than moved: there is no shape to rebuild. Carrying an Expr2 puts an f64 inside a salsa-cached, PartialEq-compared value, which has three consequences and the stored form only addresses two. Stripping Loc is load-bearing: raw, an edit that only moves an equation's byte offsets stopped enumerate_agg_nodes backdating, which a test now pins. Stripping ArrayBounds is a guard, inert today because the ASTs this query walks are lowered against an empty model scope and carry no bounds at all. The f64 itself cannot be normalized away: a derived PartialEq is not reflexive on NaN, so a model whose hoisted reducer holds a nan literal never backdates this query -- an incrementality regression this commit introduces, disclosed on the field, in CLAUDE.md, and in a characterization test asserting today's broken behaviour. That is the GH #987/#981 class, fixed at the root by making AST float literals compare by bit pattern, and that fix must invert the test rather than delete it. The same f64 is why AggNode loses Eq and its unconditional Debug. The carried builtin is read only for synthetic aggs; the variable-backed arm's copy is unread today and is kept so AggNode has one shape. The same change settles GH #982, which read reducer_collapses_to_scalar and builtin_routes_through_agg as one question answered twice, inverted on SIZE and RANK. They are two questions: the first is about the reducer's result type (does the subtree fit in a scalar slot -- SIZE does, RANK does not) and gates the freeze/capture paths; the second is about LTM routing (did enumerate_agg_nodes mint a node -- SIZE is Constant and never is, RANK gets an array-valued agg) and sets OccurrenceSite::in_reducer. Both already derive from the single reducer_kind_from_name table, so there was no second table to collapse; what was missing was the argument and a test. Mutating each set in each direction showed three of the four cells already pinned and one not: the SIZE membership of the GH #779 decline, whose inertness rests on the freezability gate reading the same predicate -- move one and it becomes reachable with nothing noticing. A shared cfg(test) decision table now covers every arm of reducer_kind_from_name against all three derived predicates, including the agreement gate's own name-keyed restatement of the routing rule, whose drift would have quietly weakened that gate rather than failing it. RANK is deliberately not flipped: that would turn a scored bare source into a loud decline, a user-visible change with no evidence behind it. --- src/simlin-engine/CLAUDE.md | 4 +- src/simlin-engine/src/ast/expr2.rs | 71 ++ src/simlin-engine/src/builtins.rs | 59 ++ src/simlin-engine/src/db/ltm/link_scores.rs | 38 +- src/simlin-engine/src/db/ltm/loops.rs | 31 +- src/simlin-engine/src/db/ltm/parse.rs | 64 +- src/simlin-engine/src/db/ltm_ir.rs | 21 +- src/simlin-engine/src/ltm/graph.rs | 14 +- src/simlin-engine/src/ltm/polarity.rs | 196 +++--- src/simlin-engine/src/ltm/tests.rs | 28 +- src/simlin-engine/src/ltm_agg.rs | 608 +++++++++++++++--- src/simlin-engine/src/ltm_augment.rs | 79 ++- src/simlin-engine/src/ltm_augment_tests.rs | 17 + .../src/ltm_classifier_agreement_tests.rs | 22 + 14 files changed, 938 insertions(+), 314 deletions(-) diff --git a/src/simlin-engine/CLAUDE.md b/src/simlin-engine/CLAUDE.md index 3bbb9655e..a8213d09c 100644 --- a/src/simlin-engine/CLAUDE.md +++ b/src/simlin-engine/CLAUDE.md @@ -125,7 +125,7 @@ The primary compilation path uses salsa tracked functions for fine-grained incre - **`src/db/ltm/`** - LTM (Loops That Matter) variable generation as salsa tracked functions, split into a directory (one large cohesive concern with heavy internal coupling; a directory keeps the internal helpers `pub(super)`/`pub(crate)` *within* the `ltm` subtree without widening to crate scope, and the names that escape -- the `db.rs`-root `pub use ltm::{...}` / `use ltm::*` surface and the `ltm::` paths `src/db/assemble.rs` uses -- are re-exported at `src/db/ltm/mod.rs`): - **`mod.rs`** - the orchestration root: `model_ltm_variables` (the unified entry point), the entry/glue (`ltm_module_idents`, the submodule `use`/`pub use` wiring, the `#[cfg(test)] #[path]` test mount of the sibling `src/db/ltm_tests.rs` -- which stays one level up from the directory, so the relative mount path is one segment up), and `LtmImplicitVarMeta`/`model_ltm_implicit_var_info`. - **`equation.rs`** - `LtmEquation` (+ `LtmArm`), the typed, authoritative representation of an `LtmSyntheticVar`'s equation (GH #965 storage/compile boundary). Mirrors the three `datamodel::Equation` shapes but carries a parsed `Expr0` per arm (plus the generator's exact source-form `text`, kept for the char-dump/diagnostics only), so the fragment compiler and the layout implicit-var scan consume the AST directly -- normal operation never prints an equation to `datamodel::Equation` text and re-parses it back to compile (the former transient-`Aux` round trip, 2-3 parses per equation, GH #655 finding 3). The `Expr0` is `Expr0::new(text)` built once at generation, so this is behavior-neutral: the compiled AST is exactly what the old text parse produced (the char goldens stay byte-identical). Owns the shaping helpers `dimensions`/`scalarize`/`retarget_dims` (they re-tag dimensions and select arms, never rewrite an arm's AST) and `to_flow_ast` (resolves the dimension NAMES to `Dimension`s and yields the `Ast` for lowering). - - **`parse.rs`** - LTM equation parsing (`parse_ltm_equation` and friends, `reconstruct_ltm_var_lowered`): builds the flow-phase `Ast` from the stored `LtmEquation` (no text reparse) and runs the SAME `instantiate_implicit_modules` visitor the ordinary variable parse runs, so PREVIOUS/INIT capture auxes and stdlib module calls expand identically; flow-phase only (LTM vars are flow-phase scalar auxes; the init phase would only re-synthesize same-named helpers that dedup away). Plus the thin free-function wrappers (`ltm_equation_dimensions`, `scalarize_ltm_equation`, `retarget_ltm_equation_dims`) over the `LtmEquation` methods so the call sites keep their spelling. + - **`parse.rs`** - LTM equation parsing (`parse_ltm_equation`): builds the flow-phase `Ast` from the stored `LtmEquation` (no text reparse) and runs the SAME `instantiate_implicit_modules` visitor the ordinary variable parse runs, so PREVIOUS/INIT capture auxes and stdlib module calls expand identically; flow-phase only (LTM vars are flow-phase scalar auxes; the init phase would only re-synthesize same-named helpers that dedup away). Plus the thin free-function wrappers (`ltm_equation_dimensions`, `scalarize_ltm_equation`, `retarget_ltm_equation_dims`) over the `LtmEquation` methods so the call sites keep their spelling. It has exactly ONE caller family -- `compile.rs`, compiling a synthetic variable it is about to emit bytecode for. The `reconstruct_ltm_var_lowered` helper is gone (GH #983): its two callers were *emitters* recovering a fact the enumerator already held, by printing an `AggNode`'s reducer subexpression, re-parsing it, re-lowering it against a freshly built scope, and re-classifying the result. Both now read `ltm_agg::AggNode::reducer`, so nothing on the emission path parses at all, and the "arrayed aggs must be reconstructed as `ApplyToAll` over `result_dims`" hazard both call sites documented no longer exists -- there is no equation shape to rebuild. - **`compile.rs`** - per-equation compilation: `compile_ltm_var_fragment`, `link_score_equation_text_shaped`, the shared `compile_ltm_equation_fragment` core, `compile_ltm_synthetic_fragment`, `model_ltm_fragment_diagnostics`, `compile_ltm_implicit_var_fragment`. - **`loops.rs`** - loop assembly: output-port discovery (`find_model_output_ports`), the read-slice / cartesian-subscript helpers (`cartesian_subscripts`, `read_slice_rows`), the tiered loop builders (`build_loops_from_tiered`, `build_element_level_loops`), and the cross-element-through-aggregate recovery cluster (`recover_cross_agg_loops`, `recover_agg_hop_polarities`, `stitch_cross_agg_petals`, `MAX_CROSS_AGG_LOOPS`, `AggLoopBudgetGuard`). - **`link_scores.rs`** - the per-edge / per-shape / through-aggregate link-score emitters (`link_score_dimensions`, `try_cross_dimensional_link_scores`, `try_scalar_to_arrayed_link_scores`, `try_disjoint_dim_arrayed_link_scores`, `emit_per_shape_link_scores`, `emit_source_to_agg_link_scores`, `emit_agg_to_target_link_scores`, `emit_link_scores_for_edge`, `emit_unscoreable_disjoint_edge_warning`), lifted verbatim out of `model_ltm_variables`'s body (they were capture-nothing top-level `fn`s nested in the function). @@ -181,7 +181,7 @@ The unit subsystem is partial-result throughout: a single bad declaration or one - `graph.rs` - public `CausalGraph` type owning model adjacency, stock identity, an optional variable AST map for polarity, and optional recursively-built sub-graphs for referenced sub-models. `CausalGraph::from_model` populates both for **every** referenced sub-model (DynamicModule and stockless passthrough alike, since GH #698 -- the passthrough's sub-graph drives the discovery-mode per-exit-port pathway recompute; a pathless module's sub-graph enumerates no pathways and is harmless). The lightweight edge-only `db::analysis` constructors (`causal_graph_from_edges` / `causal_graph_from_element_edges`) leave both EMPTY; the production discovery path enriches the element-level graph via `causal_graph_from_element_edges_with_modules` (sharing `model_variables_and_module_graphs` with `causal_graph_with_modules`). The `variables()` / `module_graph()` accessors expose the variable map and a module instance's sub-graph for that recompute. Drives `find_loops_with_limit` / `compute_cycle_partitions` / `all_link_polarities` / `enumerate_pathways_to_outputs`. `Loop` struct carries `id`, `links`, `stocks`, `polarity`, and `dimensions` (non-empty for A2A loops where per-element evaluation is needed; empty for scalar or mixed loops). - `tests.rs` - integration-style tests spanning all submodules; preserved as a single file to avoid splitting shared test fixtures. - **`src/ltm_finding.rs`** - Strongest-path loop discovery algorithm (Eberlein & Schoenberg 2020). Post-processes simulation results containing link score synthetic variables to find the most important loops at each timestep. `discover_loops_with_graph(results, causal_graph, stocks, ltm_vars, dims, expansion, sub_model_output_ports, budget)` is the primary entry point: `ltm_vars` and `dims` enable A2A link score expansion (per-element edges from `parse_link_offsets`); when empty, all link scores are treated as scalar. A Bare A2A link score is dimensioned over the TARGET's dims, so `expand_a2a_link_offsets` subscripts the TARGET node per element but PROJECTS the SOURCE node onto `from`'s OWN declared dims -- bare for a scalar feeder (`scale → growth`, GH #790), the same-element diagonal for an equal-dim feeder, the broadcast/partial-collapse form for a lower-dim feeder, and the positionally-mapped diagonal for a `State→Region` pair (GH #527) -- by reusing the element graph's own `db::expand_same_element` rule, so the discovery search graph's node names match `model_element_causal_edges` node-for-node (GH #754; before this, subscripting BOTH endpoints with the score's full tuple minted phantom from-nodes like `scale[a]`/`boost[r,a]`/`x[s]` that named no real node, so every loop through such a feeder dangled and was silently undiscoverable). The from-var declared dims + dimension-mapping context ride in a `LinkExpansionContext` (`declared_dims` + `dim_ctx`) that `analyze_model` builds via the public `analysis::build_link_expansion_context` (the SAME `variable_dimensions` / `project_dimensions_context` queries the element graph reads); the db-less `discover_loops(&Results, &Project)` convenience path passes `LinkExpansionContext::default()` (no A2A expansion runs there). Element-mapped (non-positional) pairs never reach the projection: `db::ltm::link_score_dimensions` declines to retarget them (no Bare A2A score; the GH #758 loud skip fires instead) under the GH #756 positional-only gate. `SearchGraph` provides DFS-based strongest-path traversal from stock nodes. The post-simulation score recompute (path → `FoundLoop`) applies the **per-exit-port pathway selection** (`recompute_module_input_edge_series`, GH #698): a loop edge `x → m` into a module is recomputed by max-abs-selecting over the sub-model's `m·$⁚ltm⁚path⁚{entry}⁚{idx}` pathway scores that end at the exit port the loop reads (recovered from the next link `m → y`), instead of reading the module *composite* (which max-abs-selects across ALL ports and so can pick a wrong-signed port for a multi-output module -- flipping the loop polarity vs exhaustive). The pathway indices come from the module's sub-graph via the same `enumerate_pathways_to_outputs_with_truncation` over the same sorted output-port set the sub-model emitted against, so they match index-for-index (the discovery-mode equivalent of exhaustive's `compute_module_link_overrides`). That port set is NOT re-derived parent-scoped (which would shift indices when another project model reads an extra output port -- GH #698 / PR #705 r3353097150): `discover_loops_with_graph` takes a `SubModelOutputPorts` map (sub-model canonical name -> emitted sorted port set) that `analyze_model` builds from the SAME emission decision via `analysis::build_sub_model_output_ports` -> `db::ltm::sub_model_output_ports` (identity-by-construction); the db-less `discover_loops(&Results, &Project)` convenience path reconstructs the same project-wide-union + stdlib-`output` semantics in `project_sub_model_output_ports`. Falls back to the base composite offset for single-output / pathless / indeterminate-port / sub-model-absent-from-map edges. Because the discovery graph is element-level, the recompute `strip_subscript`s `link.from`/`link.to`/`next.from`/`next.to` before its name matches (arrayed loop nodes carry `[elem]`; mirrors the exhaustive twin's stripping -- PR #705 r3353758167); this is currently latent parity defense, since an arrayed loop through a multi-output module is not yet discoverable end-to-end (a scalar module output feeding an arrayed reader scores a constant-0 link, dropping the loop -- GH #716). Returns a `DiscoveryResult` whose `FoundLoop`s carry per-timestep link/loop/pathway scores, ranked **competitive-first** (`rank_and_filter`): loops sharing their cycle partition with at least one other discovered loop come first, ordered by mean partition-relative importance; loops trivially ALONE in their partition (relative score exactly +/-1 by construction -- zero information, e.g. C-LEARN's isolated gas-uptake decay loops) sort after all competing loops and are dropped first under the `MAX_LOOPS` cap. Each `FoundLoop` carries a result-scoped dense `partition` index into `DiscoveryResult::partitions` (`DiscoveredPartition { stocks, loop_count }`, first-appearance order; NOT stable across runs/edits -- key on the stock set for durable identity), threaded through `analysis::ModelAnalysis::partitions` / `LoopSummary::partition`, the FFI `SimlinDiscoveredPartition` / `SimlinDiscoveredLoop.partition`, and pysimlin `Analysis.partitions` / `Loop.partition`. Also hosts the link-set synthetic-node collapse: `trim_synthetic_aggs_from_loop_links` collapses `$⁚ltm⁚agg⁚{n}` nodes out of a single loop's link *cycle*, and the public `collapse_synthetic_links(Vec)` generalizes that to ALL synthetic/macro/module-internal nodes (`ltm::is_synthetic_node_name`) over an arbitrary link *set* -- each chain `X -> $⁚…internal… -> Y` collapses to one composite edge `X -> Y` with product polarity and the per-timestep max-magnitude path score (the composite link score, LTM ref 6.3/6.4); a purely-internal cycle/source is dropped. `CollapsibleLink { from, to, polarity, score: Option> }` is the abstract shape, so structural-only callers (`score = None`) and LTM-run callers share one impl. libsimlin's `simlin_analyze_get_links(.., include_internal=false)` collapses the `get_links()` set through this; `include_internal=true` returns the raw graph. The `#[cfg(test)]` tests live in the sibling `src/ltm_finding_tests.rs` (mounted via `#[cfg(test)] #[path]`, split out for the per-file line cap). -- **`src/ltm_agg.rs`** - Aggregate-node enumeration for LTM. `enumerate_agg_nodes` (salsa-tracked) walks every variable's `Expr2` AST left-to-right depth-first and identifies each maximal array-reducer subexpression (`SUM`/`MEAN`/`MIN`/`MAX`/`STDDEV`/`RANK`/`SIZE`); AST-identical subexpressions (keyed by canonical printed equation text) dedupe to one `AggNode`. Two kinds: **synthetic** (`is_synthetic == true`, the reducer is a sub-expression of a larger equation -- a `$⁚ltm⁚agg⁚{n}` aux is minted) and **variable-backed** (`is_synthetic == false`, the reducer is the entire dt-equation of a scalar/A2A variable like `total_population = SUM(pop[*])` -- the variable itself is the agg; EXCEPT a whole-RHS reducer whose shape the variable-backed machinery cannot express -- a MAPPED iterated axis (GH #534: the name-based link-score path cannot remap, so its `Wildcard` partial would silently stub to 0) or NON-ALIGNED result dims (GH #764 / shape-expressiveness T4: a broadcast over extra owner dims `out[D1,D3] = SUM(matrix[D1,*])` or permuted axes `out[D2,D1] = SUM(cube[D1,D2,*])`, where a per-`(row, slot)` slot cannot name a complete owner element) -- which mints a synthetic agg instead (`variable_backed_shape_is_expressible`, the ONE minting condition), riding the two-half emitters + the GH #528 projection). Each `AggNode` carries a `sources: Vec` -- one entry per source variable, SORTED by canonical name and deduped (salsa cache-equality + emission-order determinism; T2 of the shape-expressiveness design), each with its OWN `read_slice: Vec` (one `AxisRead ∈ {Pinned(elem), Iterated{dim, source_dim}, Reduced{subset}}` per THAT source's axes -- which rows of it the reducer actually reads; a scalar feeder like `scale` in `SUM(pop[*] * scale)` carries an empty slice; `Iterated` carries the (target, source) canonical dim pair, equal for the literal case; `Reduced.subset` is `None` for the full extent or the proper-subdimension element subset for a `SUM(arr[*:Sub])` StarRange, GH #766, decided per axis by `classify_axis_access` -- the single per-axis classifier of the shape-expressiveness design) -- and a `result_dims` (the `Iterated` axes' TARGET dims, datamodel-cased -- empty for a whole-extent or pinned-slice reduce). Under the I1 acceptance (`accept_source_slices`, T5 of the shape-expressiveness design / GH #767) every arrayed CO-SOURCE (`Reduced`-bearing slice) carries the identical *canonical* slice (`AggNode::canonical_read_slice` -- the first `Reduced`-bearing source slice, falling back to the first non-empty for the degenerate no-co-source agg), while an ITERATED-DIM PROJECTION FEEDER -- a source whose slice is all-`Iterated` over exactly the canonical slice's iterated target dims, in order, unmapped (`frac[D1]` in `SUM(matrix[D1,*] * frac[D1])`, per-result-slot constant) -- is accepted with ITS OWN slice (`AggNode::source_is_projection_feeder` is the discriminator); per-source consumers (`emit_agg_routed_edges`, `emit_source_to_agg_link_scores`) read `AggNode::source_read_slice(from)`, and name-keyed consumers use `AggNode::reads_var`. A feeder's link-score half is the per-`(row, slot)` CHANGED-LAST equation (`ltm_augment::generate_iterated_feeder_to_agg_equation`, emitted via `link_scores::iterated_feeder_row_scores` from both the synthetic source-half and `try_cross_dimensional_link_scores`' variable-backed feeder branch): the reducer text pinned to the slot with only the feeder frozen -- the arrayed generalization of the GH #737 scalar-feeder convention, exactly complementary per slot to the co-source rows' changed-first numerators for a bilinear body; the co-source rows' changed-first body partial pins the mismatched-arity feeder dep BY DIM NAME (`pin_body_to_row`'s GH #767 extension) so `PREVIOUS(frac[d1·r])` is held frozen at the row instead of bailing to the delta-ratio fallback. `compute_read_slice` decides hoistability: a whole-extent reduce (`SUM(pop[*])` ⇒ all-`Reduced`), a sliced one (`SUM(pop[NYC,*])` ⇒ `[Pinned(nyc), Reduced]`, `SUM(matrix[D1,*])` over an A2A-`D1` body ⇒ `[Iterated(d1,d1), Reduced]` → an arrayed agg over `D1`), a mixed one (`SUM(matrix3d[D1,NYC,*])` over an A2A-`D1` body ⇒ `[Iterated(d1,d1), Pinned(nyc), Reduced]`), or a positionally-MAPPED sliced one (GH #534: `SUM(matrix[State,*])` over `matrix[Region,D2]` with a positional `State→Region` mapping ⇒ `[Iterated{state, region}, Reduced]`, `result_dims = [State]` -- the agg is arrayed over the TARGET's iterated dim and the three `Iterated`-axis consumers remap each source row to the slot of its positionally-corresponding target element via `iterated_axis_slot_elements`, the preimage inversion of `mapped_element_correspondence`, so the positional-only/element-map gate is inherited) is hoisted; the carve-outs are (a) a reducer over a *dynamic index* (`SUM(pop[idx,*])`, `idx` non-literal ⇒ not statically describable ⇒ not hoisted, reference stays on the conservative path -- `db::ltm_ir` reclassifies it as `DynamicIndex`), (b) an ELEMENT-mapped sliced reducer the correspondence declines (execution resolves positionally and ignores the map, GH #756; a POSITIONAL mapping is accepted in either declaration direction since GH #757) ⇒ `compute_read_slice` returns `None`, conservative -- (c) a multi-source slice combination outside the I1 acceptance -- co-sources with differing slices, one variable read with two different slices (I3b), or a no-`Reduced` source that is not the pure iterated projection (a Pinned-axis mix like `SUM(matrix[D1,*] * w[D1, c1])`, a dim-subset/permuted feeder, or any mapped Iterated axis in a feeder combination) -- `combined_read_slice`/`accept_source_slices` return `None`, (d) a StarRange naming a NON-subdimension of its axis (a mid-edit inconsistency; declined rather than silently widened to the full extent), and (e) `RANK` (GH #771: array-valued, so `reducer_is_hoistable` requires `reducer_collapses_to_scalar` and RANK references stay `Direct` -- a bare arg classifies `Bare`, scored by the GH #742 arrayed-capture path; loops through the rank ORDERING are a documented residual). A multi-source reducer whose arrayed args *agree* (`SUM(a[*] + b[*])`, `a`, `b` over the same dim) mints one agg with one `AggSource` per variable, each carrying the shared canonical slice. `AggNodesResult` exposes `aggs` (first-encounter order), `agg_for_key` (by canonical reducer text), and `aggs_in_var` (which aggs occur in a variable's equation, so the element-graph reroute can ask "which agg of `to` reads `from`?"). A variable-backed reduce with a NON-TRIVIAL statically-describable slice is gated by `variable_backed_reduce_agg` (GH #752, generalized by GH #765 / T3 of the shape-expressiveness design) -- the SAME gate `model_element_causal_edges`' dispatch, `build_element_level_loops`' per-circuit routing, and `try_cross_dimensional_link_scores`' row derivation consume, so edges, loop routing, and scores always cover the identical read rows (all three derive from `read_slice_rows`, invariant I4). Accepted: an ALIGNED partial reduce (`row_sum[D1] = SUM(matrix[D1,*])`, `result_dims` equal to the variable's own dims in order -- Pinned-mixed `outf[D1] = MEAN(cube[D1,x,*])` and subset `out[D1] = MEAN(matrix[D1,*:Sub])` slices included: the divisor is the true read count and unread rows get neither edges nor scores) gets read-slice element edges straight onto the variable's element nodes and per-circuit scalar loops whose both-subscripted `matrix[d1,d2]→row_sum[d1]` links resolve the per-`(row, slot)` scores; a scalar-result Pinned/subset slice on a SCALAR owner (`total = SUM(pop[nyc,*])`, `total = SUM(arr[*:Sub])`) routes the read rows into the bare `to` node with matching per-read-row scores; and the ARRAYED-owner scalar-result Pinned/subset BROADCAST slice (`share[Region] = SUM(pop[nyc,*])`, no `Iterated` axis -- GH #777) fans each read row across the FULL target element set (`emit_agg_routed_edges`' broadcast arm emits `pop[nyc,d2] → share[e]` for every `e`; `emit_broadcast_reduce_link_scores` emits the matching per-(read-row, full-target-element) scalar scores `pop[nyc,d2]→share[e]`, the section-3 `PerElement` rule applied to a variable-backed reducer owner), with loop circuits routed to the per-circuit scalar path via `is_broadcast_reduce_edge` -- the read rows are independent of `to`'s dims, so the RELATED-dim (`share[Region]`) and DISJOINT-dim (`share[D9]`) spellings emit identically. Declined: a pure full-extent variable-backed agg keeps the normal reference walker's edges (the true reads for that shape, inert skip). The GH #764 broadcast/permuted result shapes never reach this gate since T4 -- they mint synthetic aggs at enumeration -- so the gate's Iterated-arm alignment check is defense-in-depth. `scalar_feeder_of_variable_backed_agg` (GH #790) is the scalar sibling of `source_is_projection_feeder`: it composes `variable_backed_reduce_agg` with an empty-slice + genuine-`Reduced`-canonical check to recognize a SCALAR FEEDER of a whole-RHS variable-backed reduce (`scale` in `growth[D1] = SUM(matrix[D1,*] * scale)`), so `try_scalar_to_arrayed_link_scores` can route it to the single Bare A2A changed-last feeder score instead of the uncompilable per-target-element partials. `is_synthetic_agg_name` / `synthetic_agg_name` are the `$⁚ltm⁚agg⁚{n}` name helpers. +- **`src/ltm_agg.rs`** - Aggregate-node enumeration for LTM. `enumerate_agg_nodes` (salsa-tracked) walks every variable's `Expr2` AST left-to-right depth-first and identifies each maximal array-reducer subexpression (`SUM`/`MEAN`/`MIN`/`MAX`/`STDDEV`/`RANK`/`SIZE`); AST-identical subexpressions (keyed by canonical printed equation text) dedupe to one `AggNode`. Two kinds: **synthetic** (`is_synthetic == true`, the reducer is a sub-expression of a larger equation -- a `$⁚ltm⁚agg⁚{n}` aux is minted) and **variable-backed** (`is_synthetic == false`, the reducer is the entire dt-equation of a scalar/A2A variable like `total_population = SUM(pop[*])` -- the variable itself is the agg; EXCEPT a whole-RHS reducer whose shape the variable-backed machinery cannot express -- a MAPPED iterated axis (GH #534: the name-based link-score path cannot remap, so its `Wildcard` partial would silently stub to 0) or NON-ALIGNED result dims (GH #764 / shape-expressiveness T4: a broadcast over extra owner dims `out[D1,D3] = SUM(matrix[D1,*])` or permuted axes `out[D2,D1] = SUM(cube[D1,D2,*])`, where a per-`(row, slot)` slot cannot name a complete owner element) -- which mints a synthetic agg instead (`variable_backed_shape_is_expressible`, the ONE minting condition), riding the two-half emitters + the GH #528 projection). Each `AggNode` carries a `sources: Vec` -- one entry per source variable, SORTED by canonical name and deduped (salsa cache-equality + emission-order determinism; T2 of the shape-expressiveness design), each with its OWN `read_slice: Vec` (one `AxisRead ∈ {Pinned(elem), Iterated{dim, source_dim}, Reduced{subset}}` per THAT source's axes -- which rows of it the reducer actually reads; a scalar feeder like `scale` in `SUM(pop[*] * scale)` carries an empty slice; `Iterated` carries the (target, source) canonical dim pair, equal for the literal case; `Reduced.subset` is `None` for the full extent or the proper-subdimension element subset for a `SUM(arr[*:Sub])` StarRange, GH #766, decided per axis by `classify_axis_access` -- the single per-axis classifier of the shape-expressiveness design) -- and a `result_dims` (the `Iterated` axes' TARGET dims, datamodel-cased -- empty for a whole-extent or pinned-slice reduce). Under the I1 acceptance (`accept_source_slices`, T5 of the shape-expressiveness design / GH #767) every arrayed CO-SOURCE (`Reduced`-bearing slice) carries the identical *canonical* slice (`AggNode::canonical_read_slice` -- the first `Reduced`-bearing source slice, falling back to the first non-empty for the degenerate no-co-source agg), while an ITERATED-DIM PROJECTION FEEDER -- a source whose slice is all-`Iterated` over exactly the canonical slice's iterated target dims, in order, unmapped (`frac[D1]` in `SUM(matrix[D1,*] * frac[D1])`, per-result-slot constant) -- is accepted with ITS OWN slice (`AggNode::source_is_projection_feeder` is the discriminator); per-source consumers (`emit_agg_routed_edges`, `emit_source_to_agg_link_scores`) read `AggNode::source_read_slice(from)`, and name-keyed consumers use `AggNode::reads_var`. A feeder's link-score half is the per-`(row, slot)` CHANGED-LAST equation (`ltm_augment::generate_iterated_feeder_to_agg_equation`, emitted via `link_scores::iterated_feeder_row_scores` from both the synthetic source-half and `try_cross_dimensional_link_scores`' variable-backed feeder branch): the reducer text pinned to the slot with only the feeder frozen -- the arrayed generalization of the GH #737 scalar-feeder convention, exactly complementary per slot to the co-source rows' changed-first numerators for a bilinear body; the co-source rows' changed-first body partial pins the mismatched-arity feeder dep BY DIM NAME (`pin_body_to_row`'s GH #767 extension) so `PREVIOUS(frac[d1·r])` is held frozen at the row instead of bailing to the delta-ratio fallback. `compute_read_slice` decides hoistability: a whole-extent reduce (`SUM(pop[*])` ⇒ all-`Reduced`), a sliced one (`SUM(pop[NYC,*])` ⇒ `[Pinned(nyc), Reduced]`, `SUM(matrix[D1,*])` over an A2A-`D1` body ⇒ `[Iterated(d1,d1), Reduced]` → an arrayed agg over `D1`), a mixed one (`SUM(matrix3d[D1,NYC,*])` over an A2A-`D1` body ⇒ `[Iterated(d1,d1), Pinned(nyc), Reduced]`), or a positionally-MAPPED sliced one (GH #534: `SUM(matrix[State,*])` over `matrix[Region,D2]` with a positional `State→Region` mapping ⇒ `[Iterated{state, region}, Reduced]`, `result_dims = [State]` -- the agg is arrayed over the TARGET's iterated dim and the three `Iterated`-axis consumers remap each source row to the slot of its positionally-corresponding target element via `iterated_axis_slot_elements`, the preimage inversion of `mapped_element_correspondence`, so the positional-only/element-map gate is inherited) is hoisted; the carve-outs are (a) a reducer over a *dynamic index* (`SUM(pop[idx,*])`, `idx` non-literal ⇒ not statically describable ⇒ not hoisted, reference stays on the conservative path -- `db::ltm_ir` reclassifies it as `DynamicIndex`), (b) an ELEMENT-mapped sliced reducer the correspondence declines (execution resolves positionally and ignores the map, GH #756; a POSITIONAL mapping is accepted in either declaration direction since GH #757) ⇒ `compute_read_slice` returns `None`, conservative -- (c) a multi-source slice combination outside the I1 acceptance -- co-sources with differing slices, one variable read with two different slices (I3b), or a no-`Reduced` source that is not the pure iterated projection (a Pinned-axis mix like `SUM(matrix[D1,*] * w[D1, c1])`, a dim-subset/permuted feeder, or any mapped Iterated axis in a feeder combination) -- `combined_read_slice`/`accept_source_slices` return `None`, (d) a StarRange naming a NON-subdimension of its axis (a mid-edit inconsistency; declined rather than silently widened to the full extent), and (e) `RANK` (GH #771: array-valued, so `reducer_is_hoistable` requires `reducer_collapses_to_scalar` and RANK references stay `Direct` -- a bare arg classifies `Bare`, scored by the GH #742 arrayed-capture path; loops through the rank ORDERING are a documented residual). A multi-source reducer whose arrayed args *agree* (`SUM(a[*] + b[*])`, `a`, `b` over the same dim) mints one agg with one `AggSource` per variable, each carrying the shared canonical slice. `AggNodesResult` exposes `aggs` (first-encounter order), `agg_for_key` (by canonical reducer text), and `aggs_in_var` (which aggs occur in a variable's equation, so the element-graph reroute can ask "which agg of `to` reads `from`?"). A variable-backed reduce with a NON-TRIVIAL statically-describable slice is gated by `variable_backed_reduce_agg` (GH #752, generalized by GH #765 / T3 of the shape-expressiveness design) -- the SAME gate `model_element_causal_edges`' dispatch, `build_element_level_loops`' per-circuit routing, and `try_cross_dimensional_link_scores`' row derivation consume, so edges, loop routing, and scores always cover the identical read rows (all three derive from `read_slice_rows`, invariant I4). Accepted: an ALIGNED partial reduce (`row_sum[D1] = SUM(matrix[D1,*])`, `result_dims` equal to the variable's own dims in order -- Pinned-mixed `outf[D1] = MEAN(cube[D1,x,*])` and subset `out[D1] = MEAN(matrix[D1,*:Sub])` slices included: the divisor is the true read count and unread rows get neither edges nor scores) gets read-slice element edges straight onto the variable's element nodes and per-circuit scalar loops whose both-subscripted `matrix[d1,d2]→row_sum[d1]` links resolve the per-`(row, slot)` scores; a scalar-result Pinned/subset slice on a SCALAR owner (`total = SUM(pop[nyc,*])`, `total = SUM(arr[*:Sub])`) routes the read rows into the bare `to` node with matching per-read-row scores; and the ARRAYED-owner scalar-result Pinned/subset BROADCAST slice (`share[Region] = SUM(pop[nyc,*])`, no `Iterated` axis -- GH #777) fans each read row across the FULL target element set (`emit_agg_routed_edges`' broadcast arm emits `pop[nyc,d2] → share[e]` for every `e`; `emit_broadcast_reduce_link_scores` emits the matching per-(read-row, full-target-element) scalar scores `pop[nyc,d2]→share[e]`, the section-3 `PerElement` rule applied to a variable-backed reducer owner), with loop circuits routed to the per-circuit scalar path via `is_broadcast_reduce_edge` -- the read rows are independent of `to`'s dims, so the RELATED-dim (`share[Region]`) and DISJOINT-dim (`share[D9]`) spellings emit identically. Declined: a pure full-extent variable-backed agg keeps the normal reference walker's edges (the true reads for that shape, inert skip). The GH #764 broadcast/permuted result shapes never reach this gate since T4 -- they mint synthetic aggs at enumeration -- so the gate's Iterated-arm alignment check is defense-in-depth. `scalar_feeder_of_variable_backed_agg` (GH #790) is the scalar sibling of `source_is_projection_feeder`: it composes `variable_backed_reduce_agg` with an empty-slice + genuine-`Reduced`-canonical check to recognize a SCALAR FEEDER of a whole-RHS variable-backed reduce (`scale` in `growth[D1] = SUM(matrix[D1,*] * scale)`), so `try_scalar_to_arrayed_link_scores` can route it to the single Bare A2A changed-last feeder score instead of the uncompilable per-target-element partials. Two predicates here answer what LOOKS like one question -- "is this reference inside a reducer?" -- and their answers are inverted on exactly `SIZE` and `RANK`; the inversion is the DEFINITION of the difference, not a disagreement (GH #982, assessed and left as two predicates). `reducer_collapses_to_scalar` is about the reducer's RESULT TYPE (does the subtree fit in a scalar slot?) and is read by the two freeze/capture gates plus the GH #779 bare-reducer-feeder decline: `SIZE` is a count so it collapses, `RANK` is array-valued so it does not. `builtin_routes_through_agg` is about LTM ROUTING (did `enumerate_agg_nodes` mint a node for this call?) and sets `db::ltm_ir::OccurrenceSite::in_reducer`: `SIZE` is `Constant` and is never hoisted, `RANK` gets an array-valued agg. Both read the ONE `reducer_kind_from_name` table, and `builtin_routes_through_agg` is the disjunction of the enumerator's own two minting branches (`reducer_is_hoistable` and `array_valued_rank_arg`) rather than a restatement of them. The `#[cfg(test)]` `REDUCER_DECISION_TABLE` pins all three derived predicates row by row over every arm of the kind table -- including the agreement gate's name-keyed twin of the routing predicate -- so no cell can move silently. `is_synthetic_agg_name` / `synthetic_agg_name` are the `$⁚ltm⁚agg⁚{n}` name helpers. Each node also carries `reducer: BuiltinFn` -- the reducer call the enumerator classified when it decided the hoist, of which `equation_text` is the printed rendering (GH #983). It is what makes the link-score and polarity emitters parse-free: `ltm_augment::classify_reducer_in_builtin` reads the kind/name/body off it and `ltm::CausalGraph::source_to_agg_polarity` analyses it directly, where both used to print `equation_text`, re-parse it, re-lower it against a freshly built scope and re-derive the classification -- per (agg, source) pair, with both fallible steps returning early and silently zeroing the agg's loop score. It is stored in `Expr2::strip_loc_and_bounds` form, which removes TWO of the three ways this field can make the salsa-cached `AggNodesResult` compare unequal to an identical rebuild: `Loc` (load-bearing -- two AST-identical occurrences differ in byte offsets, so raw storage would make the dedup winner observable and would stop `enumerate_agg_nodes` backdating across an offset-only edit) and `ArrayBounds` (a guard -- inert today, since `reconstruct_model_variables` lowers against an empty model scope and allocates no bound). Neither reader looks at either. It does NOT remove the third and cannot: `Expr2::Const` holds an `f64`, and a derived `PartialEq` over one is not reflexive on NaN, so a model whose hoisted reducer contains a `nan` literal permanently defeats this query's backdating. That is the GH #987/#981 class -- the same mechanism that forces `db::stages_tests`' stdlib oracle onto `npv` -- fixed at the root by making AST float literals compare by bit pattern, and pinned here meanwhile by the characterization test `a_nan_literal_in_a_reducer_defeats_agg_backdating`, which that fix must invert. The stored builtin is read only for SYNTHETIC aggs (both readers filter to those); the variable-backed arm's copy is unread today, kept so `AggNode` has one shape. `AggNode`/`AggNodesResult` are consequently `PartialEq` but not `Eq` -- same `f64` -- and `Debug` only under `debug-derive`. - **`src/ltm_augment.rs`** - Equation generators for LTM synthetic variables: `generate_link_score_equation_for_link` (ceteris-paribus link scores; takes `RefShape` and source dimension elements to drive per-shape PREVIOUS wrapping), `generate_loop_score_variables` (emits one `loop_score` per loop as a dimension-shaped `datamodel::Equation`: `Scalar` for scalar loops, `ApplyToAll` for dimensioned loops whose links resolve through Bare A2A names, and per-slot `Equation::Arrayed` for dimensioned loops backed by per-element circuits via `Loop.slot_links` -- GH #653; relative loop scores are computed post-simulation in `ltm_post.rs`), `build_partial_equation_shaped` (the `#[cfg(test)]` TEXT entry point for the ceteris-paribus wrap; arrayed-per-element-equation (`Ast::Arrayed`) targets get one partial per element assembled into an `Equation::Arrayed`). **Production never parses a target equation**: `wrap_changed_first_ast` takes an `Expr0` lowered straight from the target's `Expr2` by `patch::expr2_to_expr0` (which is what `expr2_to_string` prints, so the former print->reparse was a parse of our own output), and every per-occurrence decision -- access shape, the GH #526 other-dep verdict, the literal-element index guard, and the `PerElement` row pinning -- is a lookup into the `db::ltm_ir` occurrence IR by the structural child-index path the wrap tracks, which equals the occurrence's `SiteId` BY CONSTRUCTION now that both walk the same tree. A subscripted source reference the IR did NOT record -- reachable under a `LOOKUP` TABLE argument, which the walker skips as static data ("not a causal edge", which is right about ATTRIBUTION) -- still has to COMPILE, so the pin-only descent discharges it by NAME (`pin_dimension_name_indices`: an index naming one of the TARGET's iterated dimensions becomes the source element this target element reads on that axis, the same structural substitution `pin_bare_source_ref` performs for a bare `Var`). That is a lowering-completeness rule, not a second classifier: it consults no occurrence, infers no shape, and never makes the reference live-selectable -- it asks the SHARED row derivation `per_element_row_for_target` (hence `DimensionsContext::mapped_element_correspondence`) which element an axis reads, so the identity axis and a positionally-MAPPED one (`effect[State, old]` over an `effect[Region, Age]` source, either declaration direction) are one arm and it accepts exactly the mapped pairs `ltm_agg::classify_axis_access` accepts. An index the source's axis DECLARES as an element is resolved BEFORE that dimension-name reading, matching `compiler::subscript`'s `normalize_subscripts3` ("First check if it's a named dimension element (takes priority)") -- the two readings collide when a dimension declares an element whose name is also a dimension the target iterates, and there the element must win or the pin spells a row the simulation never reads. `classify_axis_access` still has the opposite order (GH #986: a precision gap today, since its dimension-first reading finds no mapping and declines). An index that already selects a FIXED element is left verbatim, needing no pin and reading nothing at the current step: a numeric literal, arithmetic over numeric literals (`index_expr_selects_a_fixed_element`, deliberately the narrowest sound predicate -- `compiler::invariance` is the engine's run-invariance derivation but consumes lowered `Expr` and answers a wider question), or an `@N` position index, which `compiler::context` resolves to a concrete element offset in scalar context (spelling `@N` out here would be a second implementation of position syntax). A 0-arity BUILTIN index is where that widening deliberately stops: `TIME` and `PI` are indistinguishable inside `Expr0::App` without a fourth copy of the builtin classification, so the arm stays conservatively loud. What the SHARED derivation declines -- an unmapped or element-mapped pair (GH #756), a transposition, a dimension the target does not iterate -- is declined LOUDLY (`WrapOutcome::missing_occurrence` -> warned skip) rather than emitted with its dimension-name subscript intact, which would not resolve in a scalar fragment. The whole two-verdict space (`Unspellable`, a compilability verdict loud everywhere; `RuntimeRead`, a ceteris-paribus verdict loud only in a bare table argument) is ENUMERATED cell by cell in `ltm_augment_pin_tests.rs`'s two verdict enumerations rather than sampled. There is ONE access-shape classifier family, on `Expr2`; the Expr0 mirror is test-support only and lives in `ltm_augment_wrap_test_support.rs` (kept because three wrap unit tests -- unparseable text, empty text, and the Fig. 2 Q4 `SUM(w[from]) + from` shape the engine REJECTS as a model -- cannot be db-backed fixtures), with `ltm_classifier_agreement_tests.rs` proving it matches the IR field for field (`SiteId` path, `shape`, `axes`, `in_reducer`) corpus-wide, `link_score_var_name` (synthetic name helper: Bare gets the canonical `{from}\u{2192}{to}` form, FixedIndex prepends `[elem]` to from; the obsolete per-shape `\u{205A}wildcard`/`\u{205A}dynamic` Wildcard/DynamicIndex suffixes were retired -- those shapes now collapse onto the Bare name, since *every statically-describable* inlined reducer (whole-extent or sliced) is hoisted into a `$⁚ltm⁚agg⁚{n}` node and only a `DynamicIndex` reference -- `arr[i+1]`, a range, or the not-hoistable dynamic-index reducer carve-out `SUM(pop[idx,*])` -- a whole-RHS variable-backed reducer's `Wildcard` argument, or a de-hoisted array-valued reducer's `Wildcard` arg (`RANK(pop[*], 1)`, GH #771) reach this function), `quote_ident` (identifier quoting for equations). Array support: `classify_reducer` (walks target Expr2 AST to identify reducing builtins -- Linear for SUM/MEAN, Nonlinear for MIN/MAX/STDDEV/RANK, Constant for SIZE -- a thin reader of `ltm_agg::reducer_kind`; it also hands back the reducer's array-argument AST as `ClassifiedReducer::body`, lowered from `Expr2` rather than printed, so the body-aware row partials never re-parse it), `generate_element_to_scalar_equation` (per-element link score equations for arrayed-to-scalar edges, used by both the variable-backed-reducer path and the `source[d] → $⁚ltm⁚agg⁚{n}` half) which dispatches on `ReducerKind` -- `generate_linear_partial` (SUM/MEAN algebraic shortcut), `generate_nonlinear_partial` (MIN/MAX nested binary calls; STDDEV the unrolled population-variance `sqrt` ceteris-paribus partial -- divisor `N`, matching `vm.rs::Opcode::ArrayStddev`, with the mean string-inlined; RANK the documented delta-ratio stand-in pinned by `test_generate_rank_keeps_delta_ratio` -- an order statistic, non-differentiable and unreachable via a real model RHS), `generate_scalar_to_element_equation` (per-element link score for the `$⁚ltm⁚agg⁚{n} → target[e]` half; takes a `source_ref_override: Option<&str>` so a multi-slot arrayed agg's `Δsource` denominator carries the projected `agg[]` subscript instead of the bare agg name, which wouldn't compile as a scalar), `substitute_reducers_in_expr0` (textually replaces a recognized reducer subexpression in an `Expr0` with its agg name, for the `$⁚ltm⁚agg⁚{n} → target` link score), `resolve_link_score_name_for_loop` (picks the Bare-or-FixedIndex link-score name a loop-score reference should target). Module link score formulas (black-box delta-ratio and composite-ref) are inlined directly into `module_link_score_equation` in `db.rs` (called by the per-shape `link_score_equation_text_shaped`). The partial-equation builders (`build_partial_equation_shaped`/`_with_live_ref`, `subscript_idents_at_element`) and every link-score equation generator return `Result<_, PartialEquationError>`: a parse failure (genuine `Err`, or an empty `Ok(None)` equation) has no AST to PREVIOUS-wrap, so emitting the unwrapped input would silently produce a non-ceteris-paribus "partial" identical to the full equation (link score magnitude constant |Δz/Δz| = 1) -- a hidden attribution error that compiles cleanly (GH #311). The db-bearing callers (`link_score_equation_text_shaped` and the `src/db/ltm/link_scores.rs` emitters) convert the error into a `Warning` (`emit_ltm_partial_equation_warning`, naming the variable + offending equation text) and skip the variable -- distinct from `model_ltm_fragment_diagnostics`, which only catches *compile* failures. The failure is effectively unreachable in production (the text is always a `print_eqn` re-print; an empty equation is rejected as an `EmptyEquation` Error upstream), so this is defense-in-depth. The `#[cfg(test)]` tests live in the sibling `src/ltm_augment_tests.rs` (split out for the per-file line cap). Four more siblings are `#[path]`-mounted into `ltm_augment` purely for that cap, so every caller still names their items `crate::ltm_augment::*`: **`ltm_augment_occurrence.rs`** (the wrap's read side of the occurrence IR -- `SlotOccurrences` groups a target's stream by slot ONCE and is the only way to obtain an `OccurrenceLookup`, so the borrow forces callers to hoist it out of their per-element loop), **`ltm_augment_post_transform.rs`** (the concrete-form lowerings: the agg-name substitution, and the `PerElement` row pinning the wrap calls AS IT DESCENDS -- the wrap is the only place that knows both the occurrence and whether it is about to FREEZE the reference, which is what picks the bare row for the live occurrence over the qualified row for every other one), **`ltm_augment_with_lookup.rs`** (the GH #910 implicit-WITH-LOOKUP rules), and **`ltm_augment_wrap_test_support.rs`** (the `#[cfg(test)]` occurrence reconstruction + the Expr0 classifier mirror described above). - **`src/ltm_augment_with_lookup.rs`** - The implicit-WITH-LOOKUP rules for LTM (GH #910), re-exported from `ltm_augment` so every `crate::ltm_augment::*` path is unchanged. A `v = WITH LOOKUP(input, table)` variable is lowered by the compiler to `LOOKUP(v, input)` (`compiler::apply_implicit_with_lookup`), so a link-score partial that RE-EVALUATES such a target's equation is in gf-INPUT units while the guard form's `PREVIOUS(target)` anchor is in gf-OUTPUT units. `is_implicit_with_lookup` carries the coverage doc: every partial is either a **full re-evaluation** (class 1 -- must be wrapped in the gf application) or a **delta-ratio stand-in** (class 2 -- the RANK arm and the nested-arithmetic arm, already in output units, must NEVER be wrapped or a gf output is fed back through the gf). `WithLookupSlotRefs` resolves the target's table reference ONCE per target (`NoGf` / `Shared` / `PerElement`), so a per-element-gf target costs one row-major `SubscriptIterator` walk rather than one per element; an arrayed target's shared table is pinned as `to[1,...]` because a bare arrayed reference resolves each iterated element's own table offset, which the VM reads as NaN past `table_count`. `compose_with_lookup_polarity` (in `src/ltm/polarity.rs`) is the polarity twin, mirroring `apply_implicit_with_lookup`'s placeholder-Positive and zero-point-table rules. The polarity tests live in `src/ltm/with_lookup_tests.rs`, a child module of `ltm::tests`. - **`src/ltm_post.rs`** - Post-simulation relative loop *and link* score computation. `compute_rel_loop_scores(results, loop_partitions)` normalizes each loop's `loop_score` series against the sum of absolute scores within its cycle partition, using SAFEDIV-0 semantics (zero denominator -> zero result). Called after simulation rather than emitted as synthetic equations to avoid O(P^2) equation-text growth on models with dense partitions. `loop_partitions` is an `IndexMap` (re-exported `engine::indexmap`). `compute_rel_loop_scores*` walk its **emission** order rather than re-sorting the loop ids: the partition-sum denominator accumulates `|loop_score|` in that order, and emission order keeps the IEEE-754 (non-associative) sum bit-for-bit identical to the pre-#461 compile-time emitter (GH #468). Emission order is itself deterministic across salsa cache invalidations / processes because `assign_loop_ids` orders loops by a content-derived key (`ltm::graph::loop_id_sort_key`), so it never flaps even though `IndexMap`'s `PartialEq` (salsa cache equality) is order-insensitive. `compute_rel_link_scores(links, step_count)` is the link-level analogue (GH #652): raw link scores divide by the change in the *target*, so they are not comparable across targets and ranking by raw magnitude surfaces numerically-degenerate links (near-constant targets blow up the score). It groups the input `RelLinkInput { to, score }` links by their `to` target and normalizes each link's score by the per-target, per-timestep sum of `|score|` over all that target's *scored* inputs -- a **signed** value in `[-1, 1]` (sign kept like `compute_rel_loop_scores`), with the same `denom_summand` NaN-exclusion / Inf-retention / SAFEDIV-0 semantics. Denominator scope is the scored inputs only: complete in discovery mode (every causal edge scored) but covering just the in-loop subset in exhaustive mode (documented caveat on the fn). libsimlin's `analyze_links_core` calls it over the final (post-synthetic-collapse) link set so the per-target denominator matches the links the caller receives. diff --git a/src/simlin-engine/src/ast/expr2.rs b/src/simlin-engine/src/ast/expr2.rs index 4db51ad96..653be837a 100644 --- a/src/simlin-engine/src/ast/expr2.rs +++ b/src/simlin-engine/src/ast/expr2.rs @@ -108,6 +108,20 @@ impl IndexExpr2 { } } + /// The [`Expr2::strip_loc_and_bounds`] twin for one subscript index. + pub(crate) fn strip_loc_and_bounds(self) -> Self { + let loc = Loc::default(); + match self { + IndexExpr2::Wildcard(_) => IndexExpr2::Wildcard(loc), + IndexExpr2::StarRange(dim, _) => IndexExpr2::StarRange(dim, loc), + IndexExpr2::Range(l, r, _) => { + IndexExpr2::Range(l.strip_loc_and_bounds(), r.strip_loc_and_bounds(), loc) + } + IndexExpr2::DimPosition(n, _) => IndexExpr2::DimPosition(n, loc), + IndexExpr2::Expr(e) => IndexExpr2::Expr(e.strip_loc_and_bounds()), + } + } + #[allow(dead_code)] pub(crate) fn get_var_loc(&self, ident: &str) -> Option { match self { @@ -194,6 +208,63 @@ pub trait Expr2Context { } impl Expr2 { + /// The expression with every `Loc` zeroed and every [`ArrayBounds`] + /// annotation dropped -- its position- and lowering-independent form. + /// + /// Both stripped fields are artifacts of *where* an expression was + /// written rather than *what* it means: a `Loc` is a byte range into one + /// variable's equation text, and a `Temp` bound carries a temp id the + /// lowering context handed out in equation order. Two occurrences of the + /// same subexpression in different equations therefore differ in both + /// while denoting the same thing, so a cache that stores an expression + /// keyed on its canonical printed form (`ltm_agg::AggNode`) must store + /// this form or its value stops being a function of its key. + /// + /// Necessary, not sufficient: `Expr2::Const` holds an `f64`, whose `==` is + /// not reflexive on NaN, so a NaN-bearing expression compares unequal to + /// itself however it is normalized. See `ltm_agg::AggNode::reducer` for + /// that residual and the root fix it waits on. + pub(crate) fn strip_loc_and_bounds(self) -> Self { + let loc = Loc::default(); + match self { + Expr2::Const(text, value, _) => Expr2::Const(text, value, loc), + Expr2::Var(ident, _, _) => Expr2::Var(ident, None, loc), + Expr2::App(builtin, _, _) => Expr2::App( + builtin + .map(|arg| arg.strip_loc_and_bounds()) + .strip_own_locs(), + None, + loc, + ), + Expr2::Subscript(ident, indices, _, _) => Expr2::Subscript( + ident, + indices + .into_iter() + .map(IndexExpr2::strip_loc_and_bounds) + .collect(), + None, + loc, + ), + Expr2::Op1(op, rhs, _, _) => { + Expr2::Op1(op, Box::new(rhs.strip_loc_and_bounds()), None, loc) + } + Expr2::Op2(op, lhs, rhs, _, _) => Expr2::Op2( + op, + Box::new(lhs.strip_loc_and_bounds()), + Box::new(rhs.strip_loc_and_bounds()), + None, + loc, + ), + Expr2::If(cond, then_e, else_e, _, _) => Expr2::If( + Box::new(cond.strip_loc_and_bounds()), + Box::new(then_e.strip_loc_and_bounds()), + Box::new(else_e.strip_loc_and_bounds()), + None, + loc, + ), + } + } + /// Extract the array bounds from an expression, if it has one pub(crate) fn get_array_bounds(&self) -> Option<&ArrayBounds> { match self { diff --git a/src/simlin-engine/src/builtins.rs b/src/simlin-engine/src/builtins.rs index 3d3dc9d38..bf7f94e0b 100644 --- a/src/simlin-engine/src/builtins.rs +++ b/src/simlin-engine/src/builtins.rs @@ -269,6 +269,65 @@ impl BuiltinFn { .unwrap() } + /// Zero every `Loc` this builtin carries ITSELF -- the three lookup forms + /// and `IsModuleInput` are the only variants with one. Argument + /// expressions are untouched; a caller normalizing a whole tree strips + /// those with [`Self::map`]. + /// + /// Written as an exhaustive match with no catch-all (the `Loc`-free + /// variants are bound whole through one or-pattern) so a new + /// `Loc`-carrying variant is a compile error here rather than a silently + /// retained source position. + /// NOTE: keep variant coverage in sync with `try_map` above. + pub(crate) fn strip_own_locs(self) -> Self { + use BuiltinFn::*; + match self { + Lookup(a, b, _) => Lookup(a, b, Loc::default()), + LookupForward(a, b, _) => LookupForward(a, b, Loc::default()), + LookupBackward(a, b, _) => LookupBackward(a, b, Loc::default()), + IsModuleInput(id, _) => IsModuleInput(id, Loc::default()), + locless @ (Abs(_) + | Arccos(_) + | Arcsin(_) + | Arctan(_) + | Cos(_) + | Exp(_) + | Inf + | Int(_) + | Ln(_) + | Log10(_) + | Max(_, _) + | Mean(_) + | Min(_, _) + | Pi + | Pulse(_, _, _) + | Quantum(_, _) + | Ramp(_, _, _) + | SafeDiv(_, _, _) + | Sign(_) + | Sshape(_, _, _) + | Sin(_) + | Sqrt(_) + | Step(_, _) + | Tan(_) + | Time + | TimeStep + | StartTime + | FinalTime + | Rank(_, _) + | Size(_) + | Stddev(_) + | Sum(_) + | VectorSelect(_, _, _, _, _) + | VectorElmMap(_, _) + | VectorSortOrder(_, _) + | AllocateAvailable(_, _, _) + | AllocateByPriority(_, _, _, _, _) + | Previous(_, _) + | Init(_)) => locless, + } + } + /// Call a closure on each expression argument by reference. /// NOTE: keep variant coverage in sync with `try_map` above. pub fn for_each_expr_ref(&self, mut f: F) diff --git a/src/simlin-engine/src/db/ltm/link_scores.rs b/src/simlin-engine/src/db/ltm/link_scores.rs index 5e840222a..f7340a747 100644 --- a/src/simlin-engine/src/db/ltm/link_scores.rs +++ b/src/simlin-engine/src/db/ltm/link_scores.rs @@ -26,9 +26,7 @@ use super::compile::{ShapedLinkScore, link_score_equation_text_shaped}; use super::loops::{ ReadSliceRow, ReadSliceRowParts, cartesian_subscripts, read_slice_row_parts, read_slice_rows, }; -use super::parse::{ - ltm_equation_dimensions, reconstruct_ltm_var_lowered, retarget_ltm_equation_dims, -}; +use super::parse::{ltm_equation_dimensions, retarget_ltm_equation_dims}; /// The occurrence-stream slot each target element maps to when the /// ceteris-paribus wrap walks one slot's expression at a time. @@ -2862,24 +2860,22 @@ pub(super) fn emit_source_to_agg_link_scores( vars.extend(feeder_vars); return; } - // Reconstruct a transient (parsed + lowered) `Variable` from the - // agg's equation text so `classify_reducer` can read the reducer - // kind/name. An arrayed agg (`result_dims` non-empty -- a sliced - // reducer like `SUM(matrix[D1,*])` over an A2A-`D1` body) must be - // reconstructed as an `ApplyToAll` over `result_dims`; treating - // `matrix[d1,*]` as a scalar equation is a type error, so the lowered - // reconstruction would fail and no per-read-row source link scores - // would be emitted (silently zeroing the synthetic agg's loop score). - // Mirrors the agg-aux emission above. - let agg_eqn = if agg.result_dims.is_empty() { - LtmEquation::scalar(agg.equation_text.clone()) - } else { - LtmEquation::apply_to_all(agg.result_dims.clone(), agg.equation_text.clone()) - }; - let Some(agg_var) = reconstruct_ltm_var_lowered(db, &agg.name, &agg_eqn, model, project) else { - return; - }; - let Some(classified) = crate::ltm_augment::classify_reducer(&agg_var, from) else { + // The reducer kind / name / body come straight off the aggregate node + // (GH #983). This used to print `agg.equation_text`, re-parse it, re-lower + // it against a freshly built scope and re-classify the result -- a closed + // round trip through our own printer and parser, run once per + // (agg, source) pair, whose two fallible steps both returned early and + // silently zeroed the agg's loop score. It also forced the emitter to + // rebuild the equation SHAPE by hand: an arrayed agg had to be + // reconstructed as an `ApplyToAll` over `result_dims`, because treating + // `matrix[d1,*]` as a scalar equation is a type error whose lowering + // failure emitted no scores at all. Reading the classified builtin + // removes both hazards -- there is no shape to rebuild. + let Some(classified) = crate::ltm_augment::classify_reducer_in_builtin( + &agg.reducer, + from, + /* is_top_level = */ true, + ) else { return; }; if classified.kind == crate::ltm_augment::ReducerKind::Constant { diff --git a/src/simlin-engine/src/db/ltm/loops.rs b/src/simlin-engine/src/db/ltm/loops.rs index bc68eea15..26343cfdb 100644 --- a/src/simlin-engine/src/db/ltm/loops.rs +++ b/src/simlin-engine/src/db/ltm/loops.rs @@ -1794,33 +1794,17 @@ where /// first), so the hop polarity -- and hence the polarity-prefixed loop ids /// the runtime join is keyed on -- is decided in exactly one place. fn source_to_agg_hop_polarity( - db: &dyn Db, - model: SourceModel, - project: SourceProject, var_graph: &crate::ltm::CausalGraph, from_var_level: &str, agg: &crate::ltm_agg::AggNode, ) -> crate::ltm::LinkPolarity { - use crate::ltm::LinkPolarity; - - // Reconstruct the agg's lowered AST from its equation text (the agg is - // not a model variable, so the graph's variable map has no AST for it). - // Mirrors `emit_source_to_agg_link_scores`' reconstruction. - let agg_eqn = if agg.result_dims.is_empty() { - super::LtmEquation::scalar(agg.equation_text.clone()) - } else { - super::LtmEquation::apply_to_all(agg.result_dims.clone(), agg.equation_text.clone()) - }; - let Some(agg_var) = - super::parse::reconstruct_ltm_var_lowered(db, &agg.name, &agg_eqn, model, project) - else { - return LinkPolarity::Unknown; - }; - let Some(agg_ast) = agg_var.ast() else { - return LinkPolarity::Unknown; - }; + // The agg's body is the reducer call the enumerator classified, carried + // on the node (GH #983). It used to be recovered by printing + // `equation_text`, re-parsing it and re-lowering it against a freshly + // built scope -- and an agg whose reconstruction failed came back + // `Unknown`, degrading every loop through it to `Undetermined`. let source = Ident::::new(from_var_level); - var_graph.source_to_agg_polarity(&source, agg_ast) + var_graph.source_to_agg_polarity(&source, &agg.reducer) } /// Recover the polarity of synthetic-aggregate-node hops in `loops` (GH #516). @@ -1888,8 +1872,7 @@ pub(crate) fn recover_agg_hop_polarities( .iter() .find(|(name, _)| name.as_str() == to_var_level) { - let p = - source_to_agg_hop_polarity(db, model, project, var_graph, from_var_level, agg); + let p = source_to_agg_hop_polarity(var_graph, from_var_level, agg); if p != LinkPolarity::Unknown { link.polarity = p; patched = true; diff --git a/src/simlin-engine/src/db/ltm/parse.rs b/src/simlin-engine/src/db/ltm/parse.rs index 83a698a57..508957940 100644 --- a/src/simlin-engine/src/db/ltm/parse.rs +++ b/src/simlin-engine/src/db/ltm/parse.rs @@ -17,17 +17,16 @@ //! itself (`scalarize`, `retarget_dims`, `dimensions`); the thin wrappers below //! keep the call sites' free-function spelling. -use std::collections::{HashMap, HashSet}; +use std::collections::HashSet; use crate::builtins_visitor::{empty_macro_registry, instantiate_implicit_modules}; use crate::common::{Canonical, Ident}; use crate::datamodel; use crate::dimensions::DimensionsContext; -use crate::db::{Db, ParsedVariableResult, SourceModel, SourceProject}; -use crate::db::{project_datamodel_dims, project_dimensions_context}; +use crate::db::ParsedVariableResult; -use super::{LtmEquation, ltm_module_idents}; +use super::LtmEquation; /// Parse an LTM synthetic variable's typed equation into a /// `Variable` plus any implicit helper/module variables the @@ -100,63 +99,6 @@ pub(super) fn parse_ltm_equation( } } -pub(super) fn parse_ltm_equation_for_model_with_ids( - db: &dyn Db, - var_name: &str, - equation: &LtmEquation, - project: SourceProject, - module_idents: &HashSet>, - model_var_names: &HashSet>, -) -> ParsedVariableResult { - let dims = project_datamodel_dims(db, project); - parse_ltm_equation( - var_name, - equation, - dims, - Some(module_idents), - Some(model_var_names), - ) -} - -/// Parse *and lower* an LTM equation to a `Variable` -/// (the type `classify_reducer` and the rest of the type-checked AST machinery -/// expect). Mirrors the parse-then-lower boilerplate `compile_ltm_equation_fragment` -/// does; scoped with an empty model set and the project's datamodel dimensions. -/// Returns `None` if the equation fails to parse. -pub(super) fn reconstruct_ltm_var_lowered( - db: &dyn Db, - var_name: &str, - equation: &LtmEquation, - model: SourceModel, - project: SourceProject, -) -> Option { - let dim_context = project_dimensions_context(db, project); - let module_idents = ltm_module_idents(db, model, project); - let model_var_names = super::ltm_model_var_names(db, model, project); - let parsed = parse_ltm_equation_for_model_with_ids( - db, - var_name, - equation, - project, - module_idents, - model_var_names, - ); - if parsed - .variable - .equation_errors() - .is_some_and(|e| !e.is_empty()) - { - return None; - } - let models = HashMap::new(); - let scope = crate::model::ScopeStage0 { - models: &models, - dimensions: dim_context, - model_name: "", - }; - Some(crate::model::lower_variable(&scope, &parsed.variable)) -} - /// The dimension names an LTM `LtmEquation` carries (datamodel casing), /// or `&[]` for a scalar one. Thin wrapper over [`LtmEquation::dimensions`] /// so the call sites keep their free-function spelling. diff --git a/src/simlin-engine/src/db/ltm_ir.rs b/src/simlin-engine/src/db/ltm_ir.rs index 264261b46..b25ebbd4f 100644 --- a/src/simlin-engine/src/db/ltm_ir.rs +++ b/src/simlin-engine/src/db/ltm_ir.rs @@ -1486,13 +1486,20 @@ pub(crate) struct OccurrenceSite { /// gate, which scopes its per-occurrence shape comparison to non-reducer /// references and asserts both streams agree on reducer context. /// - /// NOT yet consumed by the transform's GH #779 bare-reducer-feeder decline, - /// which re-derives "inside a scalar reducer" with its own walk: the two - /// decisions use two different reducer SETS - /// (`ltm_agg::reducer_collapses_to_scalar` includes `SIZE` and excludes - /// `RANK`; `builtin_routes_through_agg`, which sets this bit, does the - /// opposite), so consuming it there would flip a bare source inside - /// `RANK(...)` from scored to loudly declined. Tracked as GH #982. + /// Deliberately NOT consumed by the transform's GH #779 + /// bare-reducer-feeder decline, which asks a DIFFERENT question with its + /// own walk: "does this subtree collapse to a scalar?" + /// (`ltm_agg::reducer_collapses_to_scalar`) rather than "did an aggregate + /// node get minted for it?" (`builtin_routes_through_agg`, which sets this + /// bit). The two answers are inverted on exactly `SIZE` and `RANK`, and + /// that inversion IS the difference between the questions -- `SIZE` is a + /// scalar count that is never hoisted, `RANK` is array-valued but gets an + /// array-valued agg. Consuming this bit there would flip a bare source + /// inside `RANK(...)` from scored to loudly declined, a user-visible score + /// change with no argument behind it. Assessed and resolved that way in + /// GH #982; `ltm_agg::reducer_collapses_to_scalar`'s doc carries the full + /// comparison and `ltm_agg::REDUCER_DECISION_TABLE` pins both predicates, + /// row by row, so neither can move silently. pub in_reducer: bool, /// `true` iff the occurrence is reachable ONLY through another reference's /// subscript index (`other_arr[from]`). Such an occurrence is excluded diff --git a/src/simlin-engine/src/ltm/graph.rs b/src/simlin-engine/src/ltm/graph.rs index 93b884866..d0affcb26 100644 --- a/src/simlin-engine/src/ltm/graph.rs +++ b/src/simlin-engine/src/ltm/graph.rs @@ -1279,19 +1279,19 @@ impl CausalGraph { } /// Polarity of a `feeder → $⁚ltm⁚agg` hop: the discriminating analysis of - /// the agg's (lowered) body with respect to a *scalar feeder* referenced - /// inside the reducer (GH #737 review follow-up). `agg_ast` is the agg's - /// own lowered equation (the caller reconstructs it from - /// `AggNode::equation_text`, since the agg has no node in this - /// variable-level graph). See + /// the agg's reducer call with respect to a *scalar feeder* referenced + /// inside it (GH #737 review follow-up). `reducer` is + /// `ltm_agg::AggNode::reducer`, the classified builtin the agg stands in + /// for -- the agg has no node in this variable-level graph, so there is + /// no AST here to look it up from. See /// [`super::polarity::analyze_source_to_agg_polarity`] for the /// positive-by-convention Mul rule and the Unknown fallback. pub(crate) fn source_to_agg_polarity( &self, feeder: &Ident, - agg_ast: &crate::ast::Ast, + reducer: &crate::builtins::BuiltinFn, ) -> LinkPolarity { - super::polarity::analyze_source_to_agg_polarity(agg_ast, feeder, &self.variables) + super::polarity::analyze_source_to_agg_polarity(reducer, feeder, &self.variables) } /// Compute per-link polarity for all edges in the causal graph. diff --git a/src/simlin-engine/src/ltm/polarity.rs b/src/simlin-engine/src/ltm/polarity.rs index 84b666afa..44c3df639 100644 --- a/src/simlin-engine/src/ltm/polarity.rs +++ b/src/simlin-engine/src/ltm/polarity.rs @@ -12,6 +12,7 @@ use std::collections::HashMap; use crate::ast::{Ast, BinaryOp, Expr2, IndexExpr2}; +use crate::builtins::BuiltinFn; use crate::common::{Canonical, Ident}; use crate::variable::Variable; @@ -54,12 +55,25 @@ pub(super) fn analyze_link_polarity( /// An agg's body is the one context where a blanket monotone-Positive /// label was previously applied unconditionally, so a convention-scoped /// analysis is strictly more accurate there. +/// +/// `reducer` is the reducer call itself (`ltm_agg::AggNode::reducer`), not a +/// whole equation: an aggregate node's equation IS one reducer application, +/// so the `Ast` wrapper the caller used to reconstruct carried no +/// information -- [`analyze_ast_polarity`] treats `Ast::Scalar` and +/// `Ast::ApplyToAll` through the same arm, and an agg is never +/// `Ast::Arrayed`. pub(super) fn analyze_source_to_agg_polarity( - agg_ast: &Ast, + reducer: &BuiltinFn, source: &Ident, variables: &HashMap, Variable>, ) -> LinkPolarity { - analyze_ast_polarity(agg_ast, source, variables, /* mul_convention = */ true) + analyze_builtin_polarity( + reducer, + source, + LinkPolarity::Positive, + Some(variables), + /* mul_convention = */ true, + ) } /// Shared AST-level dispatch for [`analyze_link_polarity`] / @@ -267,80 +281,28 @@ pub(super) fn analyze_expr_polarity_with_context( ) } -/// The recursive polarity walk. `mul_convention` enables the -/// positive-by-convention Mul one-side rule (feeder-hop analysis ONLY; see -/// [`analyze_source_to_agg_polarity`]); `false` reproduces the general -/// analyzer exactly. -fn analyze_expr_polarity_impl( - expr: &Expr2, +/// The `Expr2::App` half of [`analyze_expr_polarity_impl`], reachable from a +/// `BuiltinFn` on its own so that a hoisted reducer's polarity +/// ([`analyze_source_to_agg_polarity`]) can be analysed without wrapping the +/// builtin back up in an `Expr2::App` it was taken out of. +/// +/// The `App` node's own `ArrayBounds` and `Loc` are not read by any arm, so +/// nothing is lost by dropping the wrapper. +fn analyze_builtin_polarity( + builtin: &BuiltinFn, from_var: &Ident, current_polarity: LinkPolarity, variables: Option<&HashMap, Variable>>, mul_convention: bool, ) -> LinkPolarity { - match expr { - Expr2::Const(_, _, _) => LinkPolarity::Unknown, - Expr2::Var(ident, _, _) => { - let normalized = normalize_module_ref(ident); - if &normalized == from_var || ident == from_var { - current_polarity - } else { - LinkPolarity::Unknown - } - } - // Whole-array reductions wrap a Subscript around the same identifier - // that a scalar reference would carry as Expr2::Var. The reducer arms - // below (Sum/Mean/single-arg Max/Min) recurse into their argument; for - // the production case `SUM(x[*])` that argument lowers to - // `Subscript(x, [Wildcard], _, _)`, not `Var(x, ...)`. Mirror the Var - // handler so the identifier comparison succeeds and the reducer's - // monotonicity guarantee carries through. - // - // When the array name matches `from_var`, the indices still need - // inspection: if any index expression also references `from_var` - // (e.g. `arr[INT(arr[i])]` or `arr[arr]`), the relationship is - // non-monotone -- shifting `from_var` moves both the lookup target - // and the index in lockstep -- and we must return Unknown. The - // dominant cases (literal, wildcard, range, expressions over OTHER - // variables) leave indices independent of `from_var`, and the - // reducer's monotonicity guarantee carries through unchanged. - // - // When the array name does NOT match `from_var`, contribute Unknown: - // we can't classify references that thread through another array - // here. Combining operators above (Add/Sub/Mul/Div, Mean variadic) - // detect any `from_var` reference inside indices via their own - // `expr_references_var` checks. - Expr2::Subscript(ident, indices, _, _) => { - let normalized = normalize_module_ref(ident); - if &normalized == from_var || ident == from_var { - if indices.iter().any(|idx| match idx { - IndexExpr2::Expr(e) => expr_references_var(e, from_var), - IndexExpr2::Range(lo, hi, _) => { - expr_references_var(lo, from_var) || expr_references_var(hi, from_var) - } - IndexExpr2::Wildcard(_) - | IndexExpr2::StarRange(_, _) - | IndexExpr2::DimPosition(_, _) => false, - }) { - LinkPolarity::Unknown - } else { - current_polarity - } - } else { - LinkPolarity::Unknown - } - } + match builtin { // All three lookup variants share the `(table_expr, index_expr, loc)` // shape and the same polarity story: the result is non-decreasing in // the index when the table is, so the link polarity is the argument's // monotonicity composed with the table's. - Expr2::App( - crate::builtins::BuiltinFn::Lookup(table_expr, index_expr, _) - | crate::builtins::BuiltinFn::LookupForward(table_expr, index_expr, _) - | crate::builtins::BuiltinFn::LookupBackward(table_expr, index_expr, _), - _, - _, - ) => { + BuiltinFn::Lookup(table_expr, index_expr, _) + | BuiltinFn::LookupForward(table_expr, index_expr, _) + | BuiltinFn::LookupBackward(table_expr, index_expr, _) => { let arg_polarity = analyze_expr_polarity_impl( index_expr, from_var, @@ -360,17 +322,16 @@ fn analyze_expr_polarity_impl( // Non-decreasing single-arg builtins: propagate inner polarity. // Int (floor) is a step function with discontinuities, but is still // non-decreasing, which is sufficient for polarity propagation. - Expr2::App(crate::builtins::BuiltinFn::Exp(inner), _, _) - | Expr2::App(crate::builtins::BuiltinFn::Ln(inner), _, _) - | Expr2::App(crate::builtins::BuiltinFn::Log10(inner), _, _) - | Expr2::App(crate::builtins::BuiltinFn::Sqrt(inner), _, _) - | Expr2::App(crate::builtins::BuiltinFn::Arctan(inner), _, _) - | Expr2::App(crate::builtins::BuiltinFn::Int(inner), _, _) => { + BuiltinFn::Exp(inner) + | BuiltinFn::Ln(inner) + | BuiltinFn::Log10(inner) + | BuiltinFn::Sqrt(inner) + | BuiltinFn::Arctan(inner) + | BuiltinFn::Int(inner) => { analyze_expr_polarity_impl(inner, from_var, current_polarity, variables, mul_convention) } // Max/Min (scalar two-arg form): non-decreasing in each argument - Expr2::App(crate::builtins::BuiltinFn::Max(a, Some(b)), _, _) - | Expr2::App(crate::builtins::BuiltinFn::Min(a, Some(b)), _, _) => { + BuiltinFn::Max(a, Some(b)) | BuiltinFn::Min(a, Some(b)) => { let pol_a = analyze_expr_polarity_impl( a, from_var, @@ -417,10 +378,10 @@ fn analyze_expr_polarity_impl( // form MEAN(a, b, c); for polarity that form is still monotone in each // argument, so we combine arg polarities the same way Add does (any // disagreement collapses to Unknown). - Expr2::App(crate::builtins::BuiltinFn::Sum(arg), _, _) => { + BuiltinFn::Sum(arg) => { analyze_expr_polarity_impl(arg, from_var, current_polarity, variables, mul_convention) } - Expr2::App(crate::builtins::BuiltinFn::Mean(args), _, _) => { + BuiltinFn::Mean(args) => { let mut combined = LinkPolarity::Unknown; for arg in args { let arg_pol = analyze_expr_polarity_impl( @@ -457,16 +418,87 @@ fn analyze_expr_polarity_impl( } // Array reducers MAX/MIN (single-arg form): max/min of a monotone family // is monotone, so propagate the inner polarity. - Expr2::App(crate::builtins::BuiltinFn::Max(a, None), _, _) - | Expr2::App(crate::builtins::BuiltinFn::Min(a, None), _, _) => { + BuiltinFn::Max(a, None) | BuiltinFn::Min(a, None) => { analyze_expr_polarity_impl(a, from_var, current_polarity, variables, mul_convention) } // STDDEV is non-monotone (variance has no fixed sign w.r.t. inputs). // RANK depends on the rest of the array, so its sign w.r.t. one element // is not determined. Both must explicitly return Unknown. - Expr2::App(crate::builtins::BuiltinFn::Stddev(_), _, _) - | Expr2::App(crate::builtins::BuiltinFn::Rank(_, _), _, _) => LinkPolarity::Unknown, - Expr2::App(_, _, _) => LinkPolarity::Unknown, + BuiltinFn::Stddev(_) | BuiltinFn::Rank(_, _) => LinkPolarity::Unknown, + _ => LinkPolarity::Unknown, + } +} + +/// The recursive polarity walk. `mul_convention` enables the +/// positive-by-convention Mul one-side rule (feeder-hop analysis ONLY; see +/// [`analyze_source_to_agg_polarity`]); `false` reproduces the general +/// analyzer exactly. +fn analyze_expr_polarity_impl( + expr: &Expr2, + from_var: &Ident, + current_polarity: LinkPolarity, + variables: Option<&HashMap, Variable>>, + mul_convention: bool, +) -> LinkPolarity { + match expr { + Expr2::Const(_, _, _) => LinkPolarity::Unknown, + Expr2::Var(ident, _, _) => { + let normalized = normalize_module_ref(ident); + if &normalized == from_var || ident == from_var { + current_polarity + } else { + LinkPolarity::Unknown + } + } + // Whole-array reductions wrap a Subscript around the same identifier + // that a scalar reference would carry as Expr2::Var. The reducer arms + // below (Sum/Mean/single-arg Max/Min) recurse into their argument; for + // the production case `SUM(x[*])` that argument lowers to + // `Subscript(x, [Wildcard], _, _)`, not `Var(x, ...)`. Mirror the Var + // handler so the identifier comparison succeeds and the reducer's + // monotonicity guarantee carries through. + // + // When the array name matches `from_var`, the indices still need + // inspection: if any index expression also references `from_var` + // (e.g. `arr[INT(arr[i])]` or `arr[arr]`), the relationship is + // non-monotone -- shifting `from_var` moves both the lookup target + // and the index in lockstep -- and we must return Unknown. The + // dominant cases (literal, wildcard, range, expressions over OTHER + // variables) leave indices independent of `from_var`, and the + // reducer's monotonicity guarantee carries through unchanged. + // + // When the array name does NOT match `from_var`, contribute Unknown: + // we can't classify references that thread through another array + // here. Combining operators above (Add/Sub/Mul/Div, Mean variadic) + // detect any `from_var` reference inside indices via their own + // `expr_references_var` checks. + Expr2::Subscript(ident, indices, _, _) => { + let normalized = normalize_module_ref(ident); + if &normalized == from_var || ident == from_var { + if indices.iter().any(|idx| match idx { + IndexExpr2::Expr(e) => expr_references_var(e, from_var), + IndexExpr2::Range(lo, hi, _) => { + expr_references_var(lo, from_var) || expr_references_var(hi, from_var) + } + IndexExpr2::Wildcard(_) + | IndexExpr2::StarRange(_, _) + | IndexExpr2::DimPosition(_, _) => false, + }) { + LinkPolarity::Unknown + } else { + current_polarity + } + } else { + LinkPolarity::Unknown + } + } + Expr2::App(builtin, _, _) => analyze_builtin_polarity( + builtin, + from_var, + current_polarity, + variables, + mul_convention, + ), Expr2::Op2(op, left, right, _, _) => { let left_pol = analyze_expr_polarity_impl( left, diff --git a/src/simlin-engine/src/ltm/tests.rs b/src/simlin-engine/src/ltm/tests.rs index aa5a44ae3..bdae54a53 100644 --- a/src/simlin-engine/src/ltm/tests.rs +++ b/src/simlin-engine/src/ltm/tests.rs @@ -5923,12 +5923,15 @@ fn test_source_to_agg_polarity_discriminates_body_sign() { loc, ) }; - let sum = |arg: Expr2| Expr2::App(crate::builtins::BuiltinFn::Sum(Box::new(arg)), None, loc); + // The analyzed unit is the reducer CALL, exactly as it rides on + // `ltm_agg::AggNode::reducer` (GH #983) -- an aggregate node's equation is + // one reducer application, so there is no enclosing `Ast` to supply. + let sum = |arg: Expr2| crate::builtins::BuiltinFn::Sum(Box::new(arg)); let empty_vars = HashMap::new(); // SUM(pop[*] * scale): the co-factor is a bare arrayed reference, // positive by the SD labeling convention -> Positive. - let headline = Ast::Scalar(sum(op2(BinaryOp::Mul, pop_wild(), var("scale")))); + let headline = sum(op2(BinaryOp::Mul, pop_wild(), var("scale"))); assert_eq!( analyze_source_to_agg_polarity(&headline, &scale, &empty_vars), LinkPolarity::Positive, @@ -5938,11 +5941,11 @@ fn test_source_to_agg_polarity_discriminates_body_sign() { // SUM(pop[*] * (1 - scale)): the feeder enters through a negation -> // Negative (the review's demonstrated counterexample to the blanket // monotone-Positive label). - let negating = Ast::Scalar(sum(op2( + let negating = sum(op2( BinaryOp::Mul, pop_wild(), op2(BinaryOp::Sub, cnst(1.0), var("scale")), - ))); + )); assert_eq!( analyze_source_to_agg_polarity(&negating, &scale, &empty_vars), LinkPolarity::Negative, @@ -5953,11 +5956,11 @@ fn test_source_to_agg_polarity_discriminates_body_sign() { // its value sign is derived, not a convention -- so the hop polarity is // indeterminate and must fall back to Unknown (never a confident wrong // label). - let compound = Ast::Scalar(sum(op2( + let compound = sum(op2( BinaryOp::Mul, op2(BinaryOp::Sub, var("k"), pop_wild()), var("scale"), - ))); + )); assert_eq!( analyze_source_to_agg_polarity(&compound, &scale, &empty_vars), LinkPolarity::Unknown, @@ -5966,11 +5969,8 @@ fn test_source_to_agg_polarity_discriminates_body_sign() { // STDDEV(pop[*] * scale): a non-monotone reducer stays Unknown even // with a convention-positive body. - let stddev = Ast::Scalar(Expr2::App( - crate::builtins::BuiltinFn::Stddev(Box::new(op2(BinaryOp::Mul, pop_wild(), var("scale")))), - None, - loc, - )); + let stddev = + crate::builtins::BuiltinFn::Stddev(Box::new(op2(BinaryOp::Mul, pop_wild(), var("scale")))); assert_eq!( analyze_source_to_agg_polarity(&stddev, &scale, &empty_vars), LinkPolarity::Unknown, @@ -5981,7 +5981,11 @@ fn test_source_to_agg_polarity_discriminates_body_sign() { // headline body is Unknown there -- the convention applies ONLY to the // feeder-hop analysis, never to general link polarity. assert_eq!( - analyze_link_polarity(&headline, &scale, &empty_vars), + analyze_link_polarity( + &Ast::Scalar(Expr2::App(headline, None, loc)), + &scale, + &empty_vars + ), LinkPolarity::Unknown, "the general analyzer must NOT adopt the convention Mul rule" ); diff --git a/src/simlin-engine/src/ltm_agg.rs b/src/simlin-engine/src/ltm_agg.rs index 016522763..1ca04cd0a 100644 --- a/src/simlin-engine/src/ltm_agg.rs +++ b/src/simlin-engine/src/ltm_agg.rs @@ -208,6 +208,29 @@ pub(crate) fn reducer_kind_from_name(name: &str, arity: usize) -> Option bool { reducer_kind_from_name(name, arity).is_some() && name != "rank" } @@ -222,6 +245,93 @@ pub(crate) fn reducer_kind(builtin: &BuiltinFn) -> Option { reducer_kind_from_name(builtin.name(), builtin_reducer_arity(builtin)) } +/// The reducer decision table as DATA, for the tests that pin it. +/// +/// One row per `(name, arity)` pair needed to reach every arm of +/// [`reducer_kind_from_name`]: its six `Some` arms name seven functions +/// (`min | max` share an arm), two of those arms carry an `arity == 1` guard +/// so `mean`/`min`/`max` each need a failing-arity row as well, and the +/// catch-all needs one row -- 7 + 3 + 1 = 11 rows. Each row carries the kind +/// and all three derived predicates, so the `SIZE`/`RANK` inversion between +/// [`reducer_collapses_to_scalar`] and [`builtin_routes_through_agg`] +/// (GH #982) is pinned in BOTH directions rather than asserted from one side. +/// +/// Shared with `ltm_augment::classifier_agreement_tests`, whose name-based +/// twin of `builtin_routes_through_agg` must agree row for row. +#[cfg(test)] +pub(crate) const REDUCER_DECISION_TABLE: &[ReducerDecisionRow] = &[ + ReducerDecisionRow::new("sum", 1, Some(ReducerKind::Linear), true, true, true), + ReducerDecisionRow::new("mean", 1, Some(ReducerKind::Linear), true, true, true), + ReducerDecisionRow::new("mean", 2, None, false, false, false), + ReducerDecisionRow::new("min", 1, Some(ReducerKind::Nonlinear), true, true, true), + ReducerDecisionRow::new("min", 2, None, false, false, false), + ReducerDecisionRow::new("max", 1, Some(ReducerKind::Nonlinear), true, true, true), + ReducerDecisionRow::new("max", 2, None, false, false, false), + ReducerDecisionRow::new("stddev", 1, Some(ReducerKind::Nonlinear), true, true, true), + // The two inverted cells: array-valued but agg-routed ... + ReducerDecisionRow::new("rank", 1, Some(ReducerKind::Nonlinear), false, false, true), + // ... and scalar-valued but never routed. + ReducerDecisionRow::new("size", 1, Some(ReducerKind::Constant), true, false, false), + ReducerDecisionRow::new("abs", 1, None, false, false, false), +]; + +/// One row of [`REDUCER_DECISION_TABLE`]. +#[cfg(test)] +pub(crate) struct ReducerDecisionRow { + pub name: &'static str, + pub arity: usize, + pub kind: Option, + /// [`reducer_collapses_to_scalar`]: does the subtree evaluate to a scalar? + pub collapses_to_scalar: bool, + /// [`reducer_is_hoistable`]: does a scalar-reducer agg get minted? + pub is_hoistable: bool, + /// [`builtin_routes_through_agg`]: do references inside it route to an agg? + pub routes_through_agg: bool, +} + +#[cfg(test)] +impl ReducerDecisionRow { + const fn new( + name: &'static str, + arity: usize, + kind: Option, + collapses_to_scalar: bool, + is_hoistable: bool, + routes_through_agg: bool, + ) -> Self { + ReducerDecisionRow { + name, + arity, + kind, + collapses_to_scalar, + is_hoistable, + routes_through_agg, + } + } + + /// The `BuiltinFn` this row's `(name, arity)` names, so the builtin-keyed + /// predicates can be checked against the same row as the name-keyed ones. + /// Panics on an unknown row, which keeps the table and this constructor in + /// step. + pub(crate) fn builtin(&self) -> BuiltinFn { + match (self.name, self.arity) { + ("sum", 1) => BuiltinFn::Sum(Box::new(0)), + ("mean", n) => BuiltinFn::Mean((0..n as i32).collect()), + ("min", 1) => BuiltinFn::Min(Box::new(0), None), + ("min", 2) => BuiltinFn::Min(Box::new(0), Some(Box::new(1))), + ("max", 1) => BuiltinFn::Max(Box::new(0), None), + ("max", 2) => BuiltinFn::Max(Box::new(0), Some(Box::new(1))), + ("stddev", 1) => BuiltinFn::Stddev(Box::new(0)), + // `RANK(arr, dir)` reports arity 1: `builtin_reducer_arity` counts + // only the reduced argument, and the deciders ignore arity here. + ("rank", 1) => BuiltinFn::Rank(Box::new(0), Box::new(1)), + ("size", 1) => BuiltinFn::Size(Box::new(0)), + ("abs", 1) => BuiltinFn::Abs(Box::new(0)), + other => unreachable!("no BuiltinFn for decision-table row {other:?}"), + } + } +} + /// The arity [`reducer_kind_from_name`] / [`reducer_collapses_to_scalar`] /// key on. Only `MEAN`/`MIN`/`MAX` are arity-sensitive; for everything else /// the deciders ignore the arity argument. @@ -254,10 +364,20 @@ pub(crate) fn reducer_is_hoistable(builtin: &BuiltinFn) -> bool { } /// `true` when references inside `builtin` may route through an aggregate -/// node. Scalar reducers use [`reducer_is_hoistable`]; array-valued `RANK` -/// uses a synthetic arrayed agg whose source half has RANK-specific routing. +/// node -- the LTM ROUTING question, and the sole setter of +/// `db::ltm_ir::OccurrenceSite::in_reducer`. +/// +/// It is the disjunction of [`agg_candidate_for_builtin`]'s two minting +/// branches, read from the branches themselves rather than restated: +/// [`reducer_is_hoistable`] is the scalar-reducer branch and +/// [`array_valued_rank_arg`] is the array-valued one. Stating it that way is +/// what keeps "can this call mint an agg" and "do references in it route to +/// one" from drifting apart. +/// +/// See [`reducer_collapses_to_scalar`] for why the two "is this inside a +/// reducer?" predicates deliberately disagree on `SIZE` and `RANK` (GH #982). pub(crate) fn builtin_routes_through_agg(builtin: &BuiltinFn) -> bool { - reducer_is_hoistable(builtin) || matches!(builtin, BuiltinFn::Rank(_, _)) + reducer_is_hoistable(builtin) || array_valued_rank_arg(builtin).is_some() } /// How one *source axis* of a hoisted reducer is consumed. @@ -410,7 +530,8 @@ pub struct AggSource { } /// One aggregate node: the stand-in for a maximal reducer subexpression. -#[derive(Clone, Debug, PartialEq, Eq, salsa::Update)] +#[cfg_attr(feature = "debug-derive", derive(Debug))] +#[derive(Clone, PartialEq, salsa::Update)] pub struct AggNode { /// The aggregate node's name. For a synthetic agg this is /// `$⁚ltm⁚agg⁚{n}`; for a variable-backed agg this is the owning @@ -449,6 +570,66 @@ pub struct AggNode { /// output element in the same iterated context, so the source→agg half /// fans each read row out across all non-pinned result-axis slots. pub array_valued_rank: bool, + /// The reducer call itself: the very `BuiltinFn` this enumerator + /// classified when it decided the hoist, of which `equation_text` is the + /// printed rendering. + /// + /// It is here so that no downstream consumer has to recover the reducer's + /// kind, name, or body by parsing and re-lowering `equation_text` (GH + /// #983): [`crate::ltm_augment::classify_reducer_in_builtin`] reads the + /// first two and hands back the third, and + /// `db::ltm::loops::source_to_agg_hop_polarity` analyses it directly. + /// + /// **Read only for SYNTHETIC aggs.** Both readers filter to those + /// (`recover_agg_hop_polarities` on `is_synthetic`; every + /// `emit_source_to_agg_link_scores` call site on `is_synthetic_agg_name` or + /// the IR's synthetic-only `routed_aggs`), so the copy `register_agg` + /// stores on the variable-backed arm is unread today. It is stored anyway + /// so `AggNode` has ONE shape -- an `Option` here would add a branch to + /// every reader to encode a fact about who happens to call them -- but + /// nothing pins it: corrupting it to a wrong kind, name and body leaves the + /// whole suite green, while the same corruption on the synthetic arm reds a + /// dozen-plus tests across the char goldens and the cross-agg recovery + /// suite. + /// + /// # What the stored form does and does not normalize + /// + /// Stored in [`Expr2::strip_loc_and_bounds`] form, which removes TWO of the + /// three ways this field can make the salsa-cached `AggNodesResult` + /// compare unequal to an identical rebuild. It does not remove the third. + /// + /// * `Loc` -- removed, and load-bearing. Two AST-identical occurrences in + /// different equations carry different `Loc`s, so storing them raw would + /// make *which occurrence won the dedup* observable and would stop + /// `enumerate_agg_nodes` backdating across an edit that only moves an + /// equation's byte offsets. Neither reader looks at a `Loc` + /// ([`crate::patch::expr2_to_expr0`] carries them along unread; the + /// polarity analyzer matches on none), so removing them changes no answer. + /// Pinned by `the_carried_reducer_is_normalized_so_offset_only_edits_backdate`. + /// * `ArrayBounds` -- removed as a GUARD, not a live requirement: the ASTs + /// this enumerator walks come from + /// `db::analysis::reconstruct_model_variables`, which lowers against an + /// EMPTY model scope, so `Expr2Context::get_dimensions` resolves nothing + /// and every bound is already `None`. Were that to change, a bound would + /// carry the temp id the lowering context happened to hand out in + /// equation order, and the cached value would become sensitive to + /// unrelated edits elsewhere in the owning equation. Neither reader looks + /// at one either -- `expr2_to_expr0` drops them outright. + /// * The `f64` on `Expr2::Const` -- **NOT removed, and not removable.** A + /// derived `PartialEq` over an `f64` is not reflexive on NaN, so a model + /// whose hoisted reducer contains a `nan` literal + /// (`out = 1 + SUM(pop[*] * nan)`) yields two enumerations that are + /// structurally identical and compare UNEQUAL -- permanently defeating + /// this query's backdating, so every revision bump re-executes + /// `model_element_causal_edges` / `model_ltm_reference_sites` / + /// `model_ltm_variables`. No normalization can fix it here: dropping a + /// NaN literal would change what the equation means. It is the GH + /// #987/#981 class (the same mechanism that forces `db::stages_tests`' + /// stdlib oracle onto `npv`), and it is fixed at the ROOT by making AST + /// float literals compare by bit pattern. Today's broken behaviour is + /// pinned by `a_nan_literal_in_a_reducer_defeats_agg_backdating`, which + /// that fix must invert. + pub reducer: BuiltinFn, } impl AggNode { @@ -550,7 +731,8 @@ impl AggNode { /// reuses a variable-backed agg of the same text (which would otherwise be /// filtered out by the `is_synthetic` checks downstream, leaving the inline /// reducer on the conservative direct-scoring path -- a name-ordering bug). -#[derive(Clone, Debug, PartialEq, Eq, Default, salsa::Update)] +#[cfg_attr(feature = "debug-derive", derive(Debug))] +#[derive(Clone, PartialEq, Default, salsa::Update)] pub struct AggNodesResult { /// Aggregate nodes in first-encounter (deterministic) order. pub aggs: Vec, @@ -770,6 +952,7 @@ fn walk_var_equation( // (`rowsum[D1] = SUM(matrix[D1, *])`) keeps `[D1]` as its result dims. let key = crate::patch::expr2_to_string(expr); let result_dims = candidate.result_dims; + let reducer = candidate.reducer; // DECLINE the degenerate square-source shape (repeated result dim, // GH #778/#785): the per-axis emission paths pin subscript indices by // dim name and disagree across the duplicated occurrence. Declining @@ -787,6 +970,7 @@ fn walk_var_equation( var_name: var_name.to_string(), result_dims, array_valued_rank: false, + reducer, }, sources, ); @@ -1000,6 +1184,7 @@ fn walk_subexpr_for_aggs( AggKind::Synthetic { result_dims: candidate.result_dims, array_valued_rank: candidate.array_valued_rank, + reducer: candidate.reducer, }, sources, ); @@ -1048,12 +1233,23 @@ struct AggCandidate { slices: CombinedReadSlices, result_dims: Vec, array_valued_rank: bool, + /// The classified reducer call, normalized for storage on the node + /// (see [`AggNode::reducer`]). + reducer: BuiltinFn, } fn agg_candidate_for_builtin( builtin: &BuiltinFn, ctx: &AggWalkCtx<'_>, ) -> Option { + // Cloned only once the builtin has been accepted as a candidate, so the + // non-reducer majority of App nodes pays nothing. + let normalized = || { + builtin + .clone() + .map(Expr2::strip_loc_and_bounds) + .strip_own_locs() + }; if let Some(rank_arg) = array_valued_rank_arg(builtin) { let source_vars = rank_source_vars(rank_arg, ctx.variables)?; let slices = rank_combined_read_slice(rank_arg, ctx)?; @@ -1066,6 +1262,7 @@ fn agg_candidate_for_builtin( slices, result_dims, array_valued_rank: true, + reducer: normalized(), }); } @@ -1077,10 +1274,20 @@ fn agg_candidate_for_builtin( slices, result_dims, array_valued_rank: false, + reducer: normalized(), }) } -fn array_valued_rank_arg(builtin: &BuiltinFn) -> Option<&Expr2> { +/// The ranked argument of an ARRAY-VALUED reducer, or `None` for every other +/// builtin -- the array-valued half of [`agg_candidate_for_builtin`]'s +/// two-branch minting decision (the other half being +/// [`reducer_is_hoistable`]). +/// +/// Generic over the contained expression type because it inspects only the +/// builtin's identity: [`builtin_routes_through_agg`] reads it so that "which +/// builtins can mint an array-valued agg" is stated once, here, rather than +/// restated as a second `matches!` beside the routing predicate. +fn array_valued_rank_arg(builtin: &BuiltinFn) -> Option<&E> { match builtin { BuiltinFn::Rank(arg, _) => Some(arg), _ => None, @@ -1209,12 +1416,14 @@ enum AggKind { Synthetic { result_dims: Vec, array_valued_rank: bool, + reducer: BuiltinFn, }, /// The owning variable already is the aggregate node. VariableBacked { var_name: String, result_dims: Vec, array_valued_rank: bool, + reducer: BuiltinFn, }, } @@ -1291,6 +1500,7 @@ fn register_agg( AggKind::Synthetic { result_dims, array_valued_rank, + reducer, } => { if let Some(&existing) = result.synthetic_by_key.get(key) { existing @@ -1304,6 +1514,7 @@ fn register_agg( sources, is_synthetic: true, array_valued_rank, + reducer, }); let idx = result.aggs.len() - 1; result.synthetic_by_key.insert(key.to_string(), idx); @@ -1314,6 +1525,7 @@ fn register_agg( var_name, result_dims, array_valued_rank, + reducer, } => { // Each whole-RHS-reducer variable is its own aggregate node; // never deduped, and not entered in `synthetic_by_key`. @@ -1324,6 +1536,7 @@ fn register_agg( sources, is_synthetic: false, array_valued_rank, + reducer, }); result.aggs.len() - 1 } @@ -4296,69 +4509,62 @@ mod tests { assert!(synthetic[0].result_dims.is_empty()); } - /// AC1.2: the consolidated `reducer_kind` table classifies every array - /// reducer the LTM machinery cares about, and `reducer_is_hoistable` - /// derives the right hoisted subset. `reducer_kind` is - /// generic over the contained expression type, so `BuiltinFn::` - /// literals suffice -- it only inspects structure and arity. + /// AC1.2 + GH #982: the consolidated reducer table classifies every array + /// reducer the LTM machinery cares about, and all THREE derived predicates + /// agree with the table row for row. + /// + /// The rows come from [`REDUCER_DECISION_TABLE`], whose 11 entries are + /// derived from `reducer_kind_from_name`'s own arms (see its doc): every + /// arm, every arity guard in both directions, and the catch-all. Each row + /// asserts the name-keyed decider, the builtin-keyed decider, the arity + /// the builtin form reports, and the three consumers -- so the `SIZE` / + /// `RANK` inversion between `reducer_collapses_to_scalar` ("does this + /// collapse to a scalar?") and `builtin_routes_through_agg` ("did an agg + /// get minted for it?") is pinned in both directions on both rows, and an + /// edit to either predicate reds here rather than drifting silently + /// (GH #982). #[test] fn reducer_kind_classifies_every_array_reducer() { - use crate::builtins::BuiltinFn; - - let sum = BuiltinFn::Sum(Box::new(0i32)); - let mean_1 = BuiltinFn::Mean(vec![0i32]); - let mean_2 = BuiltinFn::Mean(vec![0i32, 1i32]); - let min_1 = BuiltinFn::Min(Box::new(0i32), None); - let min_2 = BuiltinFn::Min(Box::new(0i32), Some(Box::new(1i32))); - let max_1 = BuiltinFn::Max(Box::new(0i32), None); - let max_2 = BuiltinFn::Max(Box::new(0i32), Some(Box::new(1i32))); - let stddev = BuiltinFn::Stddev(Box::new(0i32)); - let rank = BuiltinFn::Rank(Box::new(0i32), Box::new(1i32)); - let size = BuiltinFn::Size(Box::new(0i32)); - let abs = BuiltinFn::Abs(Box::new(0i32)); - - assert_eq!(reducer_kind(&sum), Some(ReducerKind::Linear)); - assert_eq!(reducer_kind(&mean_1), Some(ReducerKind::Linear)); - // Multi-argument MEAN is a scalar mean-of-arguments, not a reducer. - assert_eq!(reducer_kind(&mean_2), None); - assert_eq!(reducer_kind(&min_1), Some(ReducerKind::Nonlinear)); - // 2-arg MIN/MAX are scalar binary builtins, not reducers. - assert_eq!(reducer_kind(&min_2), None); - assert_eq!(reducer_kind(&max_1), Some(ReducerKind::Nonlinear)); - assert_eq!(reducer_kind(&max_2), None); - assert_eq!(reducer_kind(&stddev), Some(ReducerKind::Nonlinear)); - assert_eq!(reducer_kind(&rank), Some(ReducerKind::Nonlinear)); - assert_eq!(reducer_kind(&size), Some(ReducerKind::Constant)); - assert_eq!(reducer_kind(&abs), None); - - // Scalar-hoistable: recognized AND not Constant AND scalar-valued (I5) -- - // SUM / 1-arg MEAN / 1-arg MIN / 1-arg MAX / STDDEV. SIZE is - // recognized but never hoisted (its link score is always 0); RANK is - // not scalar-hoistable (array-valued -- it uses its own agg path, - // GH #771/#776); 2-arg MIN/MAX are not recognized at all. - assert!(reducer_is_hoistable(&sum)); - assert!(reducer_is_hoistable(&mean_1)); - assert!(reducer_is_hoistable(&min_1)); - assert!(reducer_is_hoistable(&max_1)); - assert!(reducer_is_hoistable(&stddev)); - assert!(!reducer_is_hoistable(&rank)); - assert!(!reducer_is_hoistable(&size)); - assert!(!reducer_is_hoistable(&mean_2)); - assert!(!reducer_is_hoistable(&min_2)); - assert!(!reducer_is_hoistable(&max_2)); - assert!(!reducer_is_hoistable(&abs)); - - // `reducer_kind_from_name` is the raw lowercase + arity decider that - // `is_array_reducer_name` reads: SIZE included; mean/min/max only at - // arity 1; sum/stddev/rank/size at any arity. - assert_eq!(reducer_kind_from_name("sum", 1), Some(ReducerKind::Linear)); - assert_eq!(reducer_kind_from_name("mean", 1), Some(ReducerKind::Linear)); - assert_eq!(reducer_kind_from_name("mean", 2), None); - assert_eq!( - reducer_kind_from_name("min", 1), - Some(ReducerKind::Nonlinear) - ); - assert_eq!(reducer_kind_from_name("min", 2), None); + for row in REDUCER_DECISION_TABLE { + let name = row.name; + let arity = row.arity; + let builtin = row.builtin(); + assert_eq!( + builtin_reducer_arity(&builtin), + arity, + "{name}/{arity}: the builtin form must report the row's arity" + ); + assert_eq!( + reducer_kind_from_name(name, arity), + row.kind, + "{name}/{arity}: reducer_kind_from_name" + ); + assert_eq!( + reducer_kind(&builtin), + row.kind, + "{name}/{arity}: reducer_kind" + ); + assert_eq!( + reducer_collapses_to_scalar(name, arity), + row.collapses_to_scalar, + "{name}/{arity}: reducer_collapses_to_scalar" + ); + assert_eq!( + reducer_is_hoistable(&builtin), + row.is_hoistable, + "{name}/{arity}: reducer_is_hoistable" + ); + assert_eq!( + builtin_routes_through_agg(&builtin), + row.routes_through_agg, + "{name}/{arity}: builtin_routes_through_agg" + ); + } + + // The table keys `stddev`/`rank`/`size`/`sum` at one arity each, but + // `reducer_kind_from_name` ignores arity for them -- only + // `mean`/`min`/`max` carry an `arity == 1` guard. Spot-check one, so a + // stray arity guard added to an arity-insensitive arm is caught. assert_eq!( reducer_kind_from_name("stddev", 7), Some(ReducerKind::Nonlinear) @@ -4367,11 +4573,6 @@ mod tests { reducer_kind_from_name("rank", 2), Some(ReducerKind::Nonlinear) ); - assert_eq!( - reducer_kind_from_name("size", 1), - Some(ReducerKind::Constant) - ); - assert_eq!(reducer_kind_from_name("abs", 1), None); } /// GH #776: a RANK subexpression mints an ARRAY-valued synthetic agg, @@ -5413,4 +5614,267 @@ mod tests { ] ); } + + /// GH #983: every recognized reducer's classification is CARRIED on the + /// SYNTHETIC node it decided, so no emitter has to recover it by + /// re-parsing [`AggNode::equation_text`]. + /// + /// Scope, stated because "every recognized reducer" is only one of the two + /// axes here: this covers all seven reducers on the SYNTHETIC producer arm, + /// which is the arm both readers actually reach. `register_agg`'s + /// variable-backed arm stores a `reducer` too and no test covers it, because + /// no reader consumes it -- see [`AggNode::reducer`]'s doc. + /// + /// The rows are the reducer set itself, not a sample. `reducer_kind` is + /// the only admission test, and [`crate::ltm_augment`]'s + /// `classify_builtin_if_references_source` -- the function that reads the + /// kind, name and body back off `AggNode::reducer` -- destructures exactly + /// SEVEN `BuiltinFn` variants (`Sum`, `Mean`, `Min`, `Max`, `Stddev`, + /// `Rank`, `Size`) and calls `unreachable!()` on the rest, so those seven + /// are the whole space and this table has seven rows. Each row states the + /// three facts the emitters read: whether a node is minted at all, and (if + /// so) the `ReducerKind` and uppercase name the carried builtin classifies + /// to. + /// + /// `SIZE` is the one row that mints nothing (`ReducerKind::Constant` is + /// never hoisted -- its link score is always 0), and `RANK` is the one + /// row that mints an ARRAY-valued node; both are properties of the + /// enumerator this table would notice changing. + #[test] + fn every_reducer_carries_its_classification_on_the_agg_node() { + use crate::ltm_augment::{ReducerKind, classify_reducer_in_builtin}; + + // (equation for `out`, out's dims, expected (kind, name, array_valued_rank)) + struct Row { + equation: &'static str, + out_dims: &'static [&'static str], + expected: Option<(ReducerKind, &'static str, bool)>, + } + let rows = [ + Row { + equation: "1 + SUM(pop[*])", + out_dims: &[], + expected: Some((ReducerKind::Linear, "SUM", false)), + }, + Row { + equation: "1 + MEAN(pop[*])", + out_dims: &[], + expected: Some((ReducerKind::Linear, "MEAN", false)), + }, + Row { + equation: "1 + MIN(pop[*])", + out_dims: &[], + expected: Some((ReducerKind::Nonlinear, "MIN", false)), + }, + Row { + equation: "1 + MAX(pop[*])", + out_dims: &[], + expected: Some((ReducerKind::Nonlinear, "MAX", false)), + }, + Row { + equation: "1 + STDDEV(pop[*])", + out_dims: &[], + expected: Some((ReducerKind::Nonlinear, "STDDEV", false)), + }, + Row { + // Array-valued: the node is arrayed over the ranked axis, so + // its consumer must be arrayed too. + equation: "1 + RANK(pop[*], 1)", + out_dims: &["Region"], + expected: Some((ReducerKind::Nonlinear, "RANK", true)), + }, + Row { + // `ReducerKind::Constant`: recognized, never hoisted. + equation: "1 + SIZE(pop[*])", + out_dims: &[], + expected: None, + }, + ]; + + for Row { + equation, + out_dims, + expected, + } in rows + { + let mut project = TestProject::new("carried") + .named_dimension("Region", &["nyc", "boston"]) + .array_aux("pop[Region]", "1"); + project = if out_dims.is_empty() { + project.scalar_aux("out", equation) + } else { + project.array_aux_direct( + "out", + out_dims.iter().map(|d| (*d).to_string()).collect(), + equation, + None, + ) + }; + let aggs = agg_nodes(&project); + let synthetic: Vec<&AggNode> = aggs.aggs.iter().filter(|a| a.is_synthetic).collect(); + + let Some((kind, name, array_valued_rank)) = expected else { + assert!( + synthetic.is_empty(), + "{equation}: a Constant reducer must mint no aggregate node" + ); + continue; + }; + assert_eq!( + synthetic.len(), + 1, + "{equation}: expected exactly one synthetic aggregate node" + ); + let agg = synthetic[0]; + assert_eq!( + agg.array_valued_rank, array_valued_rank, + "{equation}: array_valued_rank" + ); + + let classified = classify_reducer_in_builtin(&agg.reducer, "pop", true) + .unwrap_or_else(|| panic!("{equation}: the carried builtin must classify")); + assert_eq!(classified.kind, kind, "{equation}: kind"); + assert_eq!(classified.name, name, "{equation}: name"); + assert!( + classified.is_bare, + "{equation}: an aggregate node's equation IS the reducer call" + ); + // The body is the reducer's array argument, taken from the AST the + // enumerator walked rather than re-parsed from `equation_text`. + assert_eq!( + crate::ast::print_eqn(&classified.body), + "pop[*]", + "{equation}: body" + ); + } + } + + /// GH #983: the carried reducer is stored in + /// [`crate::ast::Expr2::strip_loc_and_bounds`] form, so a `Loc`-only edit + /// leaves the salsa-cached `AggNodesResult` equal. + /// + /// **This is one of the two ways the backdating claim can fail, and it + /// measures only this one.** The other is `f64` non-reflexivity on a NaN + /// literal, which normalization cannot touch; + /// [`a_nan_literal_in_a_reducer_defeats_agg_backdating`] pins that arm + /// (today: broken) and names the root fix. + /// + /// The two spellings differ ONLY in a leading term that mints no + /// aggregate node of its own -- a `SIZE` call, which is recognized as a + /// reducer but never hoisted -- so `SUM(pop[*])` moves to a different byte + /// offset and nothing else about the enumeration changes. The whole + /// `AggNodesResult` must therefore compare EQUAL, which is exactly + /// salsa's backdating criterion and so is the invalidation claim: an edit + /// that changes neither the reducer nor its sources must not invalidate + /// this query's consumers, as it did not before the node carried an AST + /// at all. + /// + /// It is the `Loc` half of the normalization that this measures. The + /// `ArrayBounds` half is inert TODAY and cannot be measured, because + /// `db::analysis::reconstruct_model_variables` -- the source of the ASTs + /// this query walks -- lowers against an EMPTY model scope, so + /// `Expr2Context::get_dimensions` resolves nothing and no bound is ever + /// allocated. The leading `SIZE(other[*])` is chosen over a bare constant + /// so that this test starts measuring the bounds half the moment that + /// stops being true: its arrayed argument would take the first temp id + /// and push `pop[*]` to the second. + /// + /// The second assertion is a weaker independent check on the same + /// property (re-normalizing the stored builtin is a no-op); it catches + /// normalization being dropped entirely but, being idempotence, cannot + /// by itself catch the normalization being made too weak. That is what + /// the equality assertion above is for. + #[test] + fn the_carried_reducer_is_normalized_so_offset_only_edits_backdate() { + let build = |leading: &str| { + TestProject::new("normalized") + .named_dimension("Region", &["nyc", "boston"]) + .array_aux("pop[Region]", "1") + .array_aux("other[Region]", "2") + .scalar_aux("out", &format!("{leading} + SUM(pop[*])")) + }; + let before = agg_nodes(&build("1")); + let after = agg_nodes(&build("SIZE(other[*])")); + + let sum_agg = |r: &AggNodesResult| { + r.aggs + .iter() + .find(|a| a.equation_text == "sum(pop[*])") + .cloned() + .expect("the SUM subexpression must be hoisted") + }; + assert_eq!( + before, after, + "an edit that only moves the reducer's byte offsets must leave the \ + enumerated aggregate nodes equal, so salsa backdates" + ); + let agg = sum_agg(&before); + assert_eq!( + agg.reducer + .clone() + .map(crate::ast::Expr2::strip_loc_and_bounds) + .strip_own_locs(), + agg.reducer, + "the stored reducer must already be normalized" + ); + } + + // CHARACTERIZATION, NOT A CONTRACT: this asserts today's BROKEN behaviour + // so the fix has something to turn red. + // + // `AggNode::reducer` (GH #983) puts an `Expr2` inside the salsa-cached, + // `PartialEq`-compared `AggNodesResult`, and `Expr2::Const` holds a bare + // `f64`. A derived `PartialEq` over an `f64` is not reflexive on NaN, so a + // model whose hoisted reducer contains a `nan` literal enumerates to a + // value that is structurally identical to an independent rebuild and + // compares UNEQUAL to it. Salsa's backdating criterion is exactly that + // equality, so such a model can never backdate `enumerate_agg_nodes`: + // every revision bump, from any unrelated edit anywhere in the project, + // re-executes `model_element_causal_edges`, `model_ltm_reference_sites` + // and `model_ltm_variables` -- the whole LTM compile. + // + // `strip_loc_and_bounds` closes the `Loc` and `ArrayBounds` channels (see + // `the_carried_reducer_is_normalized_so_offset_only_edits_backdate`); it + // cannot close this one, because there is no normalization that removes a + // NaN literal without changing what the equation means. This is the GH + // #987/#981 class -- the same mechanism that forces `db::stages_tests`' + // stdlib value oracle onto `npv` rather than `smth1` -- and the root fix is + // to make AST float literals compare by BIT PATTERN. + // + // When that lands, this test MUST be inverted, not deleted: switch to + // `assert_eq`, rename the test (it asserts the opposite of + // `defeats_agg_backdating` afterwards) along with the + // `nan_backdating_is_broken` project name, and update the three rustdocs + // that cite it by name -- `AggNode::reducer`, + // `the_carried_reducer_is_normalized_so_offset_only_edits_backdate`, and + // the `ltm_agg.rs` bullet in `simlin-engine/CLAUDE.md`. Its failure is the + // signal that the root fix reached this query. + #[test] + fn a_nan_literal_in_a_reducer_defeats_agg_backdating() { + let build = || { + TestProject::new("nan_backdating_is_broken") + .named_dimension("Region", &["nyc", "boston"]) + .array_aux("pop[Region]", "1") + .scalar_aux("out", "1 + SUM(pop[*] * nan)") + }; + + // Guard against a vacuous pass: the reducer really is hoisted, so the + // NaN really does ride on a stored `AggNode::reducer`. + let enumerated = agg_nodes(&build()); + let synthetic: Vec<&AggNode> = enumerated.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!( + synthetic.len(), + 1, + "the NaN-bearing reducer must still be hoisted for this to measure anything" + ); + + assert_ne!( + agg_nodes(&build()), + agg_nodes(&build()), + "TODAY'S BEHAVIOUR, not the desired one: two enumerations of an \ + identical NaN-bearing model compare unequal, so `enumerate_agg_nodes` \ + can never backdate for this model. Making AST float literals compare \ + by bit pattern (GH #987/#981) must flip this to `assert_eq!`." + ); + } } diff --git a/src/simlin-engine/src/ltm_augment.rs b/src/simlin-engine/src/ltm_augment.rs index 0f68015d9..0c5afdc5d 100644 --- a/src/simlin-engine/src/ltm_augment.rs +++ b/src/simlin-engine/src/ltm_augment.rs @@ -1417,12 +1417,24 @@ fn contains_unfreezable_previous(expr: &Expr0) -> bool { /// must not be touched. References already inside a `PREVIOUS(...)`/`INIT(...)` /// call are skipped (already lagged/frozen, not a live read this partial /// must account for). The reducer set comes from -/// `reducer_collapses_to_scalar`, so it also includes SIZE -- harmless: an -/// equation whose only reducer is SIZE keeps the changed-first convention -/// (the whole reducer is freezable as `PREVIOUS(size(...))`), so it does -/// not reach this gate. RANK is excluded by that predicate: it is +/// [`crate::ltm_agg::reducer_collapses_to_scalar`], so it also includes SIZE +/// -- harmless: an equation whose only reducer is SIZE keeps the changed-first +/// convention (the whole reducer is freezable as `PREVIOUS(size(...))`, +/// because [`expr_is_array_slice_valued`] reads the SAME predicate), so it +/// does not reach this gate. RANK is excluded by that predicate: it is /// array-valued and uses its own agg-routing path (GH #771/#776), so its /// bare arg is not this scalar-reducer feeder shape. +/// +/// This is deliberately NOT `db::ltm_ir::OccurrenceSite::in_reducer`, which +/// looks like the same "is this reference inside a reducer?" question but is +/// the LTM ROUTING one ("did an aggregate node get minted for this call?") and +/// therefore inverts on exactly SIZE and RANK. Consuming the IR bit here would +/// flip a bare arrayed source inside `RANK(...)` from scored (via the GH #742 +/// arrayed-capture path) to loudly declined -- a user-visible score change +/// with no argument behind it. Assessed in GH #982 and left as two predicates; +/// `ltm_agg::reducer_collapses_to_scalar`'s doc carries the comparison and +/// `ltm_agg::REDUCER_DECISION_TABLE` pins both of them row by row, so neither +/// can drift. fn references_bare_source_inside_reducer( expr: &Expr0, source: &Ident, @@ -4381,28 +4393,7 @@ fn classify_reducer_in_expr( match expr { Expr2::App(builtin, _, _) => { - // Check if this builtin is a reducer whose argument references - // the source variable. - if let Some((kind, name, body)) = - classify_builtin_if_references_source(builtin, source_ident) - { - return Some(ClassifiedReducer { - kind, - name, - is_bare: is_top_level, - body, - }); - } - // Even if this particular App node isn't the reducer we want, - // recurse into its arguments to find nested reducers. - // Any reducer found inside a non-reducer App is nested. - let mut result = None; - builtin.for_each_expr_ref(|sub_expr| { - if result.is_none() { - result = classify_reducer_in_expr(sub_expr, source_ident, false); - } - }); - result + classify_reducer_in_builtin(builtin, source_ident, is_top_level) } Expr2::Op1(_, inner, _, _) => classify_reducer_in_expr(inner, source_ident, false), Expr2::Op2(_, lhs, rhs, _, _) => classify_reducer_in_expr(lhs, source_ident, false) @@ -4416,6 +4407,42 @@ fn classify_reducer_in_expr( } } +/// [`classify_reducer_in_expr`]'s `App` arm, reachable directly from a +/// `BuiltinFn` the caller already holds. +/// +/// This is how the link-score emitters read the reducer kind, name and body +/// off [`crate::ltm_agg::AggNode::reducer`] (GH #983): an aggregate node's +/// equation IS one reducer call, so `is_top_level = true` reproduces exactly +/// what walking a reconstructed `Ast::Scalar(App(..))` used to produce -- +/// including the nested-reducer fallback below, which fires when the outer +/// reducer's array argument does not itself reference `source_ident`. +pub(crate) fn classify_reducer_in_builtin( + builtin: &crate::builtins::BuiltinFn, + source_ident: &str, + is_top_level: bool, +) -> Option { + // Check if this builtin is a reducer whose argument references the + // source variable. + if let Some((kind, name, body)) = classify_builtin_if_references_source(builtin, source_ident) { + return Some(ClassifiedReducer { + kind, + name, + is_bare: is_top_level, + body, + }); + } + // Even if this particular App node isn't the reducer we want, recurse + // into its arguments to find nested reducers. Any reducer found inside a + // non-reducer App is nested. + let mut result = None; + builtin.for_each_expr_ref(|sub_expr| { + if result.is_none() { + result = classify_reducer_in_expr(sub_expr, source_ident, false); + } + }); + result +} + /// If `builtin` is a recognized array reducer (per /// [`crate::ltm_agg::reducer_kind`]) whose array argument references the source /// variable, return its `(ReducerKind, uppercase function name, body text)`, diff --git a/src/simlin-engine/src/ltm_augment_tests.rs b/src/simlin-engine/src/ltm_augment_tests.rs index 72599f0e4..d192928a1 100644 --- a/src/simlin-engine/src/ltm_augment_tests.rs +++ b/src/simlin-engine/src/ltm_augment_tests.rs @@ -5214,6 +5214,23 @@ fn references_bare_source_inside_reducer_detects_only_the_dangerous_shape() { &frac, false )); + // SIZE completes the reducer set (`reducer_kind_from_name` recognizes + // seven functions; the loop above covers five and RANK is the sixth). + // It IS in this predicate's set, because the question here is "does the + // subtree collapse to a scalar" and a count does -- see + // `ltm_agg::reducer_collapses_to_scalar`, whose OTHER consumer + // (`expr_is_array_slice_valued`, the GH #743 freezability test) reads the + // same predicate and is what makes this cell INERT: an equation whose only + // reducer is SIZE is freezable as `PREVIOUS(size(...))`, so its + // changed-first partial succeeds and the changed-last chooser never + // reaches this gate. The membership is pinned anyway, because the + // inertness is a property of the two call sites AGREEING, and nothing else + // would notice one of them moving (GH #982). + assert!(references_bare_source_inside_reducer( + &parse("SIZE(frac)"), + &frac, + false + )); // A different variable inside the reducer is irrelevant. assert!(!references_bare_source_inside_reducer( &parse("SUM(matrix[D1, *] * other)"), diff --git a/src/simlin-engine/src/ltm_classifier_agreement_tests.rs b/src/simlin-engine/src/ltm_classifier_agreement_tests.rs index 2550c4081..ab30c92cd 100644 --- a/src/simlin-engine/src/ltm_classifier_agreement_tests.rs +++ b/src/simlin-engine/src/ltm_classifier_agreement_tests.rs @@ -195,6 +195,28 @@ fn expr0_routes_through_agg(name: &str, arity: usize) -> bool { ) } +/// The equivalence [`expr0_routes_through_agg`]'s doc asserts, as a test. +/// +/// This walk marks `in_reducer` from a NAME, so it cannot call +/// `builtin_routes_through_agg` (which needs a `BuiltinFn`) and restates the +/// rule instead. A restatement that drifts would not fail the gate loudly -- +/// it would quietly change WHICH occurrences the gate compares, weakening it +/// rather than breaking it. So the two are checked against each other over the +/// shared decision table (GH #982), whose 11 rows reach every arm of +/// `reducer_kind_from_name`. +#[test] +fn expr0_reducer_routing_twin_matches_production() { + for row in crate::ltm_agg::REDUCER_DECISION_TABLE { + assert_eq!( + expr0_routes_through_agg(row.name, row.arity), + row.routes_through_agg, + "{}/{}: the Expr0 twin must agree with builtin_routes_through_agg", + row.name, + row.arity + ); + } +} + /// Walk a reparsed `Expr0` tree left-to-right depth-first, recording every /// occurrence of a model variable (a head `Var`/`Subscript` whose ident is a /// key of `variables`), mirroring the structure of the Expr2 production walker From 4a27663d350da25b4352b7ce1514c059c3689d0f Mon Sep 17 00:00:00 2001 From: Bobby Powers Date: Mon, 27 Jul 2026 18:14:35 -0700 Subject: [PATCH 06/17] engine: compare AST float literals by bit pattern Expr0..Expr3's `Const` held a bare `f64`, so any value containing a NaN literal was not equal to ITSELF. Salsa decides whether to backdate a re-executed query's memo by comparing it with its own rebuild -- the `salsa::Update` derive walks fields and falls back to `PartialEq` for a field type with no `Update` impl -- so such a memo never backdated and the whole downstream query cone re-ran on every revision bump. Every stdlib SMOOTH/DELAY/TREND template declares `initial_value = NAN` and `db::sync` splices those into every project, so this needed no user action; a user model with a `nan` upstream of a cached stage paid it per keystroke on the interactive diagnostics path. The literal is now an `ast::Literal`, a Copy newtype comparing and hashing `f64::to_bits`. It has no `salsa::Update` impl on purpose: the derive's dispatch resolves to salsa's `PartialEq` fallback, so equality is defined in exactly one place and the crate gains no new `unsafe`. What defends that is `#![deny(unsafe_code)]`, which stops an `Update` impl being added at all, plus `-D warnings` on the now-unused `UpdateFallback` import if someone opts out -- an `Update` impl would otherwise take over silently through the inherent dispatch candidate. The four AST enums now also derive `Eq`, which is what keeps the fix structural: a future float-bearing variant cannot be a bare `f64`, because `Eq` rejects one at compile time, transitively through intermediate types. Bit comparison costs this layer nothing, because both distinctions it draws are unreachable in an AST literal rather than accepted. There is one `nan` spelling at the language surface yielding one canonical `f64::NAN`, so no payload is authorable; and the lexer takes no leading sign, so `-0` is a negation of the literal `0` and never a `Const`. Both premises are tripwire tests, and the only production code comparing two ASTs structurally (`mdl::writer`) reads the value out and compares with its own tolerance. `float.rs` gains the reason that is the right posture rather than a shortcut, next to the `:NA:` sentinel it contrasts with: a NaN in a system dynamics model is a diagnostic, not a value. What a practitioner sees is a line on a graph that stops, and their next task is tracing it backward through the dependency graph, because NaN is absorbing and every variable downstream shows the same symptom. So attributing a NaN to its origin is worth a lot, a NaN the engine manufactures is noise in a channel debugged by hand, and nothing needs machinery treating NaN as a structured value. `ast::Literal` carries the caching half: bit comparison wherever a float feeds a cache key, and the types still outside it -- the compiled bytecode pools and `variable::Table`'s lookup points -- keep IEEE comparison as an accepted cost of not converting them, with #642's false "only consumer is non-tracked" premise corrected there so it is not re-litigated. Two tests that had to work around the defect now do not: `db::stages_tests`' stdlib oracle uses the representative `smth1` rather than `npv`, and its whole-project oracle covers all twelve synced models instead of the three user ones. `ltm_agg`'s characterization test of the broken behaviour is inverted rather than deleted. --- docs/design/ltm--loops-that-matter.md | 2 +- src/simlin-engine/CLAUDE.md | 8 +- src/simlin-engine/src/ast/array_view.rs | 4 +- src/simlin-engine/src/ast/expr0.rs | 60 +- src/simlin-engine/src/ast/expr1.rs | 17 +- src/simlin-engine/src/ast/expr2.rs | 51 +- src/simlin-engine/src/ast/expr3.rs | 111 ++- src/simlin-engine/src/ast/literal.rs | 408 ++++++++ src/simlin-engine/src/ast/mod.rs | 74 +- src/simlin-engine/src/builtins_visitor.rs | 9 +- src/simlin-engine/src/bytecode.rs | 16 + src/simlin-engine/src/compiler/context.rs | 32 +- src/simlin-engine/src/compiler/mod.rs | 14 +- src/simlin-engine/src/compiler/subscript.rs | 4 +- src/simlin-engine/src/compiler/symbolic.rs | 14 + src/simlin-engine/src/conveyor_compile.rs | 7 +- src/simlin-engine/src/db/assemble.rs | 12 +- src/simlin-engine/src/db/fragment_compile.rs | 6 +- src/simlin-engine/src/db/ltm/compile.rs | 10 +- src/simlin-engine/src/db/ltm/equation.rs | 75 ++ src/simlin-engine/src/db/stages_tests.rs | 187 +++- src/simlin-engine/src/db/tests.rs | 8 +- src/simlin-engine/src/db/var_fragment.rs | 6 +- src/simlin-engine/src/float.rs | 59 +- src/simlin-engine/src/ltm/polarity.rs | 7 +- src/simlin-engine/src/ltm/polarity_tests.rs | 971 +++++++++++++++++++ src/simlin-engine/src/ltm/tests.rs | 962 ++---------------- src/simlin-engine/src/ltm_agg.rs | 104 +- src/simlin-engine/src/ltm_augment_tests.rs | 36 +- src/simlin-engine/src/mdl/writer.rs | 6 +- src/simlin-engine/src/mdl/writer_proptest.rs | 30 +- src/simlin-engine/src/parser/mod.rs | 10 +- src/simlin-engine/src/parser/tests.rs | 180 +++- src/simlin-engine/src/patch.rs | 6 +- src/simlin-engine/src/results.rs | 5 + src/simlin-engine/src/units.rs | 3 +- src/simlin-engine/src/variable.rs | 15 +- 37 files changed, 2352 insertions(+), 1177 deletions(-) create mode 100644 src/simlin-engine/src/ast/literal.rs create mode 100644 src/simlin-engine/src/ltm/polarity_tests.rs diff --git a/docs/design/ltm--loops-that-matter.md b/docs/design/ltm--loops-that-matter.md index 7202bacda..8da1d713a 100644 --- a/docs/design/ltm--loops-that-matter.md +++ b/docs/design/ltm--loops-that-matter.md @@ -1131,7 +1131,7 @@ LTM machinery (the Expr0-walk-time `is_array_reducer_name`, `classify_reducer`, the static-polarity `agg_reducer_is_monotone`) is a thin reader of it, so the "is this a reducer" / "what kind" answers can't drift apart. AST-identical subexpressions (keyed by canonical printed equation text, since `Expr2` is -not `Eq`) dedupe to one node. +not `Hash` and so cannot key a map directly) dedupe to one node. **Read slice and result dims.** Each `AggNode` carries a `read_slice: Vec` -- one `AxisRead ∈ {Pinned(elem), Iterated(dim), diff --git a/src/simlin-engine/CLAUDE.md b/src/simlin-engine/CLAUDE.md index a8213d09c..3dc56f97c 100644 --- a/src/simlin-engine/CLAUDE.md +++ b/src/simlin-engine/CLAUDE.md @@ -10,7 +10,7 @@ Equation text flows through these stages in order: 1. **`src/lexer/`** - Tokenizer for equation syntax. `KEYWORDS` is the equation language's reserved-word table (`if`/`then`/`else`/`not`/`mod`/`and`/`or`/`nan`; the units lexer shares it and differs only in also admitting `$` inside identifiers), and `identifierish` resolves a bare word against it BEFORE falling back to `Token::Ident`. `is_reserved_word` exposes that verdict so `ast::needs_quoting` can read this table rather than restate it -- without it a variable legally named `"if"` printed BARE and no longer re-parsed, so an unrelated `patch` rename persisted an unparseable equation (GH #976). 2. **`src/parser/`** - Recursive descent parser producing `Expr0` AST -3. **`src/ast/`** - AST type system with progressive lowering: `Expr0` (parsed) -> `Expr1` (modules expanded) -> `Expr2` (dimensions resolved) -> `Expr3` (subscripts expanded). `array_view.rs` tracks array dimensions and sparsity. `Expr2Context` trait includes `has_mapping_to()` for cross-dimension mapping lookups during `find_matching_dimension`. `needs_quoting` is the single "can this canonical name be spelled bare in an EQUATION" predicate -- character classes, the leading-character rule, and `lexer::is_reserved_word` -- and it has THREE consumers, one per producer of equation text: `print_ident` (the `print_eqn` path), `ltm_augment::quote_ident` (LTM's generated guard forms), and `mdl::xmile_compat::XmileFormatter::quote_reference` (the MDL importer, which writes equation text into `datamodel::Equation` that our own lexer reads back; it excludes `nan` alone, a disclosed residual explained in the MDL module doc). `print_eqn_proptest::a_bare_spellable_name_lexes_as_one_identifier` pins its completeness against the lexer itself, which is what makes the restatement checkable. The MDL *writer*'s `needs_mdl_quoting` is a DIFFERENT language's rule and deliberately separate -- as is `quoted_space_to_underbar`, which spells a definition-side *ident*, not equation text. A canonical name containing `"` is the one shape with no spelling at all (the lexer's quoted identifier has no escape); `patch::apply_rename_variable` refuses to rename TO one rather than persisting an equation that cannot be re-read. +3. **`src/ast/`** - AST type system with progressive lowering: `Expr0` (parsed) -> `Expr1` (modules expanded) -> `Expr2` (dimensions resolved) -> `Expr3` (subscripts expanded). `array_view.rs` tracks array dimensions and sparsity. Every layer's `Const` holds an `ast::Literal` (`literal.rs`) rather than a bare `f64`: a numeric literal is compared and hashed by BIT PATTERN, so `NaN == NaN` and a value holding one is equal to itself. Salsa backdates a re-executed query's memo by comparing old and new (`salsa::Update`'s derive falls back to `PartialEq` for a field type with no `Update` impl of its own, which is exactly what `Literal` relies on -- there is no hand-written `unsafe impl`), and every stdlib SMOOTH/DELAY/TREND template declares `initial_value = NAN`, so a bare `f64` left every project carrying stage values and parse memos that were unequal to themselves and could never backdate (GH #987/#981). The four AST enums derive `Eq` as well, which is what keeps the fix STRUCTURAL *within the AST*: a new float-bearing variant or field on any type the AST reaches must use `Literal`, because a bare `f64` fails to satisfy `Eq` and that is a compile error rather than a silently reintroduced incrementality cliff (enforced transitively -- an `f64` added to `ArrayView`, which `Expr3` reaches only indirectly, fails the same way). Bit comparison costs this layer NOTHING, because both distinctions it draws are unreachable in an AST literal by construction rather than accepted: there is one `nan` spelling yielding one canonical `f64::NAN` (a practitioner cannot author a payload), and `lexer::scan_number` takes no leading sign, so `-0` is a negation of the literal `0` and never a `Const`. Both premises are tripwire tests. `Literal`'s rustdoc carries the PROJECT'S POSITION ON FLOAT EQUALITY -- wherever a float feeds a cache key we want bit equality -- and `src/float.rs`'s module docs carry the domain reason it is the right posture rather than a shortcut. Two families keep the derived IEEE `PartialEq` as an ACCEPTED state, not an open defect -- what is accepted is the cost of NOT converting them, not a semantic property worth keeping: the compiled bytecode types (`ByteCode::literals`, `ByteCodeContext::graphical_functions`, `results::Specs`, and the symbolic `SymbolicByteCode::literals` / `PerVarBytecodes::graphical_functions` -- GH #642, closed on this reasoning), and `variable::Table`'s `x`/`y: Vec` + `GraphicalFunctionScale`s, which ride into the SAME three memos so a lookup table with a NaN y-point still defeats their backdating. Each type carries a pointer to the position; the argument lives only on `Literal`. Two things recorded there rather than rediscovered: #642's own "the only consumer is non-tracked" premise is FALSE one level down (`PerVarBytecodes` is the value of the tracked `compile_var_fragment`, read by the tracked `assemble_module`), so the missed backdate is real and knowingly accepted; and IEEE equality is also LOOSER on signed zeros, so a pool differing only in a zero's sign backdates and the stale one is kept (unlike the AST, `-0.0` IS reachable in a compiled pool -- `compiler::fold` folds `0 * -1` to it) -- pre-existing, corpus-unreached, and not what #642 describes, but it points the same way: converting those types closes both directions and gives up nothing. No other AST-equality consumer is affected either way (the numeric comparisons in `mdl::writer::exprs_equal`, `ltm::polarity`'s sign tests, and the compiler's static index resolution all read the value out with `Literal::value` and are unaffected). `Expr2Context` trait includes `has_mapping_to()` for cross-dimension mapping lookups during `find_matching_dimension`. `needs_quoting` is the single "can this canonical name be spelled bare in an EQUATION" predicate -- character classes, the leading-character rule, and `lexer::is_reserved_word` -- and it has THREE consumers, one per producer of equation text: `print_ident` (the `print_eqn` path), `ltm_augment::quote_ident` (LTM's generated guard forms), and `mdl::xmile_compat::XmileFormatter::quote_reference` (the MDL importer, which writes equation text into `datamodel::Equation` that our own lexer reads back; it excludes `nan` alone, a disclosed residual explained in the MDL module doc). `print_eqn_proptest::a_bare_spellable_name_lexes_as_one_identifier` pins its completeness against the lexer itself, which is what makes the restatement checkable. The MDL *writer*'s `needs_mdl_quoting` is a DIFFERENT language's rule and deliberately separate -- as is `quoted_space_to_underbar`, which spells a definition-side *ident*, not equation text. A canonical name containing `"` is the one shape with no spelling at all (the lexer's quoted identifier has no escape); `patch::apply_rename_variable` refuses to rename TO one rather than persisting an equation that cannot be re-read. 4. **`src/builtins.rs`** - Builtin function definitions (e.g. `MIN`, `PULSE`, `LOOKUP`, `QUANTUM`, `SSHAPE`, `VECTOR SELECT`, `VECTOR ELM MAP`, `VECTOR SORT ORDER`, `VECTOR RANK`, `ALLOCATE AVAILABLE`, `ALLOCATE BY PRIORITY`, `NPV`, `MODULO`, `PREVIOUS`, `INIT`). `is_stdlib_module_function()` is the authoritative predicate for deciding whether a function name expands to a stdlib module; the **module-function** generalization that also resolves project macros lives in **`src/module_functions.rs`** -- the pure (Functional-Core) `ModuleFunctionDescriptor`/`MacroRegistry` resolver+validator unifying stdlib functions and macros (`stdlib_descriptor`, `MacroRegistry::build`/`resolve_macro`, the shared `is_renamed_opcode_intrinsic`/`is_renamed_stdlib_module_builtin`/`is_renamed_builtin_macro_collision` precedence predicates). `MacroRegistry::build` validates in four passes: duplicate macro name and macro/model name collision (macros.AC5.3), macro-to-macro recursion (macros.AC5.2), and -- Pass 4 -- a `Variable::Module` inside a macro-marked model (macros.AC5.7, `ErrorCode::MacroContainsModule`). Pass 4 is a CYCLE-SAFETY rule and is not redundant with Pass 3: `db::project_module_graph` records only EXPLICIT module edges, so a cycle `mac ->(explicit module)-> u ->(macro call, an IMPLICIT edge)-> mac` was invisible to the gate and drove the recursive queries into salsa's dependency-graph cycle panic, while Pass 3's macro-to-macro graph cannot even express the edge through the non-macro `u`. The rejection is deliberately broader than the cycle -- an acyclic module inside a macro is rejected too -- and that cost is REAL, not zero: the shape is reachable from an ordinary XMILE file (the `` content model is shared with ``, so the reader passes a `` through unfiltered), our own XMILE writer round-trips it, and an acyclic instance compiles and SIMULATES correctly today. It is rejected anyway because narrowing to only-when-cyclic needs a second reachability analysis that must agree with `project_module_graph`'s, whose back edge is the macro CALL -- discoverable only by parsing every model's equations, which is the dependency-list cost widening the graph was rejected for; and because a macro is a TEMPLATE, so instantiating a sub-model inside one is dubious on its own terms and reads as a language rule rather than a workaround. The MDL importer cannot produce it, but not because "Vensim macros have no modules": `src/mdl/convert/multi_output.rs` DOES mint a `Variable::Module` for a multi-output `:`-list invocation, and it is unreachable from a macro body only because the scoped body sub-context hard-codes an empty materialization (`src/mdl/convert/macros.rs`). A module *targeting* a macro (how multi-output invocations work) is untouched -- only a module *inside* a macro-marked model is rejected. A **genuine passthrough macro** -- a single-parameter, single-output macro whose body is exactly `out = BUILTIN(param)` self-calling its own renamed-builtin-collision name (`:MACRO: INIT(x) = INITIAL(x)`, stored after the importer's `INITIAL`->`INIT` rename as `init = init(x)`) -- is classified once at `MacroRegistry::build` time (`classify_passthrough`, the only place the body AST is available) into a `passthrough: Option` on the descriptor; `builtins_visitor.rs` reads it to *collapse the call to the builtin opcode* (`init`->`LoadInitial`) instead of expanding the buggy per-element synthetic module (#591), falling through to the same renamed-builtin intrinsic routing the #554 self-call exception takes. Strict criteria (no additional outputs, bare-parameter argument, self-call, renamed-builtin-collision name) keep it from misfiring on a near-miss like `INIT = INIT(x) + 1`. `equation_is_module_call()` (pre-scan, renamed from `equation_is_stdlib_call`) and `contains_module_call()` (walk-time, renamed from `contains_stdlib_call`) both consult the `MacroRegistry` (a passthrough caller is still classified module-backed here -- benign, since it collapses to a flat-slot variable). `builtins_visitor.rs` handles implicit module instantiation, single-output macro inlining (an arrayed macro invocation enters the per-element path), and PREVIOUS/INIT helper rewriting: unary `PREVIOUS(x)` desugars to `PREVIOUS(x, 0)`, direct scalar args compile to `LoadPrev`, and module-backed or expression args are first rewritten through synthesized scalar helper auxes. `INIT(x)` compiles to `LoadInitial`, using the same helper rewrite when needed. Tracks `module_idents` so `PREVIOUS(module_var)` never reads a multi-slot module directly. 5. **`src/compiler/`** - Multi-pass compilation to bytecode: - `mod.rs` - Orchestration; includes A2A hoisting logic that detects array-producing builtins (VectorElmMap, VectorSortOrder, Rank, AllocateAvailable, AllocateByPriority) during array expansion, hoists them into `AssignTemp` pre-computations, and emits per-element `TempArrayElement` reads. Treats a **standalone lookup-only variable** -- a graphical-function holder whose equation is empty or the legacy MDL `"0+0"` sentinel (`crate::variable::var_is_lookup_only`/`is_empty_or_sentinel`, mirroring `mdl::writer::is_lookup_only_equation`'s "empty or sentinel" rule) -- as a non-value-bearing **static table** (`Variable::Var::is_table_only`): it is excluded from every runlist and from the saved output, so it produces NO series of its own (issue #606). Its data is reached only through `LOOKUP(table, x)` call sites (resolved by ident -> `base_gf`); a *bare* reference with no argument is a compile error (`ErrorCode::LookupReferencedWithoutArgument`). WITH LOOKUP (`var = WITH LOOKUP(input, table)`: tables present *and* a real input) is NOT lookup-only -- it is a value-bearing variable that lowers to `LOOKUP(self, input)` for every equation shape (`apply_implicit_with_lookup`): per element for arrayed variables, where a per-element gf applies each element's OWN table and a gf-less element keeps its raw input equation (GH #909) @@ -105,7 +105,7 @@ The primary compilation path uses salsa tracked functions for fine-grained incre - **`src/db/input.rs`** (a `db` submodule) - The salsa INPUT layer: the interned key types (`LtmLinkId`/`ModuleIdentContext`/`ModuleInputSet`, the last with its `empty`/`from_canonical_set`/`from_names`/`canonical_input_set` round-trip), the `SourceVariableKind` tag, the three `#[salsa::input]` structs (`SourceProject`/`SourceModel`/`SourceVariable`) that store the synced datamodel field-by-field for fine-grained invalidation, the `PinnedLoopSpec` projection of a non-deleted `LoopMetadata` (a pinned loop's name + canonical variable set, carried on `SourceModel.pinned_loops`; the `LoopMetadata` UIDs are resolved to canonical names at sync time since UIDs never enter the db), the `source_var_is_table_only` lookup-only predicate, `SourceModel.declared_variable_idents` (the ordered, pre-dedup AS-WRITTEN variable-ident list -- the raw data the GH #885 duplicate-canonical-ident check needs, which the canonical-keyed `variables` map collapses; the per-variable analogue of `SourceProject.macro_declarations`), and `datamodel_variable_from_source` (re-assembles the kind-tagged `datamodel::Variable` for the parser). Re-exported at the `db.rs` root. - **`src/db/query.rs`** (a `db` submodule) - The demand-driven read queries over the inputs: per-variable parsing (`parse_source_variable_with_module_context` + `_impl`), the project-global cached contexts (`project_units_context_result` -- carrying the units `Context` AND the `definition_errors` building it produced, with `project_units_context` a salsa PROJECTION over its `ctx` -- not a plain accessor, since a reader of one would take a dependency on the whole result and rebuild whenever only the errors moved, which is every intermediate state of typing a malformed unit equation; the projection backdates on the equal `Context` and keeps readers' dependencies exactly as narrow as they were. The errors ride the RESULT rather than a salsa accumulator because a bad unit declaration is a project-level fact that an accumulator both reported once per model and lost entirely after an unrelated revision bump -- plus `project_datamodel_dims`, `project_dimensions_context`, `project_converted_dimensions`), the module-ident context (`model_module_ident_context`), the per-variable direct-dependency extraction (`variable_direct_dependencies` + `_impl`, the `VariableDeps` value, the `canonical_module_input_set` helper), the per-name variable lookup `model_variable_by_name` (a salsa FIREWALL over the `SourceModel::variables` map field: reading the map directly gives the reader a dependency on the whole variable SET, which is what made every per-variable fragment recompile when any variable was added, deleted or renamed; the query still reads the map but returns one handle, so it backdates and a fragment depends only on the dependencies it looks up), the implicit-variable metadata (`ImplicitVarMeta`/`model_implicit_var_info`), the recursive `model_module_map`, the explicit-edges-only module-reference graph (`ModuleReferenceGraph`/`project_module_graph`, the cycle gate every compile / diagnostic / analysis entry point consults first so a module cycle is a `CircularDependency` instead of salsa's dependency-graph cycle panic -- GH #806; its rustdoc carries the argument for why omitting the IMPLICIT edges is sound now that `MacroRegistry::build`'s Pass 4 rejects the one shape that could close a cycle through one, plus the enumerated-not-proved residual that residual rests on), and the dimension-read queries (`variable_relevant_dimensions`/`variable_dimensions`/`variable_size`). `variable_relevant_dimensions` gives dimension-granularity invalidation (scalars produce an empty set, so dimension changes never invalidate them). - **`src/db/stages.rs`** (a `db` submodule) - The two name-keyed, PRE-LAYOUT model-compilation stages as salsa-cached `returns(ref)` queries. This is the ONLY place in the crate they are built from salsa inputs. `Project::from_salsa` used to carry a second inline copy, silently disagreeing on three fields; this body took `from_salsa`'s behaviour in all three, so retiring that copy was a deletion rather than a merge, and `from_salsa` now clones these memos. Three other `ModelStage0`/`ModelStage1` constructions remain, none of them a whole-model salsa build and each deliberate: the per-variable MINI Stage0 in `db::var_fragment::lower_var_fragment` and `db::ltm::compile` (scoped to one variable's dependencies -- pointing them here would add a project-wide dependency edge to every fragment compile), and `db::units::check_conveyor_param_units`' throwaway augmented `ModelStage1::new` over a CLONE of the cached Stage0 (see that module's header). `model_stage0(db, model, project) -> ModelStage0` parses a model's variables (through the per-variable `parse_source_variable_with_module_context` memos) plus the implicit SMOOTH/DELAY/TREND helpers those parses synthesize; `model_stage1(db, model, project) -> ModelStage1` lowers that Stage0's equations to `Expr2` via `ModelStage1::new` against a `ScopeStage0` holding the project's dimension context and the Stage0 of every model in `model_scope_models(db, model, project)` -- this model plus the transitive closure of the models its module variables instantiate. Both were previously rebuilt from scratch by each consumer -- `db::units::check_model_units` is tracked PER MODEL yet rebuilt both stages for EVERY project model on each call, making whole-project unit diagnostics quadratic in the model count (GH #966); the unit pass now READS these. Two decisions live here rather than at a call site, both currently inert in the stage VALUE and therefore pinned directly by `stages_tests`: `model_is_stdlib` marks a model implicit only when the `stdlib⁚` prefix is followed by a name in `stdlib::MODEL_NAMES` (a bare-prefix model an import produced is a user model, not a template), and `source_model_is_stdlib` is the canonicalizing handle-level wrapper `db::units`' unit-check skip gate now shares (GH #988) -- that gate used to carry its own looser bare-prefix spelling, so an imported `stdlib⁚` model silently lost unit checking while the stage query staged it as an ordinary user model, and `extra_module_idents` adds EVERY variable name of a stdlib model to its `ModuleIdentContext` so `PREVIOUS(module_input)` inside a stdlib body would rewrite through a scalar helper aux (no shipped stdlib body calls PREVIOUS/INIT, so this changes no parse today -- it exists so every path reaching a stdlib model's parse hits the same interned context and shares one set of parse memos instead of minting a second). Stage0 records duplicate-canonical-ident model errors from the memoized `db::model_duplicate_variables` groups (GH #885/#891); nothing on the unit path reads that field. `model_scope_models` is that closure, as an iterative worklist rather than a recursive tracked query -- a module cycle is a project a user can draw, and recursing a tracked query on one is salsa's unrecoverable dependency-graph panic (GH #806). Its edges come from each model's Stage0 `variables`, so they include the IMPLICIT modules builtin and macro expansion synthesized: a macro call targets the macro's own model, an ordinary user model that can be arrayed, which is why `db::project_module_graph` (explicit edges only, deliberately, to stay parse-free) is the wrong source here. A module's own IDENT is an edge too when it names a project model, because `resolve_relative` predates a module ident differing from its target's name and keys each dotted `src` component as a MODEL name; dropping that edge turns a project that lowered cleanly into one reporting `BadModuleInputSrc`. `check_model_units` reads the SAME closure for its inference map (`units_infer` consults its models map only for module targets, recursing along the same edges), so the two halves cannot disagree about what is reachable. The narrowing is what makes an unrelated model's edit invalidate neither a model's lowered stage nor its unit check, pinned by the execution counters in both directions; the whole-project map cost that, and cost the reads too. `model_scope_stage0` resolves the closure to Stage0s and REPAIRS a missing self entry (a handle that outlived its place in the project's model map), because without it `get_dimensions` returns `None` for every reference in the model and every arrayed equation lowers as though it were scalar; the Stage1 map in `check_model_units` deliberately does NOT repair it, since that absence is its signal that the handle is not the project's model. Three fixtures in `stages_tests` are what can see a narrowing go too far, each catching a different mistake: `arrayed_module_project` (`main` reduces over an ARRAYED output of its DIRECT target `sub_a` -- catches a self-only scope; before it existed, emptying the scope map entirely left every test in the crate green), `chain_project` (`main -> sub_a -> sub_c` with `main` reducing over `sub_a.sub_c.out_by_region`, where `sub_c` is reachable only transitively -- the only thing that catches a "self + direct targets" scope, so the closure must be TRANSITIVE), and `omitting_stdlib_models_from_the_lowering_scope_is_inert_today` (dropping the `stdlib⁚*` models wholesale is harmless for LOWERING, and that test asserts the precise reason -- no stdlib template is arrayed or instantiates a module -- so it fails the moment that stops being true; its module half doubles as the tripwire for the stdlib-is-a-SINK premise of `MacroRegistry::build`'s Pass 4 closure argument, so if it ever reds, the abort-class consequence is that `project_module_graph` no longer sees every cycle; the closure does not take that shortcut, and since the inference map shares it, a `stdlib⁚*`-skipping closure is no longer inert end to end -- it reds `unit_checking_test::test_smth1_unit_mismatch_initial`). Test-only thread-local execution counters (`reset_query_executions`/`query_executions`, covering both stage queries AND `check_model_units`, so one reset covers every counter a test in this area reads) let `stages_tests` count query-body entries: pointer equality of a `returns(ref)` memo cannot prove a body did not run, since salsa backdates a re-executed query whose value compares equal without moving the memo. -- **`src/db/stages_tests.rs`** - Tests for the two stage queries: value oracles against the salsa-free `ModelStage0::new_in_project` (Stage0) and `ModelStage1::new` over a whole-project scope of those (Stage1), including `every_shape_project`, the combined fixture carrying every shape the deleted `Project::from_salsa` copy handled -- an implicit stdlib `SMTH1` expansion, an explicit user sub-model, a macro-marked model AND a caller of it, and duplicate canonical idents; the three Stage0 semantics decisions above; the GH #966 cost claim (a whole-project `collect_all_diagnostics` BUILDS each model's Stage0/Stage1 at most once, and only for the models something reaches -- in a fixture instantiating no stdlib template, the spliced stdlib set is never staged at all, which is the scope narrowing measured; re-reading every model's stages once per model, the pre-#966 access pattern, then rebuilds none) plus its companion `project_from_salsa_reads_the_cached_stages` (`from_salsa` demands each model's stages once through the queries and the unit pass afterwards rebuilds none -- the only evidence that distinguishes reading the memo from building an equal second copy, since the deleted build produced EQUAL values by design); and diagnostic reachability across the move (a unit warning still reaches both `check_model_units::accumulated` and `collect_all_diagnostics`, as does `project_macro_registry`'s `DuplicateMacroName`, which `check_model_units` now reaches ONLY two tracked queries deeper); and the scope narrowing itself -- the closure's shape (transitive, implicit/macro edges included, module-ident edge, finite on a cycle) and its INVALIDATION consequences in both directions, since a scope that is too wide only costs incrementality while one that is too narrow answers from a stale memo. Two of those tests demonstrate a CONSEQUENCE rather than a shape, because a name-list assertion cannot show that losing an edge would mis-lower or mis-diagnose anything. `dropping_a_macro_models_scope_edge_changes_the_lowered_value` is the macro-side twin of `arrayed_module_project`: `main` reduces over an ARRAYED macro output, so removing the macro model from the scope changes the lowered VALUE of exactly that caller, with a SCALAR-output control that changes nothing -- which pins the mechanism as dimension resolution through the macro edge rather than sensitivity to the map's contents. Its fixture guard reads the model's STAGE0 (not its scope) on purpose, so that a macro-edge-dropping narrowing reds it on the value rather than on a structural check. `a_two_hop_cross_module_unit_mismatch_is_still_reported` is the unit guard for the TRANSITIVE row of the matrix (the stdlib row already had `unit_checking_test::test_smth1_unit_mismatch_initial`): a widget/gadget contradiction that closes only after `units_infer` walks BOTH module hops, so a direct-targets-only closure drops the grandchild's constraints and the diagnostic silently disappears. Note what a scope mis-lowering can and cannot corrupt: `model_stage1`'s consumers are unit checking and the test-only `Project::from_salsa`, while simulation compiles from the per-variable fragment path's own mini Stage0s and never reads this stage. The two invalidation tests drive `sync_from_datamodel_incremental`, which REUSES the `SourceProject` handle: a value read through an older `SyncResult` after an edit is the POST-edit value, so every read is made against the sync current at that moment, and each has a control step (re-syncing the identical project must re-execute nothing) so a count is attributable to the edit. Every equality oracle here ranges over USER models only: `ModelStage0` derives `PartialEq`, it compares parsed `f64` constants, and every SMOOTH/DELAY/TREND template declares `initial_value = NAN`, so a stdlib stage carrying a NaN does not compare equal even to ITSELF (GH #987/#981). The one stdlib oracle uses `npv`, the single template with no NaN literal. +- **`src/db/stages_tests.rs`** - Tests for the two stage queries: value oracles against the salsa-free `ModelStage0::new_in_project` (Stage0) and `ModelStage1::new` over a whole-project scope of those (Stage1), including `every_shape_project`, the combined fixture carrying every shape the deleted `Project::from_salsa` copy handled -- an implicit stdlib `SMTH1` expansion, an explicit user sub-model, a macro-marked model AND a caller of it, and duplicate canonical idents; the three Stage0 semantics decisions above; the GH #966 cost claim (a whole-project `collect_all_diagnostics` BUILDS each model's Stage0/Stage1 at most once, and only for the models something reaches -- in a fixture instantiating no stdlib template, the spliced stdlib set is never staged at all, which is the scope narrowing measured; re-reading every model's stages once per model, the pre-#966 access pattern, then rebuilds none) plus its companion `project_from_salsa_reads_the_cached_stages` (`from_salsa` demands each model's stages once through the queries and the unit pass afterwards rebuilds none -- the only evidence that distinguishes reading the memo from building an equal second copy, since the deleted build produced EQUAL values by design); and diagnostic reachability across the move (a unit warning still reaches both `check_model_units::accumulated` and `collect_all_diagnostics`, as does `project_macro_registry`'s `DuplicateMacroName`, which `check_model_units` now reaches ONLY two tracked queries deeper); and the scope narrowing itself -- the closure's shape (transitive, implicit/macro edges included, module-ident edge, finite on a cycle) and its INVALIDATION consequences in both directions, since a scope that is too wide only costs incrementality while one that is too narrow answers from a stale memo. Two of those tests demonstrate a CONSEQUENCE rather than a shape, because a name-list assertion cannot show that losing an edge would mis-lower or mis-diagnose anything. `dropping_a_macro_models_scope_edge_changes_the_lowered_value` is the macro-side twin of `arrayed_module_project`: `main` reduces over an ARRAYED macro output, so removing the macro model from the scope changes the lowered VALUE of exactly that caller, with a SCALAR-output control that changes nothing -- which pins the mechanism as dimension resolution through the macro edge rather than sensitivity to the map's contents. Its fixture guard reads the model's STAGE0 (not its scope) on purpose, so that a macro-edge-dropping narrowing reds it on the value rather than on a structural check. `a_two_hop_cross_module_unit_mismatch_is_still_reported` is the unit guard for the TRANSITIVE row of the matrix (the stdlib row already had `unit_checking_test::test_smth1_unit_mismatch_initial`): a widget/gadget contradiction that closes only after `units_infer` walks BOTH module hops, so a direct-targets-only closure drops the grandchild's constraints and the diagnostic silently disappears. Note what a scope mis-lowering can and cannot corrupt: `model_stage1`'s consumers are unit checking and the test-only `Project::from_salsa`, while simulation compiles from the per-variable fragment path's own mini Stage0s and never reads this stage. The two invalidation tests drive `sync_from_datamodel_incremental`, which REUSES the `SourceProject` handle: a value read through an older `SyncResult` after an edit is the POST-edit value, so every read is made against the sync current at that moment, and each has a control step (re-syncing the identical project must re-execute nothing) so a count is attributable to the edit. Every equality oracle here ranges over ALL the sync's models, stdlib templates included, and the one stdlib-only oracle uses the representative `smth1`. Both were restricted before GH #987/#981: `ModelStage0` derives `PartialEq` and compares parsed float constants, and every SMOOTH/DELAY/TREND template declares `initial_value = NAN`, so with a bare `f64` a stdlib stage did not compare equal even to ITSELF -- which is why the oracle covered user models only and the stdlib oracle had to use `npv`, the single template with no NaN literal. `Expr2::Const` now holds an `ast::Literal` compared by bit pattern, so those stages are ordinary values. - **`src/db/sync.rs`** (a `db` submodule) - Datamodel -> salsa-input sync: the `SyncResult`/`SyncedModel`/`SyncedVariable` handle maps, the `Clone`-able `PersistentSyncState`/`PersistentModelState`/`PersistentVariableState` snapshots threaded between sync calls (with `to_sync_result`/`to_synced_model`/`from_sync_result`), the one-shot stdlib-input builder (`build_stdlib_models`, feeding the root's `StdlibModels` cache), the fresh (`sync_from_datamodel`) and incremental (`sync_from_datamodel_incremental`) entry points + their per-variable helpers (`source_variable_from_datamodel`, `update_source_variable`), the `macro_declarations_from_datamodel` extractor, `pinned_loops_from_datamodel` (resolves a model's `loop_metadata` UIDs to canonical variable names for `SourceModel.pinned_loops`), and `expand_maps_to_chains` (the `maps_to`/`mappings` reachability closure the parser uses to size a variable's dimension dependency). - **`src/db/diagnostic.rs`** (a `db` submodule) - Compilation diagnostics: the `CompilationDiagnostic` salsa accumulator, the typed `Diagnostic` value (`severity` field + `DiagnosticError` variants `Equation`/`Model`/`Unit`/`Assembly`), the per-model `model_all_diagnostics` query that triggers every diagnostic source (`compile_var_fragment` per variable, the unit-check pass, and -- when `ltm_enabled` -- `model_ltm_fragment_diagnostics`), the `model_duplicate_variables` query + `emit_duplicate_variable_diagnostics` (GH #885: an Error-severity `DuplicateVariable` diagnostic per group of variables whose names canonicalize to the same ident, derived from the raw `declared_variable_idents` input; `compile_project_incremental` fails hard on the same query, and `queue_compile::build_compiled` runs the identical datamodel-level check before expansion), and the drain helpers `collect_model_diagnostics` (one model) / `collect_all_diagnostics` (whole synced project). Three diagnostics deliberately bypass the accumulator: `collect_all_diagnostics` emits each directly from a memoized derivation, and their row counts DIFFER. The **macro-registry build error** (`project_macro_registry`) is at most one row per project, naming no model or variable. The **unit-definition errors** (`project_units_context_result`) are one row per declaration that failed to parse -- several are normal -- naming no model but carrying the unit's name as `variable`. The **module-reference cycle** (`project_module_graph`) is one row per MODEL that can reach a cycle, carrying that model's name and skipping its per-model passes; that one is deliberately per-model and must stay so, since a model reaching no cycle is still processed normally and an unrelated draft cycle cannot hide a valid model's diagnostics (GH #806) -- collapsing it to a single project-level row would revert that. It is emitted by `module_cycle_diagnostic`, which `collect_model_diagnostics` consults and `collect_all_diagnostics` inherits by delegating to it. That gate used to be inline in the whole-project loop only, so the per-model entry point -- equally `pub` -- drove a cyclic model's passes into `model_module_map` and salsa's dependency-graph cycle panic; one function for the gate is what keeps the two entry points from disagreeing about whether the same project panics. What the first two share is their ORIGIN, not their count: each is derived once per project and each used to be accumulated from inside its deriving query's body, which both over-reported (every model's subtree reaches the query, so the DFS found the one value once per model) and lost the diagnostic entirely once an unrelated revision bump let the DFS prune the subtree. - **`src/db/layout.rs`** (a `db` submodule) - The salsa-tracked per-model *body* layout query `compute_layout` (offsets from 0; the root's `IMPLICIT_VAR_COUNT` shift is applied later at assembly via `VariableLayout::root_shifted`). Lays out explicit variables, implicit (SMOOTH/DELAY/TREND) helpers, and -- when `ltm_enabled` -- the LTM synthetic variables and their implicit helpers. @@ -181,7 +181,7 @@ The unit subsystem is partial-result throughout: a single bad declaration or one - `graph.rs` - public `CausalGraph` type owning model adjacency, stock identity, an optional variable AST map for polarity, and optional recursively-built sub-graphs for referenced sub-models. `CausalGraph::from_model` populates both for **every** referenced sub-model (DynamicModule and stockless passthrough alike, since GH #698 -- the passthrough's sub-graph drives the discovery-mode per-exit-port pathway recompute; a pathless module's sub-graph enumerates no pathways and is harmless). The lightweight edge-only `db::analysis` constructors (`causal_graph_from_edges` / `causal_graph_from_element_edges`) leave both EMPTY; the production discovery path enriches the element-level graph via `causal_graph_from_element_edges_with_modules` (sharing `model_variables_and_module_graphs` with `causal_graph_with_modules`). The `variables()` / `module_graph()` accessors expose the variable map and a module instance's sub-graph for that recompute. Drives `find_loops_with_limit` / `compute_cycle_partitions` / `all_link_polarities` / `enumerate_pathways_to_outputs`. `Loop` struct carries `id`, `links`, `stocks`, `polarity`, and `dimensions` (non-empty for A2A loops where per-element evaluation is needed; empty for scalar or mixed loops). - `tests.rs` - integration-style tests spanning all submodules; preserved as a single file to avoid splitting shared test fixtures. - **`src/ltm_finding.rs`** - Strongest-path loop discovery algorithm (Eberlein & Schoenberg 2020). Post-processes simulation results containing link score synthetic variables to find the most important loops at each timestep. `discover_loops_with_graph(results, causal_graph, stocks, ltm_vars, dims, expansion, sub_model_output_ports, budget)` is the primary entry point: `ltm_vars` and `dims` enable A2A link score expansion (per-element edges from `parse_link_offsets`); when empty, all link scores are treated as scalar. A Bare A2A link score is dimensioned over the TARGET's dims, so `expand_a2a_link_offsets` subscripts the TARGET node per element but PROJECTS the SOURCE node onto `from`'s OWN declared dims -- bare for a scalar feeder (`scale → growth`, GH #790), the same-element diagonal for an equal-dim feeder, the broadcast/partial-collapse form for a lower-dim feeder, and the positionally-mapped diagonal for a `State→Region` pair (GH #527) -- by reusing the element graph's own `db::expand_same_element` rule, so the discovery search graph's node names match `model_element_causal_edges` node-for-node (GH #754; before this, subscripting BOTH endpoints with the score's full tuple minted phantom from-nodes like `scale[a]`/`boost[r,a]`/`x[s]` that named no real node, so every loop through such a feeder dangled and was silently undiscoverable). The from-var declared dims + dimension-mapping context ride in a `LinkExpansionContext` (`declared_dims` + `dim_ctx`) that `analyze_model` builds via the public `analysis::build_link_expansion_context` (the SAME `variable_dimensions` / `project_dimensions_context` queries the element graph reads); the db-less `discover_loops(&Results, &Project)` convenience path passes `LinkExpansionContext::default()` (no A2A expansion runs there). Element-mapped (non-positional) pairs never reach the projection: `db::ltm::link_score_dimensions` declines to retarget them (no Bare A2A score; the GH #758 loud skip fires instead) under the GH #756 positional-only gate. `SearchGraph` provides DFS-based strongest-path traversal from stock nodes. The post-simulation score recompute (path → `FoundLoop`) applies the **per-exit-port pathway selection** (`recompute_module_input_edge_series`, GH #698): a loop edge `x → m` into a module is recomputed by max-abs-selecting over the sub-model's `m·$⁚ltm⁚path⁚{entry}⁚{idx}` pathway scores that end at the exit port the loop reads (recovered from the next link `m → y`), instead of reading the module *composite* (which max-abs-selects across ALL ports and so can pick a wrong-signed port for a multi-output module -- flipping the loop polarity vs exhaustive). The pathway indices come from the module's sub-graph via the same `enumerate_pathways_to_outputs_with_truncation` over the same sorted output-port set the sub-model emitted against, so they match index-for-index (the discovery-mode equivalent of exhaustive's `compute_module_link_overrides`). That port set is NOT re-derived parent-scoped (which would shift indices when another project model reads an extra output port -- GH #698 / PR #705 r3353097150): `discover_loops_with_graph` takes a `SubModelOutputPorts` map (sub-model canonical name -> emitted sorted port set) that `analyze_model` builds from the SAME emission decision via `analysis::build_sub_model_output_ports` -> `db::ltm::sub_model_output_ports` (identity-by-construction); the db-less `discover_loops(&Results, &Project)` convenience path reconstructs the same project-wide-union + stdlib-`output` semantics in `project_sub_model_output_ports`. Falls back to the base composite offset for single-output / pathless / indeterminate-port / sub-model-absent-from-map edges. Because the discovery graph is element-level, the recompute `strip_subscript`s `link.from`/`link.to`/`next.from`/`next.to` before its name matches (arrayed loop nodes carry `[elem]`; mirrors the exhaustive twin's stripping -- PR #705 r3353758167); this is currently latent parity defense, since an arrayed loop through a multi-output module is not yet discoverable end-to-end (a scalar module output feeding an arrayed reader scores a constant-0 link, dropping the loop -- GH #716). Returns a `DiscoveryResult` whose `FoundLoop`s carry per-timestep link/loop/pathway scores, ranked **competitive-first** (`rank_and_filter`): loops sharing their cycle partition with at least one other discovered loop come first, ordered by mean partition-relative importance; loops trivially ALONE in their partition (relative score exactly +/-1 by construction -- zero information, e.g. C-LEARN's isolated gas-uptake decay loops) sort after all competing loops and are dropped first under the `MAX_LOOPS` cap. Each `FoundLoop` carries a result-scoped dense `partition` index into `DiscoveryResult::partitions` (`DiscoveredPartition { stocks, loop_count }`, first-appearance order; NOT stable across runs/edits -- key on the stock set for durable identity), threaded through `analysis::ModelAnalysis::partitions` / `LoopSummary::partition`, the FFI `SimlinDiscoveredPartition` / `SimlinDiscoveredLoop.partition`, and pysimlin `Analysis.partitions` / `Loop.partition`. Also hosts the link-set synthetic-node collapse: `trim_synthetic_aggs_from_loop_links` collapses `$⁚ltm⁚agg⁚{n}` nodes out of a single loop's link *cycle*, and the public `collapse_synthetic_links(Vec)` generalizes that to ALL synthetic/macro/module-internal nodes (`ltm::is_synthetic_node_name`) over an arbitrary link *set* -- each chain `X -> $⁚…internal… -> Y` collapses to one composite edge `X -> Y` with product polarity and the per-timestep max-magnitude path score (the composite link score, LTM ref 6.3/6.4); a purely-internal cycle/source is dropped. `CollapsibleLink { from, to, polarity, score: Option> }` is the abstract shape, so structural-only callers (`score = None`) and LTM-run callers share one impl. libsimlin's `simlin_analyze_get_links(.., include_internal=false)` collapses the `get_links()` set through this; `include_internal=true` returns the raw graph. The `#[cfg(test)]` tests live in the sibling `src/ltm_finding_tests.rs` (mounted via `#[cfg(test)] #[path]`, split out for the per-file line cap). -- **`src/ltm_agg.rs`** - Aggregate-node enumeration for LTM. `enumerate_agg_nodes` (salsa-tracked) walks every variable's `Expr2` AST left-to-right depth-first and identifies each maximal array-reducer subexpression (`SUM`/`MEAN`/`MIN`/`MAX`/`STDDEV`/`RANK`/`SIZE`); AST-identical subexpressions (keyed by canonical printed equation text) dedupe to one `AggNode`. Two kinds: **synthetic** (`is_synthetic == true`, the reducer is a sub-expression of a larger equation -- a `$⁚ltm⁚agg⁚{n}` aux is minted) and **variable-backed** (`is_synthetic == false`, the reducer is the entire dt-equation of a scalar/A2A variable like `total_population = SUM(pop[*])` -- the variable itself is the agg; EXCEPT a whole-RHS reducer whose shape the variable-backed machinery cannot express -- a MAPPED iterated axis (GH #534: the name-based link-score path cannot remap, so its `Wildcard` partial would silently stub to 0) or NON-ALIGNED result dims (GH #764 / shape-expressiveness T4: a broadcast over extra owner dims `out[D1,D3] = SUM(matrix[D1,*])` or permuted axes `out[D2,D1] = SUM(cube[D1,D2,*])`, where a per-`(row, slot)` slot cannot name a complete owner element) -- which mints a synthetic agg instead (`variable_backed_shape_is_expressible`, the ONE minting condition), riding the two-half emitters + the GH #528 projection). Each `AggNode` carries a `sources: Vec` -- one entry per source variable, SORTED by canonical name and deduped (salsa cache-equality + emission-order determinism; T2 of the shape-expressiveness design), each with its OWN `read_slice: Vec` (one `AxisRead ∈ {Pinned(elem), Iterated{dim, source_dim}, Reduced{subset}}` per THAT source's axes -- which rows of it the reducer actually reads; a scalar feeder like `scale` in `SUM(pop[*] * scale)` carries an empty slice; `Iterated` carries the (target, source) canonical dim pair, equal for the literal case; `Reduced.subset` is `None` for the full extent or the proper-subdimension element subset for a `SUM(arr[*:Sub])` StarRange, GH #766, decided per axis by `classify_axis_access` -- the single per-axis classifier of the shape-expressiveness design) -- and a `result_dims` (the `Iterated` axes' TARGET dims, datamodel-cased -- empty for a whole-extent or pinned-slice reduce). Under the I1 acceptance (`accept_source_slices`, T5 of the shape-expressiveness design / GH #767) every arrayed CO-SOURCE (`Reduced`-bearing slice) carries the identical *canonical* slice (`AggNode::canonical_read_slice` -- the first `Reduced`-bearing source slice, falling back to the first non-empty for the degenerate no-co-source agg), while an ITERATED-DIM PROJECTION FEEDER -- a source whose slice is all-`Iterated` over exactly the canonical slice's iterated target dims, in order, unmapped (`frac[D1]` in `SUM(matrix[D1,*] * frac[D1])`, per-result-slot constant) -- is accepted with ITS OWN slice (`AggNode::source_is_projection_feeder` is the discriminator); per-source consumers (`emit_agg_routed_edges`, `emit_source_to_agg_link_scores`) read `AggNode::source_read_slice(from)`, and name-keyed consumers use `AggNode::reads_var`. A feeder's link-score half is the per-`(row, slot)` CHANGED-LAST equation (`ltm_augment::generate_iterated_feeder_to_agg_equation`, emitted via `link_scores::iterated_feeder_row_scores` from both the synthetic source-half and `try_cross_dimensional_link_scores`' variable-backed feeder branch): the reducer text pinned to the slot with only the feeder frozen -- the arrayed generalization of the GH #737 scalar-feeder convention, exactly complementary per slot to the co-source rows' changed-first numerators for a bilinear body; the co-source rows' changed-first body partial pins the mismatched-arity feeder dep BY DIM NAME (`pin_body_to_row`'s GH #767 extension) so `PREVIOUS(frac[d1·r])` is held frozen at the row instead of bailing to the delta-ratio fallback. `compute_read_slice` decides hoistability: a whole-extent reduce (`SUM(pop[*])` ⇒ all-`Reduced`), a sliced one (`SUM(pop[NYC,*])` ⇒ `[Pinned(nyc), Reduced]`, `SUM(matrix[D1,*])` over an A2A-`D1` body ⇒ `[Iterated(d1,d1), Reduced]` → an arrayed agg over `D1`), a mixed one (`SUM(matrix3d[D1,NYC,*])` over an A2A-`D1` body ⇒ `[Iterated(d1,d1), Pinned(nyc), Reduced]`), or a positionally-MAPPED sliced one (GH #534: `SUM(matrix[State,*])` over `matrix[Region,D2]` with a positional `State→Region` mapping ⇒ `[Iterated{state, region}, Reduced]`, `result_dims = [State]` -- the agg is arrayed over the TARGET's iterated dim and the three `Iterated`-axis consumers remap each source row to the slot of its positionally-corresponding target element via `iterated_axis_slot_elements`, the preimage inversion of `mapped_element_correspondence`, so the positional-only/element-map gate is inherited) is hoisted; the carve-outs are (a) a reducer over a *dynamic index* (`SUM(pop[idx,*])`, `idx` non-literal ⇒ not statically describable ⇒ not hoisted, reference stays on the conservative path -- `db::ltm_ir` reclassifies it as `DynamicIndex`), (b) an ELEMENT-mapped sliced reducer the correspondence declines (execution resolves positionally and ignores the map, GH #756; a POSITIONAL mapping is accepted in either declaration direction since GH #757) ⇒ `compute_read_slice` returns `None`, conservative -- (c) a multi-source slice combination outside the I1 acceptance -- co-sources with differing slices, one variable read with two different slices (I3b), or a no-`Reduced` source that is not the pure iterated projection (a Pinned-axis mix like `SUM(matrix[D1,*] * w[D1, c1])`, a dim-subset/permuted feeder, or any mapped Iterated axis in a feeder combination) -- `combined_read_slice`/`accept_source_slices` return `None`, (d) a StarRange naming a NON-subdimension of its axis (a mid-edit inconsistency; declined rather than silently widened to the full extent), and (e) `RANK` (GH #771: array-valued, so `reducer_is_hoistable` requires `reducer_collapses_to_scalar` and RANK references stay `Direct` -- a bare arg classifies `Bare`, scored by the GH #742 arrayed-capture path; loops through the rank ORDERING are a documented residual). A multi-source reducer whose arrayed args *agree* (`SUM(a[*] + b[*])`, `a`, `b` over the same dim) mints one agg with one `AggSource` per variable, each carrying the shared canonical slice. `AggNodesResult` exposes `aggs` (first-encounter order), `agg_for_key` (by canonical reducer text), and `aggs_in_var` (which aggs occur in a variable's equation, so the element-graph reroute can ask "which agg of `to` reads `from`?"). A variable-backed reduce with a NON-TRIVIAL statically-describable slice is gated by `variable_backed_reduce_agg` (GH #752, generalized by GH #765 / T3 of the shape-expressiveness design) -- the SAME gate `model_element_causal_edges`' dispatch, `build_element_level_loops`' per-circuit routing, and `try_cross_dimensional_link_scores`' row derivation consume, so edges, loop routing, and scores always cover the identical read rows (all three derive from `read_slice_rows`, invariant I4). Accepted: an ALIGNED partial reduce (`row_sum[D1] = SUM(matrix[D1,*])`, `result_dims` equal to the variable's own dims in order -- Pinned-mixed `outf[D1] = MEAN(cube[D1,x,*])` and subset `out[D1] = MEAN(matrix[D1,*:Sub])` slices included: the divisor is the true read count and unread rows get neither edges nor scores) gets read-slice element edges straight onto the variable's element nodes and per-circuit scalar loops whose both-subscripted `matrix[d1,d2]→row_sum[d1]` links resolve the per-`(row, slot)` scores; a scalar-result Pinned/subset slice on a SCALAR owner (`total = SUM(pop[nyc,*])`, `total = SUM(arr[*:Sub])`) routes the read rows into the bare `to` node with matching per-read-row scores; and the ARRAYED-owner scalar-result Pinned/subset BROADCAST slice (`share[Region] = SUM(pop[nyc,*])`, no `Iterated` axis -- GH #777) fans each read row across the FULL target element set (`emit_agg_routed_edges`' broadcast arm emits `pop[nyc,d2] → share[e]` for every `e`; `emit_broadcast_reduce_link_scores` emits the matching per-(read-row, full-target-element) scalar scores `pop[nyc,d2]→share[e]`, the section-3 `PerElement` rule applied to a variable-backed reducer owner), with loop circuits routed to the per-circuit scalar path via `is_broadcast_reduce_edge` -- the read rows are independent of `to`'s dims, so the RELATED-dim (`share[Region]`) and DISJOINT-dim (`share[D9]`) spellings emit identically. Declined: a pure full-extent variable-backed agg keeps the normal reference walker's edges (the true reads for that shape, inert skip). The GH #764 broadcast/permuted result shapes never reach this gate since T4 -- they mint synthetic aggs at enumeration -- so the gate's Iterated-arm alignment check is defense-in-depth. `scalar_feeder_of_variable_backed_agg` (GH #790) is the scalar sibling of `source_is_projection_feeder`: it composes `variable_backed_reduce_agg` with an empty-slice + genuine-`Reduced`-canonical check to recognize a SCALAR FEEDER of a whole-RHS variable-backed reduce (`scale` in `growth[D1] = SUM(matrix[D1,*] * scale)`), so `try_scalar_to_arrayed_link_scores` can route it to the single Bare A2A changed-last feeder score instead of the uncompilable per-target-element partials. Two predicates here answer what LOOKS like one question -- "is this reference inside a reducer?" -- and their answers are inverted on exactly `SIZE` and `RANK`; the inversion is the DEFINITION of the difference, not a disagreement (GH #982, assessed and left as two predicates). `reducer_collapses_to_scalar` is about the reducer's RESULT TYPE (does the subtree fit in a scalar slot?) and is read by the two freeze/capture gates plus the GH #779 bare-reducer-feeder decline: `SIZE` is a count so it collapses, `RANK` is array-valued so it does not. `builtin_routes_through_agg` is about LTM ROUTING (did `enumerate_agg_nodes` mint a node for this call?) and sets `db::ltm_ir::OccurrenceSite::in_reducer`: `SIZE` is `Constant` and is never hoisted, `RANK` gets an array-valued agg. Both read the ONE `reducer_kind_from_name` table, and `builtin_routes_through_agg` is the disjunction of the enumerator's own two minting branches (`reducer_is_hoistable` and `array_valued_rank_arg`) rather than a restatement of them. The `#[cfg(test)]` `REDUCER_DECISION_TABLE` pins all three derived predicates row by row over every arm of the kind table -- including the agreement gate's name-keyed twin of the routing predicate -- so no cell can move silently. `is_synthetic_agg_name` / `synthetic_agg_name` are the `$⁚ltm⁚agg⁚{n}` name helpers. Each node also carries `reducer: BuiltinFn` -- the reducer call the enumerator classified when it decided the hoist, of which `equation_text` is the printed rendering (GH #983). It is what makes the link-score and polarity emitters parse-free: `ltm_augment::classify_reducer_in_builtin` reads the kind/name/body off it and `ltm::CausalGraph::source_to_agg_polarity` analyses it directly, where both used to print `equation_text`, re-parse it, re-lower it against a freshly built scope and re-derive the classification -- per (agg, source) pair, with both fallible steps returning early and silently zeroing the agg's loop score. It is stored in `Expr2::strip_loc_and_bounds` form, which removes TWO of the three ways this field can make the salsa-cached `AggNodesResult` compare unequal to an identical rebuild: `Loc` (load-bearing -- two AST-identical occurrences differ in byte offsets, so raw storage would make the dedup winner observable and would stop `enumerate_agg_nodes` backdating across an offset-only edit) and `ArrayBounds` (a guard -- inert today, since `reconstruct_model_variables` lowers against an empty model scope and allocates no bound). Neither reader looks at either. It does NOT remove the third and cannot: `Expr2::Const` holds an `f64`, and a derived `PartialEq` over one is not reflexive on NaN, so a model whose hoisted reducer contains a `nan` literal permanently defeats this query's backdating. That is the GH #987/#981 class -- the same mechanism that forces `db::stages_tests`' stdlib oracle onto `npv` -- fixed at the root by making AST float literals compare by bit pattern, and pinned here meanwhile by the characterization test `a_nan_literal_in_a_reducer_defeats_agg_backdating`, which that fix must invert. The stored builtin is read only for SYNTHETIC aggs (both readers filter to those); the variable-backed arm's copy is unread today, kept so `AggNode` has one shape. `AggNode`/`AggNodesResult` are consequently `PartialEq` but not `Eq` -- same `f64` -- and `Debug` only under `debug-derive`. +- **`src/ltm_agg.rs`** - Aggregate-node enumeration for LTM. `enumerate_agg_nodes` (salsa-tracked) walks every variable's `Expr2` AST left-to-right depth-first and identifies each maximal array-reducer subexpression (`SUM`/`MEAN`/`MIN`/`MAX`/`STDDEV`/`RANK`/`SIZE`); AST-identical subexpressions (keyed by canonical printed equation text) dedupe to one `AggNode`. Two kinds: **synthetic** (`is_synthetic == true`, the reducer is a sub-expression of a larger equation -- a `$⁚ltm⁚agg⁚{n}` aux is minted) and **variable-backed** (`is_synthetic == false`, the reducer is the entire dt-equation of a scalar/A2A variable like `total_population = SUM(pop[*])` -- the variable itself is the agg; EXCEPT a whole-RHS reducer whose shape the variable-backed machinery cannot express -- a MAPPED iterated axis (GH #534: the name-based link-score path cannot remap, so its `Wildcard` partial would silently stub to 0) or NON-ALIGNED result dims (GH #764 / shape-expressiveness T4: a broadcast over extra owner dims `out[D1,D3] = SUM(matrix[D1,*])` or permuted axes `out[D2,D1] = SUM(cube[D1,D2,*])`, where a per-`(row, slot)` slot cannot name a complete owner element) -- which mints a synthetic agg instead (`variable_backed_shape_is_expressible`, the ONE minting condition), riding the two-half emitters + the GH #528 projection). Each `AggNode` carries a `sources: Vec` -- one entry per source variable, SORTED by canonical name and deduped (salsa cache-equality + emission-order determinism; T2 of the shape-expressiveness design), each with its OWN `read_slice: Vec` (one `AxisRead ∈ {Pinned(elem), Iterated{dim, source_dim}, Reduced{subset}}` per THAT source's axes -- which rows of it the reducer actually reads; a scalar feeder like `scale` in `SUM(pop[*] * scale)` carries an empty slice; `Iterated` carries the (target, source) canonical dim pair, equal for the literal case; `Reduced.subset` is `None` for the full extent or the proper-subdimension element subset for a `SUM(arr[*:Sub])` StarRange, GH #766, decided per axis by `classify_axis_access` -- the single per-axis classifier of the shape-expressiveness design) -- and a `result_dims` (the `Iterated` axes' TARGET dims, datamodel-cased -- empty for a whole-extent or pinned-slice reduce). Under the I1 acceptance (`accept_source_slices`, T5 of the shape-expressiveness design / GH #767) every arrayed CO-SOURCE (`Reduced`-bearing slice) carries the identical *canonical* slice (`AggNode::canonical_read_slice` -- the first `Reduced`-bearing source slice, falling back to the first non-empty for the degenerate no-co-source agg), while an ITERATED-DIM PROJECTION FEEDER -- a source whose slice is all-`Iterated` over exactly the canonical slice's iterated target dims, in order, unmapped (`frac[D1]` in `SUM(matrix[D1,*] * frac[D1])`, per-result-slot constant) -- is accepted with ITS OWN slice (`AggNode::source_is_projection_feeder` is the discriminator); per-source consumers (`emit_agg_routed_edges`, `emit_source_to_agg_link_scores`) read `AggNode::source_read_slice(from)`, and name-keyed consumers use `AggNode::reads_var`. A feeder's link-score half is the per-`(row, slot)` CHANGED-LAST equation (`ltm_augment::generate_iterated_feeder_to_agg_equation`, emitted via `link_scores::iterated_feeder_row_scores` from both the synthetic source-half and `try_cross_dimensional_link_scores`' variable-backed feeder branch): the reducer text pinned to the slot with only the feeder frozen -- the arrayed generalization of the GH #737 scalar-feeder convention, exactly complementary per slot to the co-source rows' changed-first numerators for a bilinear body; the co-source rows' changed-first body partial pins the mismatched-arity feeder dep BY DIM NAME (`pin_body_to_row`'s GH #767 extension) so `PREVIOUS(frac[d1·r])` is held frozen at the row instead of bailing to the delta-ratio fallback. `compute_read_slice` decides hoistability: a whole-extent reduce (`SUM(pop[*])` ⇒ all-`Reduced`), a sliced one (`SUM(pop[NYC,*])` ⇒ `[Pinned(nyc), Reduced]`, `SUM(matrix[D1,*])` over an A2A-`D1` body ⇒ `[Iterated(d1,d1), Reduced]` → an arrayed agg over `D1`), a mixed one (`SUM(matrix3d[D1,NYC,*])` over an A2A-`D1` body ⇒ `[Iterated(d1,d1), Pinned(nyc), Reduced]`), or a positionally-MAPPED sliced one (GH #534: `SUM(matrix[State,*])` over `matrix[Region,D2]` with a positional `State→Region` mapping ⇒ `[Iterated{state, region}, Reduced]`, `result_dims = [State]` -- the agg is arrayed over the TARGET's iterated dim and the three `Iterated`-axis consumers remap each source row to the slot of its positionally-corresponding target element via `iterated_axis_slot_elements`, the preimage inversion of `mapped_element_correspondence`, so the positional-only/element-map gate is inherited) is hoisted; the carve-outs are (a) a reducer over a *dynamic index* (`SUM(pop[idx,*])`, `idx` non-literal ⇒ not statically describable ⇒ not hoisted, reference stays on the conservative path -- `db::ltm_ir` reclassifies it as `DynamicIndex`), (b) an ELEMENT-mapped sliced reducer the correspondence declines (execution resolves positionally and ignores the map, GH #756; a POSITIONAL mapping is accepted in either declaration direction since GH #757) ⇒ `compute_read_slice` returns `None`, conservative -- (c) a multi-source slice combination outside the I1 acceptance -- co-sources with differing slices, one variable read with two different slices (I3b), or a no-`Reduced` source that is not the pure iterated projection (a Pinned-axis mix like `SUM(matrix[D1,*] * w[D1, c1])`, a dim-subset/permuted feeder, or any mapped Iterated axis in a feeder combination) -- `combined_read_slice`/`accept_source_slices` return `None`, (d) a StarRange naming a NON-subdimension of its axis (a mid-edit inconsistency; declined rather than silently widened to the full extent), and (e) `RANK` (GH #771: array-valued, so `reducer_is_hoistable` requires `reducer_collapses_to_scalar` and RANK references stay `Direct` -- a bare arg classifies `Bare`, scored by the GH #742 arrayed-capture path; loops through the rank ORDERING are a documented residual). A multi-source reducer whose arrayed args *agree* (`SUM(a[*] + b[*])`, `a`, `b` over the same dim) mints one agg with one `AggSource` per variable, each carrying the shared canonical slice. `AggNodesResult` exposes `aggs` (first-encounter order), `agg_for_key` (by canonical reducer text), and `aggs_in_var` (which aggs occur in a variable's equation, so the element-graph reroute can ask "which agg of `to` reads `from`?"). A variable-backed reduce with a NON-TRIVIAL statically-describable slice is gated by `variable_backed_reduce_agg` (GH #752, generalized by GH #765 / T3 of the shape-expressiveness design) -- the SAME gate `model_element_causal_edges`' dispatch, `build_element_level_loops`' per-circuit routing, and `try_cross_dimensional_link_scores`' row derivation consume, so edges, loop routing, and scores always cover the identical read rows (all three derive from `read_slice_rows`, invariant I4). Accepted: an ALIGNED partial reduce (`row_sum[D1] = SUM(matrix[D1,*])`, `result_dims` equal to the variable's own dims in order -- Pinned-mixed `outf[D1] = MEAN(cube[D1,x,*])` and subset `out[D1] = MEAN(matrix[D1,*:Sub])` slices included: the divisor is the true read count and unread rows get neither edges nor scores) gets read-slice element edges straight onto the variable's element nodes and per-circuit scalar loops whose both-subscripted `matrix[d1,d2]→row_sum[d1]` links resolve the per-`(row, slot)` scores; a scalar-result Pinned/subset slice on a SCALAR owner (`total = SUM(pop[nyc,*])`, `total = SUM(arr[*:Sub])`) routes the read rows into the bare `to` node with matching per-read-row scores; and the ARRAYED-owner scalar-result Pinned/subset BROADCAST slice (`share[Region] = SUM(pop[nyc,*])`, no `Iterated` axis -- GH #777) fans each read row across the FULL target element set (`emit_agg_routed_edges`' broadcast arm emits `pop[nyc,d2] → share[e]` for every `e`; `emit_broadcast_reduce_link_scores` emits the matching per-(read-row, full-target-element) scalar scores `pop[nyc,d2]→share[e]`, the section-3 `PerElement` rule applied to a variable-backed reducer owner), with loop circuits routed to the per-circuit scalar path via `is_broadcast_reduce_edge` -- the read rows are independent of `to`'s dims, so the RELATED-dim (`share[Region]`) and DISJOINT-dim (`share[D9]`) spellings emit identically. Declined: a pure full-extent variable-backed agg keeps the normal reference walker's edges (the true reads for that shape, inert skip). The GH #764 broadcast/permuted result shapes never reach this gate since T4 -- they mint synthetic aggs at enumeration -- so the gate's Iterated-arm alignment check is defense-in-depth. `scalar_feeder_of_variable_backed_agg` (GH #790) is the scalar sibling of `source_is_projection_feeder`: it composes `variable_backed_reduce_agg` with an empty-slice + genuine-`Reduced`-canonical check to recognize a SCALAR FEEDER of a whole-RHS variable-backed reduce (`scale` in `growth[D1] = SUM(matrix[D1,*] * scale)`), so `try_scalar_to_arrayed_link_scores` can route it to the single Bare A2A changed-last feeder score instead of the uncompilable per-target-element partials. Two predicates here answer what LOOKS like one question -- "is this reference inside a reducer?" -- and their answers are inverted on exactly `SIZE` and `RANK`; the inversion is the DEFINITION of the difference, not a disagreement (GH #982, assessed and left as two predicates). `reducer_collapses_to_scalar` is about the reducer's RESULT TYPE (does the subtree fit in a scalar slot?) and is read by the two freeze/capture gates plus the GH #779 bare-reducer-feeder decline: `SIZE` is a count so it collapses, `RANK` is array-valued so it does not. `builtin_routes_through_agg` is about LTM ROUTING (did `enumerate_agg_nodes` mint a node for this call?) and sets `db::ltm_ir::OccurrenceSite::in_reducer`: `SIZE` is `Constant` and is never hoisted, `RANK` gets an array-valued agg. Both read the ONE `reducer_kind_from_name` table, and `builtin_routes_through_agg` is the disjunction of the enumerator's own two minting branches (`reducer_is_hoistable` and `array_valued_rank_arg`) rather than a restatement of them. The `#[cfg(test)]` `REDUCER_DECISION_TABLE` pins all three derived predicates row by row over every arm of the kind table -- including the agreement gate's name-keyed twin of the routing predicate -- so no cell can move silently. `is_synthetic_agg_name` / `synthetic_agg_name` are the `$⁚ltm⁚agg⁚{n}` name helpers. Each node also carries `reducer: BuiltinFn` -- the reducer call the enumerator classified when it decided the hoist, of which `equation_text` is the printed rendering (GH #983). It is what makes the link-score and polarity emitters parse-free: `ltm_augment::classify_reducer_in_builtin` reads the kind/name/body off it and `ltm::CausalGraph::source_to_agg_polarity` analyses it directly, where both used to print `equation_text`, re-parse it, re-lower it against a freshly built scope and re-derive the classification -- per (agg, source) pair, with both fallible steps returning early and silently zeroing the agg's loop score. It is stored in `Expr2::strip_loc_and_bounds` form, which removes two of the three ways this field could make the salsa-cached `AggNodesResult` compare unequal to an identical rebuild: `Loc` (load-bearing -- two AST-identical occurrences differ in byte offsets, so raw storage would make the dedup winner observable and would stop `enumerate_agg_nodes` backdating across an offset-only edit) and `ArrayBounds` (a guard -- inert today, since `reconstruct_model_variables` lowers against an empty model scope and allocates no bound). Neither reader looks at either. It cannot remove the third -- dropping a `nan` literal would change what the equation means -- so that one is closed at the ROOT instead: `Expr2::Const` holds an `ast::Literal`, compared by BIT PATTERN, so a model whose hoisted reducer contains a `nan` literal backdates like any other (GH #987/#981; with a bare `f64` it never could, since `NaN != NaN`). Pinned by `a_nan_literal_in_a_reducer_does_not_defeat_agg_backdating`. The stored builtin is read only for SYNTHETIC aggs (both readers filter to those); the variable-backed arm's copy is unread today, kept so `AggNode` has one shape. `AggNode`/`AggNodesResult` derive `Eq` (reflexivity is a compile-checked property now that the literal is not a bare `f64`) and `Debug` only under `debug-derive`. - **`src/ltm_augment.rs`** - Equation generators for LTM synthetic variables: `generate_link_score_equation_for_link` (ceteris-paribus link scores; takes `RefShape` and source dimension elements to drive per-shape PREVIOUS wrapping), `generate_loop_score_variables` (emits one `loop_score` per loop as a dimension-shaped `datamodel::Equation`: `Scalar` for scalar loops, `ApplyToAll` for dimensioned loops whose links resolve through Bare A2A names, and per-slot `Equation::Arrayed` for dimensioned loops backed by per-element circuits via `Loop.slot_links` -- GH #653; relative loop scores are computed post-simulation in `ltm_post.rs`), `build_partial_equation_shaped` (the `#[cfg(test)]` TEXT entry point for the ceteris-paribus wrap; arrayed-per-element-equation (`Ast::Arrayed`) targets get one partial per element assembled into an `Equation::Arrayed`). **Production never parses a target equation**: `wrap_changed_first_ast` takes an `Expr0` lowered straight from the target's `Expr2` by `patch::expr2_to_expr0` (which is what `expr2_to_string` prints, so the former print->reparse was a parse of our own output), and every per-occurrence decision -- access shape, the GH #526 other-dep verdict, the literal-element index guard, and the `PerElement` row pinning -- is a lookup into the `db::ltm_ir` occurrence IR by the structural child-index path the wrap tracks, which equals the occurrence's `SiteId` BY CONSTRUCTION now that both walk the same tree. A subscripted source reference the IR did NOT record -- reachable under a `LOOKUP` TABLE argument, which the walker skips as static data ("not a causal edge", which is right about ATTRIBUTION) -- still has to COMPILE, so the pin-only descent discharges it by NAME (`pin_dimension_name_indices`: an index naming one of the TARGET's iterated dimensions becomes the source element this target element reads on that axis, the same structural substitution `pin_bare_source_ref` performs for a bare `Var`). That is a lowering-completeness rule, not a second classifier: it consults no occurrence, infers no shape, and never makes the reference live-selectable -- it asks the SHARED row derivation `per_element_row_for_target` (hence `DimensionsContext::mapped_element_correspondence`) which element an axis reads, so the identity axis and a positionally-MAPPED one (`effect[State, old]` over an `effect[Region, Age]` source, either declaration direction) are one arm and it accepts exactly the mapped pairs `ltm_agg::classify_axis_access` accepts. An index the source's axis DECLARES as an element is resolved BEFORE that dimension-name reading, matching `compiler::subscript`'s `normalize_subscripts3` ("First check if it's a named dimension element (takes priority)") -- the two readings collide when a dimension declares an element whose name is also a dimension the target iterates, and there the element must win or the pin spells a row the simulation never reads. `classify_axis_access` still has the opposite order (GH #986: a precision gap today, since its dimension-first reading finds no mapping and declines). An index that already selects a FIXED element is left verbatim, needing no pin and reading nothing at the current step: a numeric literal, arithmetic over numeric literals (`index_expr_selects_a_fixed_element`, deliberately the narrowest sound predicate -- `compiler::invariance` is the engine's run-invariance derivation but consumes lowered `Expr` and answers a wider question), or an `@N` position index, which `compiler::context` resolves to a concrete element offset in scalar context (spelling `@N` out here would be a second implementation of position syntax). A 0-arity BUILTIN index is where that widening deliberately stops: `TIME` and `PI` are indistinguishable inside `Expr0::App` without a fourth copy of the builtin classification, so the arm stays conservatively loud. What the SHARED derivation declines -- an unmapped or element-mapped pair (GH #756), a transposition, a dimension the target does not iterate -- is declined LOUDLY (`WrapOutcome::missing_occurrence` -> warned skip) rather than emitted with its dimension-name subscript intact, which would not resolve in a scalar fragment. The whole two-verdict space (`Unspellable`, a compilability verdict loud everywhere; `RuntimeRead`, a ceteris-paribus verdict loud only in a bare table argument) is ENUMERATED cell by cell in `ltm_augment_pin_tests.rs`'s two verdict enumerations rather than sampled. There is ONE access-shape classifier family, on `Expr2`; the Expr0 mirror is test-support only and lives in `ltm_augment_wrap_test_support.rs` (kept because three wrap unit tests -- unparseable text, empty text, and the Fig. 2 Q4 `SUM(w[from]) + from` shape the engine REJECTS as a model -- cannot be db-backed fixtures), with `ltm_classifier_agreement_tests.rs` proving it matches the IR field for field (`SiteId` path, `shape`, `axes`, `in_reducer`) corpus-wide, `link_score_var_name` (synthetic name helper: Bare gets the canonical `{from}\u{2192}{to}` form, FixedIndex prepends `[elem]` to from; the obsolete per-shape `\u{205A}wildcard`/`\u{205A}dynamic` Wildcard/DynamicIndex suffixes were retired -- those shapes now collapse onto the Bare name, since *every statically-describable* inlined reducer (whole-extent or sliced) is hoisted into a `$⁚ltm⁚agg⁚{n}` node and only a `DynamicIndex` reference -- `arr[i+1]`, a range, or the not-hoistable dynamic-index reducer carve-out `SUM(pop[idx,*])` -- a whole-RHS variable-backed reducer's `Wildcard` argument, or a de-hoisted array-valued reducer's `Wildcard` arg (`RANK(pop[*], 1)`, GH #771) reach this function), `quote_ident` (identifier quoting for equations). Array support: `classify_reducer` (walks target Expr2 AST to identify reducing builtins -- Linear for SUM/MEAN, Nonlinear for MIN/MAX/STDDEV/RANK, Constant for SIZE -- a thin reader of `ltm_agg::reducer_kind`; it also hands back the reducer's array-argument AST as `ClassifiedReducer::body`, lowered from `Expr2` rather than printed, so the body-aware row partials never re-parse it), `generate_element_to_scalar_equation` (per-element link score equations for arrayed-to-scalar edges, used by both the variable-backed-reducer path and the `source[d] → $⁚ltm⁚agg⁚{n}` half) which dispatches on `ReducerKind` -- `generate_linear_partial` (SUM/MEAN algebraic shortcut), `generate_nonlinear_partial` (MIN/MAX nested binary calls; STDDEV the unrolled population-variance `sqrt` ceteris-paribus partial -- divisor `N`, matching `vm.rs::Opcode::ArrayStddev`, with the mean string-inlined; RANK the documented delta-ratio stand-in pinned by `test_generate_rank_keeps_delta_ratio` -- an order statistic, non-differentiable and unreachable via a real model RHS), `generate_scalar_to_element_equation` (per-element link score for the `$⁚ltm⁚agg⁚{n} → target[e]` half; takes a `source_ref_override: Option<&str>` so a multi-slot arrayed agg's `Δsource` denominator carries the projected `agg[]` subscript instead of the bare agg name, which wouldn't compile as a scalar), `substitute_reducers_in_expr0` (textually replaces a recognized reducer subexpression in an `Expr0` with its agg name, for the `$⁚ltm⁚agg⁚{n} → target` link score), `resolve_link_score_name_for_loop` (picks the Bare-or-FixedIndex link-score name a loop-score reference should target). Module link score formulas (black-box delta-ratio and composite-ref) are inlined directly into `module_link_score_equation` in `db.rs` (called by the per-shape `link_score_equation_text_shaped`). The partial-equation builders (`build_partial_equation_shaped`/`_with_live_ref`, `subscript_idents_at_element`) and every link-score equation generator return `Result<_, PartialEquationError>`: a parse failure (genuine `Err`, or an empty `Ok(None)` equation) has no AST to PREVIOUS-wrap, so emitting the unwrapped input would silently produce a non-ceteris-paribus "partial" identical to the full equation (link score magnitude constant |Δz/Δz| = 1) -- a hidden attribution error that compiles cleanly (GH #311). The db-bearing callers (`link_score_equation_text_shaped` and the `src/db/ltm/link_scores.rs` emitters) convert the error into a `Warning` (`emit_ltm_partial_equation_warning`, naming the variable + offending equation text) and skip the variable -- distinct from `model_ltm_fragment_diagnostics`, which only catches *compile* failures. The failure is effectively unreachable in production (the text is always a `print_eqn` re-print; an empty equation is rejected as an `EmptyEquation` Error upstream), so this is defense-in-depth. The `#[cfg(test)]` tests live in the sibling `src/ltm_augment_tests.rs` (split out for the per-file line cap). Four more siblings are `#[path]`-mounted into `ltm_augment` purely for that cap, so every caller still names their items `crate::ltm_augment::*`: **`ltm_augment_occurrence.rs`** (the wrap's read side of the occurrence IR -- `SlotOccurrences` groups a target's stream by slot ONCE and is the only way to obtain an `OccurrenceLookup`, so the borrow forces callers to hoist it out of their per-element loop), **`ltm_augment_post_transform.rs`** (the concrete-form lowerings: the agg-name substitution, and the `PerElement` row pinning the wrap calls AS IT DESCENDS -- the wrap is the only place that knows both the occurrence and whether it is about to FREEZE the reference, which is what picks the bare row for the live occurrence over the qualified row for every other one), **`ltm_augment_with_lookup.rs`** (the GH #910 implicit-WITH-LOOKUP rules), and **`ltm_augment_wrap_test_support.rs`** (the `#[cfg(test)]` occurrence reconstruction + the Expr0 classifier mirror described above). - **`src/ltm_augment_with_lookup.rs`** - The implicit-WITH-LOOKUP rules for LTM (GH #910), re-exported from `ltm_augment` so every `crate::ltm_augment::*` path is unchanged. A `v = WITH LOOKUP(input, table)` variable is lowered by the compiler to `LOOKUP(v, input)` (`compiler::apply_implicit_with_lookup`), so a link-score partial that RE-EVALUATES such a target's equation is in gf-INPUT units while the guard form's `PREVIOUS(target)` anchor is in gf-OUTPUT units. `is_implicit_with_lookup` carries the coverage doc: every partial is either a **full re-evaluation** (class 1 -- must be wrapped in the gf application) or a **delta-ratio stand-in** (class 2 -- the RANK arm and the nested-arithmetic arm, already in output units, must NEVER be wrapped or a gf output is fed back through the gf). `WithLookupSlotRefs` resolves the target's table reference ONCE per target (`NoGf` / `Shared` / `PerElement`), so a per-element-gf target costs one row-major `SubscriptIterator` walk rather than one per element; an arrayed target's shared table is pinned as `to[1,...]` because a bare arrayed reference resolves each iterated element's own table offset, which the VM reads as NaN past `table_count`. `compose_with_lookup_polarity` (in `src/ltm/polarity.rs`) is the polarity twin, mirroring `apply_implicit_with_lookup`'s placeholder-Positive and zero-point-table rules. The polarity tests live in `src/ltm/with_lookup_tests.rs`, a child module of `ltm::tests`. - **`src/ltm_post.rs`** - Post-simulation relative loop *and link* score computation. `compute_rel_loop_scores(results, loop_partitions)` normalizes each loop's `loop_score` series against the sum of absolute scores within its cycle partition, using SAFEDIV-0 semantics (zero denominator -> zero result). Called after simulation rather than emitted as synthetic equations to avoid O(P^2) equation-text growth on models with dense partitions. `loop_partitions` is an `IndexMap` (re-exported `engine::indexmap`). `compute_rel_loop_scores*` walk its **emission** order rather than re-sorting the loop ids: the partition-sum denominator accumulates `|loop_score|` in that order, and emission order keeps the IEEE-754 (non-associative) sum bit-for-bit identical to the pre-#461 compile-time emitter (GH #468). Emission order is itself deterministic across salsa cache invalidations / processes because `assign_loop_ids` orders loops by a content-derived key (`ltm::graph::loop_id_sort_key`), so it never flaps even though `IndexMap`'s `PartialEq` (salsa cache equality) is order-insensitive. `compute_rel_link_scores(links, step_count)` is the link-level analogue (GH #652): raw link scores divide by the change in the *target*, so they are not comparable across targets and ranking by raw magnitude surfaces numerically-degenerate links (near-constant targets blow up the score). It groups the input `RelLinkInput { to, score }` links by their `to` target and normalizes each link's score by the per-target, per-timestep sum of `|score|` over all that target's *scored* inputs -- a **signed** value in `[-1, 1]` (sign kept like `compute_rel_loop_scores`), with the same `denom_summand` NaN-exclusion / Inf-retention / SAFEDIV-0 semantics. Denominator scope is the scored inputs only: complete in discovery mode (every causal edge scored) but covering just the in-loop subset in exhaustive mode (documented caveat on the fn). libsimlin's `analyze_links_core` calls it over the final (post-synthetic-collapse) link set so the per-target denominator matches the links the caller receives. @@ -196,7 +196,7 @@ The unit subsystem is partial-result throughout: a single bad declaration or one ## Utilities -- **`src/float.rs`** - Floating-point helpers. `approx_eq` is the ULP-based f64 equality used throughout. `crate::float::NA` is Vensim's `:NA:` ("missing data") sentinel: the *finite* value `-2^109` (exactly representable in f64), **NOT** IEEE NaN. It is finite by design so the Vensim existence idiom `IF THEN ELSE(x = :NA:, ...)` works (`approx_eq` matches the sentinel against itself, never against a contaminated value) and `:NA:` arithmetic stays finite rather than being poisoned by an absorbing NaN. Both `:NA:` entry points route here: the expression literal via the MDL→XMILE formatter (`src/mdl/xmile_compat.rs`) and the data-list literal via the MDL number-list parser (`src/mdl/parser.rs`, `NA_VALUE`). This is distinct from the genuine NaN the engine still produces for out-of-bounds vector reads (`vm_vector_elm_map.rs`) and empty array reducers (`vm.rs` ArrayMax/Min/Mean/Stddev). +- **`src/float.rs`** - Floating-point helpers. `approx_eq` is the ULP-based f64 equality used throughout. `crate::float::NA` is Vensim's `:NA:` ("missing data") sentinel: the *finite* value `-2^109` (exactly representable in f64), **NOT** IEEE NaN. It is finite by design so the Vensim existence idiom `IF THEN ELSE(x = :NA:, ...)` works (`approx_eq` matches the sentinel against itself, never against a contaminated value) and `:NA:` arithmetic stays finite rather than being poisoned by an absorbing NaN. Both `:NA:` entry points route here: the expression literal via the MDL→XMILE formatter (`src/mdl/xmile_compat.rs`) and the data-list literal via the MDL number-list parser (`src/mdl/parser.rs`, `NA_VALUE`). This is distinct from the genuine NaN the engine still produces for out-of-bounds vector reads (`vm_vector_elm_map.rs`) and empty array reducers (`vm.rs` ArrayMax/Min/Mean/Stddev). **The module docs are the landing page for "why does this codebase treat NaN the way it does"**, and the two conventions belong together: `NA` is finite and comparable ON PURPOSE, while a NaN is a propagating failure signal. What a practitioner sees is a line on a graph that stops, and their next task is provenance -- searching BACKWARD through the dependency graph, because NaN is absorbing and every variable downstream of the origin shows the identical symptom. Two engine consequences follow: attributing a NaN to its origin is high-value (wherever we know STRUCTURALLY that a variable must be NaN -- an unfilled equation being the clearest case -- a warning naming the variable replaces the whole hand search), and a NaN the ENGINE manufactures is noise in a channel practitioners already debug by hand (indistinguishable on the graph from the user's own division by zero, so it costs a debugging session that ends at our bug -- which is what makes GH #975 a real defect, not a cosmetic one). The technical footnote follows rather than motivates: a practitioner cannot author a NaN payload (one `nan` keyword, one canonical `f64::NAN`), so the engine needs no machinery treating NaN as a structured value, and bit-pattern comparison distinguishes nothing that exists. - **`src/io.rs`** - `atomic_write(path, contents)`: writes bytes to a sibling `.new` temp file, fsyncs it, then renames over the target. Cleans up the temp file on error. Best-effort parent-directory fsync after rename for durability on power loss. Used by MCP and CLI tools that need crash-safe file output. ## Cargo features diff --git a/src/simlin-engine/src/ast/array_view.rs b/src/simlin-engine/src/ast/array_view.rs index d032d08a4..6af3fe74e 100644 --- a/src/simlin-engine/src/ast/array_view.rs +++ b/src/simlin-engine/src/ast/array_view.rs @@ -8,7 +8,7 @@ use crate::sim_err; /// Information about a sparse (non-contiguous) dimension in an array view. /// Used when a subdimension's elements are not contiguous in the parent dimension. #[cfg_attr(feature = "debug-derive", derive(Debug))] -#[derive(PartialEq, Clone)] +#[derive(PartialEq, Eq, Clone)] pub struct SparseInfo { /// Which dimension (0-indexed) in the view is sparse pub dim_index: usize, @@ -22,7 +22,7 @@ pub struct SparseInfo { /// how we iterate over existing data (changing offsets and strides) rather than /// creating new arrays. #[cfg_attr(feature = "debug-derive", derive(Debug))] -#[derive(PartialEq, Clone)] +#[derive(PartialEq, Eq, Clone)] pub struct ArrayView { /// Dimension sizes after slicing/viewing pub dims: Vec, diff --git a/src/simlin-engine/src/ast/expr0.rs b/src/simlin-engine/src/ast/expr0.rs index 97bc502e0..cba775da8 100644 --- a/src/simlin-engine/src/ast/expr0.rs +++ b/src/simlin-engine/src/ast/expr0.rs @@ -2,6 +2,7 @@ // Use of this source code is governed by the Apache License, // Version 2.0, that can be found in the LICENSE file. +use crate::ast::literal::Literal; use crate::builtins::{Loc, UntypedBuiltinFn, is_0_arity_builtin_fn_ci}; use crate::common::{EquationError, RawIdent}; use crate::lexer::LexerType; @@ -70,10 +71,19 @@ impl BinaryOp { /// Expr0 represents a parsed equation, before any calls to /// builtin functions have been checked/resolved. +/// +/// The `Eq` derive is load-bearing, not decoration: `Expr0` rides on +/// salsa-cached values (`db::query::ParsedVariableResult`, `ModelStage0`, +/// `db::ltm::LtmArm`) whose backdating is decided by comparing an old value +/// with a rebuilt one, so a variant that is not equal to ITSELF permanently +/// defeats that comparison. A bare `f64` is exactly such a field (`NaN != +/// NaN`), and `Eq` rejects it at compile time -- which is why the literal is an +/// [`Literal`], compared by bit pattern. The same argument applies to `Expr1`, +/// `Expr2` and `Expr3`; see [`Literal`] for the full statement. #[cfg_attr(feature = "debug-derive", derive(Debug))] -#[derive(PartialEq, Clone, salsa::Update)] +#[derive(PartialEq, Eq, Clone, salsa::Update)] pub enum Expr0 { - Const(String, f64, Loc), + Const(String, Literal, Loc), Var(RawIdent, Loc), App(UntypedBuiltinFn, Loc), Subscript(RawIdent, Vec, Loc), @@ -83,7 +93,7 @@ pub enum Expr0 { } #[cfg_attr(feature = "debug-derive", derive(Debug))] -#[derive(PartialEq, Clone, salsa::Update)] +#[derive(PartialEq, Eq, Clone, salsa::Update)] pub enum IndexExpr0 { Wildcard(Loc), StarRange(RawIdent, Loc), @@ -280,7 +290,7 @@ impl Expr0 { impl Default for Expr0 { fn default() -> Self { - Expr0::Const("0.0".to_string(), 0.0, Loc::default()) + Expr0::Const("0.0".to_string(), Literal::new(0.0), Loc::default()) } } @@ -291,9 +301,9 @@ fn test_parse() { use Expr0::*; let if1 = Box::new(If( - Box::new(Const("1".to_string(), 1.0, Loc::default())), - Box::new(Const("2".to_string(), 2.0, Loc::default())), - Box::new(Const("3".to_string(), 3.0, Loc::default())), + Box::new(Const("1".to_string(), Literal::new(1.0), Loc::default())), + Box::new(Const("2".to_string(), Literal::new(2.0), Loc::default())), + Box::new(Const("3".to_string(), Literal::new(3.0), Loc::default())), Loc::default(), )); @@ -304,8 +314,8 @@ fn test_parse() { Box::new(Var(RawIdent::new_from_str("foo"), Loc::default())), Loc::default(), )), - Box::new(Const("2".to_string(), 2.0, Loc::default())), - Box::new(Const("3".to_string(), 3.0, Loc::default())), + Box::new(Const("2".to_string(), Literal::new(2.0), Loc::default())), + Box::new(Const("3".to_string(), Literal::new(3.0), Loc::default())), Loc::default(), )); @@ -319,8 +329,8 @@ fn test_parse() { )), Loc::default(), )), - Box::new(Const("1".to_string(), 1.0, Loc::default())), - Box::new(Const("0".to_string(), 0.0, Loc::default())), + Box::new(Const("1".to_string(), Literal::new(1.0), Loc::default())), + Box::new(Const("0".to_string(), Literal::new(0.0), Loc::default())), Loc::default(), )); @@ -331,8 +341,8 @@ fn test_parse() { Box::new(Var(RawIdent::new_from_str("false_input"), Loc::default())), Loc::default(), )), - Box::new(Const("1".to_string(), 1.0, Loc::default())), - Box::new(Const("0".to_string(), 0.0, Loc::default())), + Box::new(Const("1".to_string(), Literal::new(1.0), Loc::default())), + Box::new(Const("0".to_string(), Literal::new(0.0), Loc::default())), Loc::default(), )); @@ -345,13 +355,17 @@ fn test_parse() { let subscript1 = Box::new(Subscript( RawIdent::new_from_str("a"), - vec![IndexExpr0::Expr(Const("1".to_owned(), 1.0, Loc::default()))], + vec![IndexExpr0::Expr(Const( + "1".to_owned(), + Literal::new(1.0), + Loc::default(), + ))], Loc::default(), )); let subscript2 = Box::new(Subscript( RawIdent::new_from_str("a"), vec![ - IndexExpr0::Expr(Const("2".to_owned(), 2.0, Loc::default())), + IndexExpr0::Expr(Const("2".to_owned(), Literal::new(2.0), Loc::default())), IndexExpr0::Expr(App( UntypedBuiltinFn( "int".to_owned(), @@ -384,8 +398,8 @@ fn test_parse() { let subscript5 = Box::new(Subscript( RawIdent::new_from_str("a"), vec![IndexExpr0::Range( - Const("1".to_owned(), 1.0, Loc::default()), - Const("2".to_owned(), 2.0, Loc::default()), + Const("1".to_owned(), Literal::new(1.0), Loc::default()), + Const("2".to_owned(), Literal::new(2.0), Loc::default()), Loc::default(), )], Loc::default(), @@ -435,13 +449,13 @@ fn test_parse() { UntypedBuiltinFn("time".to_owned(), vec![]), Loc::default(), )), - Box::new(Const("5".to_owned(), 5.0, Loc::default())), + Box::new(Const("5".to_owned(), Literal::new(5.0), Loc::default())), Loc::default(), )], ), Loc::default(), )), - Box::new(Const("1".to_owned(), 1.0, Loc::default())), + Box::new(Const("1".to_owned(), Literal::new(1.0), Loc::default())), Loc::default(), ))], Loc::default(), @@ -460,7 +474,7 @@ fn test_parse() { RawIdent::new_from_str("matrix"), vec![ IndexExpr0::Wildcard(Loc::default()), - IndexExpr0::Expr(Const("1".to_owned(), 1.0, Loc::default())), + IndexExpr0::Expr(Const("1".to_owned(), Literal::new(1.0), Loc::default())), ], Loc::default(), )), @@ -541,7 +555,7 @@ fn test_parse() { assert!(matches!(&ast, Expr0::Const(_, _, _))); if let Expr0::Const(id, n, _) = &ast { assert_eq!("NaN", id); - assert!(n.is_nan()); + assert!(n.value().is_nan()); } let printed = ast::print_eqn(&ast); assert_eq!("NaN", &printed); @@ -609,8 +623,8 @@ fn test_safediv_operator() { UntypedBuiltinFn( "safediv".to_owned(), vec![ - Const("1".to_owned(), 1.0, Loc::default()), - Const("2".to_owned(), 2.0, Loc::default()), + Const("1".to_owned(), Literal::new(1.0), Loc::default()), + Const("2".to_owned(), Literal::new(2.0), Loc::default()), ], ), Loc::default(), diff --git a/src/simlin-engine/src/ast/expr1.rs b/src/simlin-engine/src/ast/expr1.rs index a7f7aa6e7..7ceefbcbd 100644 --- a/src/simlin-engine/src/ast/expr1.rs +++ b/src/simlin-engine/src/ast/expr1.rs @@ -3,6 +3,7 @@ // Version 2.0, that can be found in the LICENSE file. use crate::ast::expr0::{BinaryOp, Expr0, IndexExpr0, UnaryOp}; +use crate::ast::literal::Literal; pub use crate::builtins::Loc; use crate::builtins::{BuiltinFn, UntypedBuiltinFn}; use crate::common::{Canonical, EquationResult, Ident}; @@ -12,7 +13,7 @@ use crate::model::ScopeStage0; /// IndexExpr1 represents a parsed equation, after calls to /// builtin functions have been checked/resolved. #[cfg_attr(feature = "debug-derive", derive(Debug))] -#[derive(PartialEq, Clone)] +#[derive(PartialEq, Eq, Clone)] pub enum IndexExpr1 { Wildcard(Loc), // *:dimension_name @@ -54,10 +55,14 @@ impl IndexExpr1 { /// Expr represents a parsed equation, after calls to /// builtin functions have been checked/resolved. +/// +/// `Eq` is derived for the reason spelled out on [`Expr0`]: it makes +/// "reflexive, therefore backdateable" a compile-checked property, which is what +/// keeps a future float-bearing field from being a bare `f64`. #[cfg_attr(feature = "debug-derive", derive(Debug))] -#[derive(PartialEq, Clone)] +#[derive(PartialEq, Eq, Clone)] pub enum Expr1 { - Const(String, f64, Loc), + Const(String, Literal, Loc), Var(Ident, Loc), App(BuiltinFn, Loc), Subscript(Ident, Vec, Loc), @@ -275,7 +280,7 @@ impl Expr1 { "previous" => { if args.len() == 1 { let a = args.remove(0); - let zero = Expr1::Const("0".to_string(), 0.0, loc); + let zero = Expr1::Const("0".to_string(), Literal::new(0.0), loc); BuiltinFn::Previous(Box::new(a), Box::new(zero)) } else if args.len() == 2 { let b = args.remove(1); @@ -324,7 +329,7 @@ impl Expr1 { Expr1::Const(s, n, loc) => Expr1::Const(s, n, loc), Expr1::Var(id, loc) => { if let Some(off) = scope.dimensions.lookup(id.as_str()) { - Expr1::Const(id.to_string(), off as f64, loc) + Expr1::Const(id.to_string(), Literal::new(off as f64), loc) } else { Expr1::Var(id, loc) } @@ -487,6 +492,6 @@ impl Expr1 { impl Default for Expr1 { fn default() -> Self { - Expr1::Const("0.0".to_string(), 0.0, Loc::default()) + Expr1::Const("0.0".to_string(), Literal::new(0.0), Loc::default()) } } diff --git a/src/simlin-engine/src/ast/expr2.rs b/src/simlin-engine/src/ast/expr2.rs index 653be837a..efd8a389c 100644 --- a/src/simlin-engine/src/ast/expr2.rs +++ b/src/simlin-engine/src/ast/expr2.rs @@ -4,6 +4,7 @@ use crate::ast::expr0::{BinaryOp, UnaryOp}; use crate::ast::expr1::{Expr1, IndexExpr1}; +use crate::ast::literal::Literal; use crate::builtins::{BuiltinContents, BuiltinFn, Loc, walk_builtin_expr}; use crate::common::{Canonical, CanonicalDimensionName, EquationResult, Ident}; use crate::dimensions::Dimension; @@ -18,7 +19,7 @@ use crate::eqn_err; /// All complex view calculations (strides, offsets, etc.) are deferred /// to the compiler phase where we have more context. #[cfg_attr(feature = "debug-derive", derive(Debug))] -#[derive(PartialEq, Clone, salsa::Update)] +#[derive(PartialEq, Eq, Clone, salsa::Update)] pub enum ArrayBounds { /// Array bounds for a named variable (from the model) Named { @@ -70,7 +71,7 @@ impl ArrayBounds { /// IndexExpr represents a parsed equation, after calls to /// builtin functions have been checked/resolved. #[cfg_attr(feature = "debug-derive", derive(Debug))] -#[derive(PartialEq, Clone, salsa::Update)] +#[derive(PartialEq, Eq, Clone, salsa::Update)] pub enum IndexExpr2 { Wildcard(Loc), // *:dimension_name @@ -147,11 +148,17 @@ impl IndexExpr2 { /// Expr represents a parsed equation, after calls to /// builtin functions have been checked/resolved. +/// +/// `Eq` is derived for the reason spelled out on [`crate::ast::Expr0`]: this is +/// the layer that rides on `ModelStage1` and `ltm_agg::AggNodesResult`, whose +/// salsa backdating is decided by comparing a memo with its own rebuild, so a +/// field that is not equal to itself defeats it. `Eq` makes that a compile-time +/// property rather than a convention. #[allow(dead_code)] #[cfg_attr(feature = "debug-derive", derive(Debug))] -#[derive(PartialEq, Clone, salsa::Update)] +#[derive(PartialEq, Eq, Clone, salsa::Update)] pub enum Expr2 { - Const(String, f64, Loc), + Const(String, Literal, Loc), Var(Ident, Option, Loc), App(BuiltinFn, Option, Loc), Subscript(Ident, Vec, Option, Loc), @@ -624,8 +631,9 @@ impl Expr2 { Expr2::Const(_, val, _) => { // Numeric constant - interpret as 1-based index. // Guard against overflow: val must be in range [1, isize::MAX]. - if *val >= 1.0 && *val <= isize::MAX as f64 { - Some((*val as usize).saturating_sub(1)) + let val = val.value(); + if val >= 1.0 && val <= isize::MAX as f64 { + Some((val as usize).saturating_sub(1)) } else { None } @@ -811,7 +819,11 @@ impl Expr2 { let dim_name = CanonicalDimensionName::from_raw(id.as_str()); if let Some(len) = ctx.get_dimension_len(&dim_name) { // Return a constant expression with the dimension size - return Ok(Expr2::Const(len.to_string(), len as f64, loc)); + return Ok(Expr2::Const( + len.to_string(), + Literal::new(len as f64), + loc, + )); } // If we can't find the dimension length, fall through to normal processing // which will produce an appropriate error @@ -1114,7 +1126,8 @@ fn const_int_eval(ast: &Expr2) -> EquationResult { use crate::float::approx_eq; match ast { Expr2::Const(_, n, loc) => { - if approx_eq(*n, n.round()) { + let n = n.value(); + if approx_eq(n, n.round()) { Ok(n.round() as i32) } else { eqn_err!(ExpectedInteger, loc.start, loc.end) @@ -1307,7 +1320,7 @@ mod tests { fn test_const_int_eval() { // Helper to create const expression fn const_expr(val: f64) -> Expr2 { - Expr2::Const(val.to_string(), val, Loc::default()) + Expr2::Const(val.to_string(), Literal::new(val), Loc::default()) } // Test basic constants @@ -1519,7 +1532,11 @@ mod tests { let subscript_expr = Expr1::Subscript( Ident::new("matrix"), vec![ - IndexExpr1::Expr(Expr1::Const("1".to_string(), 1.0, Loc::default())), + IndexExpr1::Expr(Expr1::Const( + "1".to_string(), + Literal::new(1.0), + Loc::default(), + )), IndexExpr1::Wildcard(Loc::default()), ], Loc::default(), @@ -1560,7 +1577,7 @@ mod tests { Ident::new("vector"), vec![IndexExpr1::Expr(Expr1::Const( "2".to_string(), - 2.0, + Literal::new(2.0), Loc::default(), ))], Loc::default(), @@ -1665,7 +1682,11 @@ mod tests { let add_expr = Expr1::Op2( BinaryOp::Add, Box::new(Expr1::Var(Ident::new("array_var"), Loc::default())), - Box::new(Expr1::Const("10".to_string(), 10.0, Loc::default())), + Box::new(Expr1::Const( + "10".to_string(), + Literal::new(10.0), + Loc::default(), + )), Loc::default(), ); let expr2 = Expr2::from(add_expr, &mut ctx).unwrap(); @@ -1738,7 +1759,11 @@ mod tests { // Test if expression with array in both branches let if_expr = Expr1::If( - Box::new(Expr1::Const("1".to_string(), 1.0, Loc::default())), + Box::new(Expr1::Const( + "1".to_string(), + Literal::new(1.0), + Loc::default(), + )), Box::new(Expr1::Var(Ident::new("array_var"), Loc::default())), Box::new(Expr1::Var(Ident::new("array_var"), Loc::default())), Loc::default(), diff --git a/src/simlin-engine/src/ast/expr3.rs b/src/simlin-engine/src/ast/expr3.rs index c63b97adb..0d37eca62 100644 --- a/src/simlin-engine/src/ast/expr3.rs +++ b/src/simlin-engine/src/ast/expr3.rs @@ -5,6 +5,7 @@ use crate::ast::array_view::ArrayView; use crate::ast::expr0::{BinaryOp, UnaryOp}; use crate::ast::expr2::{ArrayBounds, Expr2, IndexExpr2}; +use crate::ast::literal::Literal; use crate::builtins::{BuiltinFn, Loc}; use crate::common::{ Canonical, CanonicalDimensionName, CanonicalElementName, EquationResult, Ident, @@ -18,7 +19,7 @@ use crate::eqn_err; /// During the expr2 → expr3 lowering pass, all wildcards are resolved /// to explicit StarRange expressions based on the variable's dimensions. #[cfg_attr(feature = "debug-derive", derive(Debug))] -#[derive(PartialEq, Clone)] +#[derive(PartialEq, Eq, Clone)] pub enum IndexExpr3 { /// Star range (*:dim or dim.*) - preserves dimension for iteration. /// This includes both user-specified star ranges AND wildcards that @@ -87,11 +88,16 @@ impl IndexExpr3 { /// - Keeps string representation in Const for debugging /// - No module-specific variants (EvalModule, ModuleInput) /// - No assignment variants (AssignCurr, AssignNext) +/// +/// `Eq` is derived for the reason spelled out on [`crate::ast::Expr0`], even +/// though this layer is not itself salsa-cached: it keeps all four layers under +/// one rule, so a float-bearing variant added here cannot be a bare `f64` and +/// then be copied down to a layer where it would matter. #[cfg_attr(feature = "debug-derive", derive(Debug))] -#[derive(PartialEq, Clone)] +#[derive(PartialEq, Eq, Clone)] pub enum Expr3 { // Core variants (similar to Expr2) - Const(String, f64, Loc), + Const(String, Literal, Loc), Var(Ident, Option, Loc), App(BuiltinFn, Option, Loc), /// Dynamic subscript - indices computed at runtime @@ -336,8 +342,8 @@ impl IndexExpr3 { { // Convert 1-based indices to 0-based for StaticRange // StaticRange stores (0-based start, 0-based exclusive end) - let start_0based = (*start_val as usize).saturating_sub(1); - let end_0based = *end_val as usize; // end is already exclusive in XMILE + let start_0based = (start_val.value() as usize).saturating_sub(1); + let end_0based = end_val.value() as usize; // end is already exclusive in XMILE return Ok(IndexExpr3::StaticRange(start_0based, end_0based, *loc)); } @@ -708,7 +714,8 @@ impl<'a> Pass1Context<'a> { if active_dim_name.as_str() == dim_name.as_str() { // Found a match - resolve to the concrete index if let Some(index) = Self::subscript_to_index(dim, sub) { - let const_expr = Expr3::Const(index.to_string(), index, loc); + let const_expr = + Expr3::Const(index.to_string(), Literal::new(index), loc); return (IndexExpr3::Expr(const_expr), false); } // subscript_to_index returned None - subscript not valid for dimension. @@ -1392,7 +1399,7 @@ mod tests { #[test] fn test_expr3_const() { - let expr = Expr3::Const("42".to_string(), 42.0, Loc::new(0, 2)); + let expr = Expr3::Const("42".to_string(), Literal::new(42.0), Loc::new(0, 2)); assert_eq!(expr.get_loc(), Loc::new(0, 2)); assert!(expr.get_array_bounds().is_none()); assert!(expr.get_array_view().is_none()); @@ -1447,7 +1454,7 @@ mod tests { #[test] fn test_expr3_assign_temp() { - let inner = Expr3::Const("1".to_string(), 1.0, Loc::new(0, 1)); + let inner = Expr3::Const("1".to_string(), Literal::new(1.0), Loc::new(0, 1)); let view = ArrayView::contiguous(vec![2, 3]); let expr = Expr3::AssignTemp(0, Box::new(inner), view); @@ -1459,8 +1466,16 @@ mod tests { fn test_expr3_strip_loc() { let expr = Expr3::Op2( BinaryOp::Add, - Box::new(Expr3::Const("1".to_string(), 1.0, Loc::new(0, 1))), - Box::new(Expr3::Const("2".to_string(), 2.0, Loc::new(4, 5))), + Box::new(Expr3::Const( + "1".to_string(), + Literal::new(1.0), + Loc::new(0, 1), + )), + Box::new(Expr3::Const( + "2".to_string(), + Literal::new(2.0), + Loc::new(4, 5), + )), None, Loc::new(0, 5), ); @@ -1714,7 +1729,11 @@ mod tests { Ident::new("matrix"), vec![ IndexExpr2::Wildcard(Loc::new(7, 8)), - IndexExpr2::Expr(Expr2::Const("2".to_string(), 2.0, Loc::new(10, 11))), + IndexExpr2::Expr(Expr2::Const( + "2".to_string(), + Literal::new(2.0), + Loc::new(10, 11), + )), ], None, Loc::new(0, 12), @@ -1737,7 +1756,7 @@ mod tests { // Second subscript: constant expression match &args[1] { IndexExpr3::Expr(Expr3::Const(_, val, _)) => { - assert_eq!(*val, 2.0); + assert_eq!(*val, Literal::new(2.0)); } _ => panic!("Expected Expr(Const) for second subscript"), } @@ -1843,7 +1862,11 @@ mod tests { vec![ IndexExpr2::Wildcard(Loc::new(5, 6)), IndexExpr2::Wildcard(Loc::new(8, 9)), - IndexExpr2::Expr(Expr2::Const("5".to_string(), 5.0, Loc::new(11, 12))), + IndexExpr2::Expr(Expr2::Const( + "5".to_string(), + Literal::new(5.0), + Loc::new(11, 12), + )), ], None, Loc::new(0, 13), @@ -1868,7 +1891,9 @@ mod tests { // Third subscript: constant expression match &args[2] { - IndexExpr3::Expr(Expr3::Const(_, val, _)) => assert_eq!(*val, 5.0), + IndexExpr3::Expr(Expr3::Const(_, val, _)) => { + assert_eq!(*val, Literal::new(5.0)) + } _ => panic!("Expected Expr(Const) for third subscript"), } } @@ -2076,7 +2101,7 @@ mod tests { Loc::new(0, 6), ); - let one = Expr3::Const("1".to_string(), 1.0, Loc::new(10, 11)); + let one = Expr3::Const("1".to_string(), Literal::new(1.0), Loc::new(10, 11)); // arr[*] + 1 - has array bounds because arr[*] is an array let add_expr = Expr3::Op2( @@ -2143,7 +2168,7 @@ mod tests { Loc::new(0, 11), ); - let one = Expr3::Const("1".to_string(), 1.0, Loc::new(14, 15)); + let one = Expr3::Const("1".to_string(), Literal::new(1.0), Loc::new(14, 15)); // arr[*, Col] + 1 - has dimension reference let add_expr = Expr3::Op2( @@ -2275,7 +2300,11 @@ mod tests { let a_add = Expr3::Op2( BinaryOp::Add, Box::new(a_sub), - Box::new(Expr3::Const("1".to_string(), 1.0, Loc::new(6, 7))), + Box::new(Expr3::Const( + "1".to_string(), + Literal::new(1.0), + Loc::new(6, 7), + )), Some(a_bounds), Loc::new(0, 7), ); @@ -2293,7 +2322,11 @@ mod tests { let b_mul = Expr3::Op2( BinaryOp::Mul, Box::new(b_sub), - Box::new(Expr3::Const("2".to_string(), 2.0, Loc::new(6, 7))), + Box::new(Expr3::Const( + "2".to_string(), + Literal::new(2.0), + Loc::new(6, 7), + )), Some(b_bounds), Loc::new(0, 7), ); @@ -2396,7 +2429,11 @@ mod tests { let false_branch = Expr3::Op2( BinaryOp::Add, Box::new(arr2_sub), - Box::new(Expr3::Const("1".to_string(), 1.0, Loc::new(7, 8))), + Box::new(Expr3::Const( + "1".to_string(), + Literal::new(1.0), + Loc::new(7, 8), + )), Some(arr2_bounds), Loc::new(0, 8), ); @@ -2448,7 +2485,11 @@ mod tests { let add_expr = Expr3::Op2( BinaryOp::Add, Box::new(subscript), - Box::new(Expr3::Const("1".to_string(), 1.0, Loc::new(10, 11))), + Box::new(Expr3::Const( + "1".to_string(), + Literal::new(1.0), + Loc::new(10, 11), + )), Some(arr_bounds), Loc::new(0, 11), ); @@ -2513,7 +2554,11 @@ mod tests { let left_op = Expr3::Op2( BinaryOp::Add, Box::new(arr_sub), - Box::new(Expr3::Const("1".to_string(), 1.0, Loc::new(7, 8))), + Box::new(Expr3::Const( + "1".to_string(), + Literal::new(1.0), + Loc::new(7, 8), + )), Some(arr_bounds.clone()), Loc::new(0, 8), ); @@ -2522,7 +2567,11 @@ mod tests { let right_op = Expr3::Op2( BinaryOp::Sub, Box::new(arr2_sub), - Box::new(Expr3::Const("1".to_string(), 1.0, Loc::new(15, 16))), + Box::new(Expr3::Const( + "1".to_string(), + Literal::new(1.0), + Loc::new(15, 16), + )), Some(arr2_bounds), Loc::new(0, 16), ); @@ -2592,7 +2641,7 @@ mod tests { Loc::new(0, 11), ); - let one = Expr3::Const("1".to_string(), 1.0, Loc::new(14, 15)); + let one = Expr3::Const("1".to_string(), Literal::new(1.0), Loc::new(14, 15)); // arr[*, Col] + 1 - the array bounds after subscripting should be [3] // because Col is pinned to a single value @@ -2669,7 +2718,7 @@ mod tests { Loc::new(0, 11), ); - let one = Expr3::Const("1".to_string(), 1.0, Loc::new(14, 15)); + let one = Expr3::Const("1".to_string(), Literal::new(1.0), Loc::new(14, 15)); let add_expr = Expr3::Op2( BinaryOp::Add, @@ -2723,7 +2772,11 @@ mod tests { let add_expr = Expr3::Op2( BinaryOp::Add, Box::new(subscript), - Box::new(Expr3::Const("1".to_string(), 1.0, Loc::new(10, 11))), + Box::new(Expr3::Const( + "1".to_string(), + Literal::new(1.0), + Loc::new(10, 11), + )), Some(arr_bounds), // Has array bounds Loc::new(0, 11), ); @@ -2876,7 +2929,11 @@ mod tests { let add_expr = Expr3::Op2( BinaryOp::Add, Box::new(subscript), - Box::new(Expr3::Const("1".to_string(), 1.0, Loc::new(23, 24))), + Box::new(Expr3::Const( + "1".to_string(), + Literal::new(1.0), + Loc::new(23, 24), + )), Some(arr_bounds), Loc::new(0, 24), ); @@ -2906,7 +2963,7 @@ mod tests { // First index (Row) should be resolved to a constant match &indices[0] { IndexExpr3::Expr(Expr3::Const(_, val, _)) => { - assert_eq!(*val, 2.0, "Row should be resolved to 2"); + assert_eq!(*val, Literal::new(2.0), "Row should be resolved to 2"); } _ => panic!("Expected Row to be resolved, got {:?}", indices[0]), } diff --git a/src/simlin-engine/src/ast/literal.rs b/src/simlin-engine/src/ast/literal.rs new file mode 100644 index 000000000..d4fa45056 --- /dev/null +++ b/src/simlin-engine/src/ast/literal.rs @@ -0,0 +1,408 @@ +// Copyright 2026 The Simlin Authors. All rights reserved. +// Use of this source code is governed by the Apache License, +// Version 2.0, that can be found in the LICENSE file. + +use std::hash::{Hash, Hasher}; + +/// The numeric value of a `Const` node in any of the four AST layers, compared +/// by **bit pattern** rather than by IEEE equality. +/// +/// `Expr0`..`Expr3` (and everything holding one: `ModelStage0`/`ModelStage1`, +/// `db::query::ParsedVariableResult`, `db::ltm::LtmEquation`, +/// `ltm_agg::AggNodesResult`, ...) derive `PartialEq` and `salsa::Update`. +/// Salsa decides whether to *backdate* a re-executed tracked function's memo by +/// comparing the old value with the new one -- `salsa::Update`'s derive walks +/// the fields and, for a field type with no `Update` impl of its own, falls back +/// to that type's `PartialEq` (`salsa::plumbing::UpdateFallback`). A bare `f64` +/// makes that comparison **non-reflexive**: `NaN != NaN`, so a value holding a +/// NaN literal never compares equal to itself, is never backdated, and reports +/// "changed" on every bit-identical re-execution -- taking the whole downstream +/// query cone with it. Every stdlib SMOOTH/DELAY/TREND template declares +/// `initial_value = NAN` and `db::sync` splices those into every project, so +/// this was reachable with no user action at all (GH #987, #981). +/// +/// Comparing the bits fixes that at the root **of the AST**, for every consumer +/// of these ASTs at once, and it stays fixed structurally: a new `Const`-like +/// variant, or any new float-bearing field on a type the AST reaches, must use +/// this type, because the AST enums also derive `Eq` and a bare `f64` cannot +/// satisfy it. That is a compile error rather than a silently reintroduced +/// incrementality cliff, and it is enforced TRANSITIVELY -- adding an `f64` to +/// `ArrayView`, which `Expr3` reaches only indirectly, fails the same way. +/// +/// **Bit comparison costs this layer nothing**, because both distinctions it +/// draws are unreachable in an AST literal by construction -- not accepted +/// tradeoffs, but distinctions that cannot occur: +/// +/// * **NaN payloads.** There is exactly one NaN spelling at the language +/// surface, the lexer's `nan` keyword, and it yields one canonical +/// `f64::NAN`. A practitioner cannot author a payload, and nothing between the +/// parser and here performs arithmetic, so every NaN literal in an AST is +/// bit-identical to every other. Bit comparison distinguishes nothing that +/// exists. (`crate::float`'s module docs carry the domain reason this is the +/// right posture rather than a shortcut: a NaN is a diagnostic signal, not a +/// value with structure worth preserving.) +/// * **Signed zero.** `lexer::scan_number` accepts no leading sign (exponent +/// sign only), so `-0` parses as a NEGATION applied to the literal `0`, never +/// as a `Const` holding `-0.0`. The two spellings were already structurally +/// unequal. Every other producer of a `Literal` -- a dimension offset, a +/// dimension length, a subscript index, the `Literal::new(0.0)` stubs -- is +/// non-negative. +/// +/// Both premises are tripwire tests rather than comments +/// (`literal_tests::plus_and_minus_zero_are_different_literals` and +/// `literal_ast_tests::a_negative_zero_literal_is_a_negation_not_a_signed_const`), +/// so if the lexer ever learns signed number tokens, someone finds out. And no +/// other consumer of these ASTs is affected either way: each reads the number +/// out with [`Literal::value`] and compares `f64`s with its own tolerance +/// (`mdl::writer::exprs_equal`, `ltm::polarity`'s sign tests, the compiler's +/// static index resolution). +/// +/// The API is exactly two operations, [`Literal::new`] and [`Literal::value`]. +/// The accessor is deliberately named rather than a `Deref` to `f64`: at a call +/// site that reads the number out, bit-equality is not what is wanted, and a +/// silent deref would make the distinction invisible. +/// +/// # Float equality in this crate: the position, and the types still outside it +/// +/// **Wherever a float feeds a cache key, we want BIT equality.** The reason is +/// domain, not taste, and it lives in `crate::float`'s module docs: a NaN in a +/// system dynamics model is a failure signal a practitioner has to trace by +/// hand, not a value whose structure is worth preserving. Nothing distinguishes +/// NaN payloads or the two signed zeros meaningfully, and nothing can author +/// them, so bit comparison never separates two floats a user could tell apart -- +/// while IEEE `==` makes a value unequal to its own rebuild, which is what +/// breaks caching. +/// +/// `Literal` implements that position for AST literals. It is not implemented +/// for: +/// +/// * the compiled bytecode types -- `bytecode::ByteCode::literals`, +/// `bytecode::ByteCodeContext::graphical_functions`, `results::Specs`, and +/// their symbolic twins `compiler::symbolic::SymbolicByteCode::literals` / +/// `PerVarBytecodes::graphical_functions` (GH #642); +/// * `variable::Table`'s `x`/`y: Vec` lookup points and its two +/// `datamodel::GraphicalFunctionScale`s, which ride on `Variable::Var` into +/// the same `ModelStage0` / `ModelStage1` / `db::query::ParsedVariableResult` +/// memos this type's fix serves. +/// +/// Both keep the derived, IEEE-based `PartialEq`. That is an accepted state, not +/// an open defect -- **what is accepted is the cost of NOT converting them, not +/// a semantic property worth keeping.** Stated exactly, because IEEE equality +/// diverges from bit equality in BOTH directions: +/// +/// 1. **Stricter on NaN -> a missed backdate (perf only).** `NaN != NaN`, so a +/// bit-identical rebuild reports "changed" and the downstream cone +/// re-executes. This is #642, closed on this reasoning. Note the corrected +/// premise, so nobody re-litigates a false one: #642's body argues it is +/// benign because "the only consumer, `compile_project_incremental`, is +/// non-tracked". That is WRONG one level down -- `PerVarBytecodes` +/// (`compiler/symbolic.rs`) is the value of the *tracked* +/// `compile_var_fragment`, which the *tracked* `assemble_module` reads. The +/// missed backdate is real; we accept it because assembly re-executes on +/// nearly any model edit anyway, so the marginal cost is about one extra +/// `assemble_module`. +/// 2. **Looser on signed zero -> a theoretical stale read.** Unlike the AST, +/// `-0.0` IS reachable in a compiled literal pool: `compiler::fold` folds +/// `0 * -1` to `-0.0` (bits `0x8000_0000_0000_0000`), and `1.0 / -0.0` is +/// `-inf`, so the sign is observable. `vec![-0.0] == vec![0.0]`, so a pool +/// differing from its rebuild ONLY in a zero's sign compares equal, salsa +/// declines to overwrite, and the older pool is kept. Measured, not reasoned. +/// Reaching it needs a model edit that changes only a folded zero's sign in +/// an otherwise byte-identical fragment and then divides by it; it is +/// pre-existing and unreached by the corpus. It is NOT what #642 describes, +/// and it points the same way: converting those types would close both +/// directions at once and give up nothing, since bit comparison is free here +/// for the reasons above. +/// +/// **What would change the decision** (judgements about cost go stale, so name +/// the trigger): bytecode-fragment recompilation showing up in an interactive +/// profile, or any of these types acquiring a further tracked consumer that +/// makes the missed backdate cost more than one assembly. Either is a reason to +/// revisit; "it feels untidy" is not. +#[derive(Copy, Clone)] +pub struct Literal(f64); + +impl Literal { + /// Wrap a parsed numeric literal. + pub const fn new(value: f64) -> Self { + Literal(value) + } + + /// The numeric value, for arithmetic, folding, and code generation. + /// + /// Everything downstream of equality wants the plain `f64`; only comparison + /// and hashing go through the bit pattern. + pub const fn value(self) -> f64 { + self.0 + } +} + +/// Forwards to the inner `f64` so that a `Const` prints as `Const("NaN", NaN, +/// ..)` rather than gaining a wrapper layer in every AST debug dump. +/// +/// Unconditional rather than gated on `debug-derive` (the crate's rule for new +/// types) under that gate's documented exception for types appearing in +/// `assert_eq!` -- `literal_tests` compares `Literal`s directly, and a +/// one-field forwarding impl costs nothing. +impl std::fmt::Debug for Literal { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + std::fmt::Debug::fmt(&self.0, f) + } +} + +/// Bit-pattern equality: reflexive on NaN, and therefore a genuine equivalence +/// relation (hence the [`Eq`] impl below, which is what lets the AST enums +/// derive `Eq` and makes reflexivity a compile-checked property of every future +/// variant). +impl PartialEq for Literal { + fn eq(&self, other: &Self) -> bool { + self.0.to_bits() == other.0.to_bits() + } +} + +impl Eq for Literal {} + +/// Consistent with [`PartialEq`]: equal literals hash equal because both are +/// defined on the same bit pattern. +/// +/// Nothing hashes a `Literal` today -- none of the AST enums derives `Hash`. +/// It exists so that the obvious future change does not silently break the +/// `Hash`/`Eq` contract: `ltm_agg`'s synthetic-agg dedup map is keyed by printed +/// equation text precisely *because* `Expr2` is not `Hash`, and a `Hash` derived +/// over a bare `f64` would disagree with bit equality on NaN. +impl Hash for Literal { + fn hash(&self, state: &mut H) { + self.0.to_bits().hash(state); + } +} + +// `Literal` deliberately has NO `salsa::Update` impl. The `salsa::Update` +// derive emits `salsa::plumbing::UpdateDispatch::::maybe_update` for +// the field, under its own `use ::salsa::plumbing::UpdateFallback as _;` -- and +// that import is the load-bearing line. `Dispatch`'s inherent candidate +// requires `D: Update`, which `Literal` does not satisfy, so the call resolves +// to the `UpdateFallback` blanket impl for `'static + PartialEq`: salsa's own +// `update_fallback`, compare-and-replace against the `PartialEq` above. Writing +// an `unsafe impl Update` by hand would give the same behaviour with a second, +// independently-maintained definition of equality that could drift from +// `PartialEq`; letting the fallback apply means there is exactly one. +// +// What defends that, precisely -- because an `Update` impl would NOT break the +// build on its own; the inherent candidate would silently take over: +// +// * `#![deny(unsafe_code)]` (`lib.rs`) is what stops one being added in the +// first place, since `Update` is an unsafe trait. +// * If someone opted out with `#[allow(unsafe_code)]`, the `use UpdateFallback` +// below goes unused, and `cargo clippy -- -D warnings` (pre-commit and CI) +// turns that warning into an error. Verified by probe. +// * The assertion itself pins the weaker but still useful property that +// `maybe_update` resolves for `Literal` AT ALL -- i.e. that it stays +// `'static + PartialEq`, which is what the fallback needs. +const _: fn() = || { + use salsa::plumbing::{UpdateDispatch, UpdateFallback as _}; + let _resolves: unsafe fn(*mut Literal, Literal) -> bool = + UpdateDispatch::::maybe_update; +}; + +#[cfg(test)] +mod literal_tests { + use super::Literal; + use std::collections::hash_map::DefaultHasher; + use std::hash::{Hash, Hasher}; + + fn hash_of(lit: Literal) -> u64 { + let mut hasher = DefaultHasher::new(); + lit.hash(&mut hasher); + hasher.finish() + } + + /// The whole point: `f64`'s own equality is not reflexive on NaN, and this + /// type's is. + #[test] + fn a_nan_literal_equals_itself() { + // `black_box` keeps this from being a NaN comparison the compiler can + // see through and lint; the non-reflexivity of bare `f64` equality is + // exactly the defect being asserted, not an accident. + let bare = std::hint::black_box(f64::NAN); + assert!( + bare != std::hint::black_box(f64::NAN), + "bare f64 equality is not reflexive on NaN -- the defect this type exists to fix" + ); + assert_eq!(Literal::new(f64::NAN), Literal::new(f64::NAN)); + assert_eq!( + hash_of(Literal::new(f64::NAN)), + hash_of(Literal::new(f64::NAN)) + ); + } + + /// The one semantic change: bit equality separates the two zeros, which + /// IEEE equality does not. Asserted in BOTH directions so that neither the + /// separation nor the numeric equality it departs from can move silently. + #[test] + fn plus_and_minus_zero_are_different_literals() { + assert_eq!(0.0_f64, -0.0_f64, "IEEE equality does not distinguish them"); + assert_ne!(Literal::new(0.0), Literal::new(-0.0)); + assert_eq!(Literal::new(-0.0), Literal::new(-0.0)); + assert_eq!(Literal::new(0.0), Literal::new(0.0)); + assert_eq!( + Literal::new(-0.0).value().to_bits(), + (-0.0_f64).to_bits(), + "the value is passed through unchanged; only comparison differs" + ); + } + + /// Ordinary literals are unaffected -- the change must not make two + /// equation texts that used to compare equal stop doing so. + #[test] + fn ordinary_literals_compare_as_before() { + for value in [0.0_f64, 1.0, -1.0, 1e-300, 1e300, f64::INFINITY, 0.1 + 0.2] { + assert_eq!(Literal::new(value), Literal::new(value)); + assert_eq!(hash_of(Literal::new(value)), hash_of(Literal::new(value))); + } + assert_ne!(Literal::new(1.0), Literal::new(2.0)); + assert_ne!(Literal::new(f64::INFINITY), Literal::new(f64::NEG_INFINITY)); + } + + /// `maybe_update` -- the mechanism salsa actually backdates on -- and not + /// merely `PartialEq`. GH #981 measured the derive returning CHANGED for + /// identical NaN-bearing input, so this is checked directly at the leaf as + /// well as on the whole AST (`ast::literal_ast_tests`). + #[test] + fn maybe_update_reports_unchanged_for_an_identical_nan() { + use salsa::plumbing::{UpdateDispatch, UpdateFallback as _}; + + let mut slot = Literal::new(f64::NAN); + let changed = { + // SAFETY (test): `&mut slot` is a valid, owned `Literal` and the + // new value is a fresh owned one, matching the + // `Update::maybe_update` contract. + #[allow(unsafe_code)] + unsafe { + UpdateDispatch::::maybe_update(&mut slot, Literal::new(f64::NAN)) + } + }; + assert!(!changed, "an identical NaN literal must backdate"); + + let changed = { + // SAFETY (test): as above. + #[allow(unsafe_code)] + unsafe { + UpdateDispatch::::maybe_update(&mut slot, Literal::new(1.0)) + } + }; + assert!(changed, "a genuinely different literal must report CHANGED"); + assert_eq!(slot.value(), 1.0, "and must have written the new value"); + } +} + +/// The same property one level up, through the PUBLIC parse API: an `Expr0` +/// built twice from identical NaN-bearing text is equal to itself and backdates. +/// +/// `Expr0` is the layer that rides on `db::query::ParsedVariableResult` and +/// `ModelStage0` (GH #987) and on `db::ltm::LtmArm` (GH #981). The higher +/// layers are covered where they are consumed: `Expr2` by +/// `ltm_agg::tests::a_nan_literal_in_a_reducer_does_not_defeat_agg_backdating` +/// and by `db::stages_tests`' whole-project value oracle, and `LtmEquation` by +/// `db::ltm::equation`'s own probe. +#[cfg(test)] +mod literal_ast_tests { + use crate::ast::Expr0; + use crate::lexer::LexerType; + + fn parse(eqn: &str) -> Expr0 { + Expr0::new(eqn, LexerType::Equation) + .expect("the fixture equations parse") + .expect("the fixture equations are non-empty") + } + + #[test] + fn two_parses_of_a_nan_bearing_equation_are_equal() { + assert_eq!(parse("1 + nan"), parse("1 + nan")); + assert_eq!(parse("nan"), parse("nan")); + assert_ne!( + parse("1 + nan"), + parse("2 + nan"), + "a genuine difference must still be visible through a NaN-bearing tree" + ); + } + + /// `maybe_update` -- what salsa actually backdates on -- and not merely + /// `PartialEq`. The derive compares fields independently and, for the + /// literal, resolves to the `PartialEq` fallback; GH #981 measured this + /// returning CHANGED for identical NaN-bearing input. + #[test] + fn maybe_update_on_an_ast_reports_unchanged_for_an_identical_nan() { + // No `UpdateFallback` import here, unlike the `Literal` probe in + // `literal_tests`: `Expr0` derives `salsa::Update`, so the inherent + // dispatch applies and the fallback is not in play. That asymmetry is + // itself the evidence that `Literal` resolves through the `PartialEq` + // fallback -- importing the trait is what makes its method visible. + use salsa::plumbing::UpdateDispatch; + + let mut slot = parse("1 + nan"); + let changed = { + // SAFETY (test): `&mut slot` is a valid, owned `Expr0` and the new + // value is a fresh owned one, matching the `Update::maybe_update` + // contract. + #[allow(unsafe_code)] + unsafe { + UpdateDispatch::::maybe_update(&mut slot, parse("1 + nan")) + } + }; + assert!( + !changed, + "an identical NaN-bearing AST must backdate, or every downstream salsa \ + query re-executes on every revision bump" + ); + + let changed = { + // SAFETY (test): as above. + #[allow(unsafe_code)] + unsafe { + UpdateDispatch::::maybe_update(&mut slot, parse("2 + nan")) + } + }; + assert!( + changed, + "a genuinely edited equation must still report CHANGED -- otherwise \ + the test above could pass by making salsa blind" + ); + assert_eq!( + slot, + parse("2 + nan"), + "and the new value must have been written" + ); + } + + /// Why the `+0.0`/`-0.0` divergence introduced by bit comparison is not + /// reachable from equation text: `lexer::scan_number` never consumes a + /// leading sign, so `-0` parses as a NEGATION of the literal `0` and the + /// `Const` itself always holds a positive zero. The two spellings therefore + /// produce structurally different trees and were already unequal. + /// + /// This is a tripwire, not a contract: if the lexer ever learns signed + /// number tokens, the divergence becomes reachable and this reds. + #[test] + fn a_negative_zero_literal_is_a_negation_not_a_signed_const() { + use crate::ast::UnaryOp; + let neg = parse("-0"); + match &neg { + Expr0::Op1(UnaryOp::Negative, inner, _) => match inner.as_ref() { + Expr0::Const(text, value, _) => { + assert_eq!(text, "0"); + assert!( + value.value().is_sign_positive(), + "the lexer hands the parser an unsigned number token" + ); + } + other => panic!("expected a Const under the negation, got {other:?}"), + }, + other => panic!("expected `-0` to parse as a negation, got {other:?}"), + } + assert_ne!( + neg, + parse("0"), + "the two spellings differ structurally, independent of float equality" + ); + } +} diff --git a/src/simlin-engine/src/ast/mod.rs b/src/simlin-engine/src/ast/mod.rs index 4520727d3..279becf49 100644 --- a/src/simlin-engine/src/ast/mod.rs +++ b/src/simlin-engine/src/ast/mod.rs @@ -17,6 +17,7 @@ mod expr0; mod expr1; mod expr2; mod expr3; +mod literal; pub use array_view::{ArrayView, SparseInfo}; pub use expr0::{BinaryOp, Expr0, IndexExpr0, UnaryOp}; @@ -25,6 +26,7 @@ pub use expr1::Expr1; pub use expr2::{ArrayBounds, Expr2, Expr2Context, IndexExpr2}; #[allow(unused_imports)] pub use expr3::{Expr3, Expr3LowerContext, IndexExpr3, Pass1Context}; +pub use literal::Literal; #[cfg_attr(feature = "debug-derive", derive(Debug))] #[derive(Clone, PartialEq, Eq, salsa::Update)] @@ -699,7 +701,11 @@ fn test_print_eqn() { ); assert_eq!( "4.7", - print_eqn(&Expr0::Const("4.7".to_string(), 4.7, Loc::new(0, 3))) + print_eqn(&Expr0::Const( + "4.7".to_string(), + Literal::new(4.7), + Loc::new(0, 3) + )) ); assert_eq!( "lookup(a, 1.0)", @@ -708,7 +714,7 @@ fn test_print_eqn() { "lookup".to_string(), vec![ Expr0::Var(RawIdent::new_from_str("a"), Loc::new(7, 8)), - Expr0::Const("1.0".to_string(), 1.0, Loc::new(10, 13)) + Expr0::Const("1.0".to_string(), Literal::new(1.0), Loc::new(10, 13)) ] ), Loc::new(0, 14), @@ -757,8 +763,16 @@ fn t_op1(op: UnaryOp, inner: Expr0) -> Expr0 { fn t_if() -> Expr0 { Expr0::If( Box::new(t_var("a")), - Box::new(Expr0::Const("1".to_string(), 1.0, Loc::default())), - Box::new(Expr0::Const("0".to_string(), 0.0, Loc::default())), + Box::new(Expr0::Const( + "1".to_string(), + Literal::new(1.0), + Loc::default(), + )), + Box::new(Expr0::Const( + "0".to_string(), + Literal::new(0.0), + Loc::default(), + )), Loc::default(), ) } @@ -776,7 +790,7 @@ fn test_print_eqn_parenthesizes_if_under_an_operator() { assert_print_reparse_roundtrip( &t_op2( BinaryOp::Add, - Expr0::Const("1".to_string(), 1.0, Loc::default()), + Expr0::Const("1".to_string(), Literal::new(1.0), Loc::default()), t_if(), ), "1 + (if (a) then (1) else (0))", @@ -983,7 +997,7 @@ impl LatexVisitor { fn walk(&mut self, expr: &Expr2) -> String { match expr { Expr2::Const(s, n, _) => { - if n.is_nan() { + if n.value().is_nan() { "\\mathrm{{NaN}}".to_owned() } else { s.clone() @@ -1062,7 +1076,7 @@ pub fn latex_eqn(expr: &Expr2) -> String { pub fn latex_eqn_expr0(expr: &Expr0) -> String { match expr { Expr0::Const(s, n, _) => { - if n.is_nan() { + if n.value().is_nan() { "\\mathrm{{NaN}}".to_owned() } else { s.clone() @@ -1168,7 +1182,7 @@ pub fn latex_eqn_expr0_annotated(expr: &Expr0) -> String { let loc = expr.get_loc(); let inner = match expr { Expr0::Const(s, n, _) => { - if n.is_nan() { + if n.value().is_nan() { "\\mathrm{{NaN}}".to_owned() } else { s.clone() @@ -1358,7 +1372,11 @@ fn test_latex_eqn() { Box::new(Expr2::Op2( BinaryOp::Sub, Box::new(Expr2::Var(Ident::new("a_c"), None, Loc::new(0, 0))), - Box::new(Expr2::Const("1".to_string(), 1.0, Loc::new(0, 0))), + Box::new(Expr2::Const( + "1".to_string(), + Literal::new(1.0), + Loc::new(0, 0) + )), None, Loc::new(0, 0), )), @@ -1375,7 +1393,11 @@ fn test_latex_eqn() { Box::new(Expr2::Op2( BinaryOp::Sub, Box::new(Expr2::Var(Ident::new("a_c"), None, Loc::new(0, 0))), - Box::new(Expr2::Const("1".to_string(), 1.0, Loc::new(0, 0))), + Box::new(Expr2::Const( + "1".to_string(), + Literal::new(1.0), + Loc::new(0, 0) + )), None, Loc::new(0, 0), )), @@ -1412,14 +1434,22 @@ fn test_latex_eqn() { ); assert_eq!( "4.7", - latex_eqn(&Expr2::Const("4.7".to_string(), 4.7, Loc::new(0, 3))) + latex_eqn(&Expr2::Const( + "4.7".to_string(), + Literal::new(4.7), + Loc::new(0, 3) + )) ); assert_eq!( "\\operatorname{lookup}(\\mathrm{a}, 1.0)", latex_eqn(&Expr2::App( crate::builtins::BuiltinFn::Lookup( Box::new(Expr2::Var(Ident::new("a"), None, Default::default())), - Box::new(Expr2::Const("1.0".to_owned(), 1.0, Default::default())), + Box::new(Expr2::Const( + "1.0".to_owned(), + Literal::new(1.0), + Default::default() + )), Default::default(), ), None, @@ -1530,8 +1560,16 @@ fn test_latex_printers_agree_on_if_under_an_operator() { let cases2 = Expr2::If( Box::new(Expr2::Var(Ident::new("a"), None, Loc::default())), - Box::new(Expr2::Const("1".to_string(), 1.0, Loc::default())), - Box::new(Expr2::Const("0".to_string(), 0.0, Loc::default())), + Box::new(Expr2::Const( + "1".to_string(), + Literal::new(1.0), + Loc::default(), + )), + Box::new(Expr2::Const( + "0".to_string(), + Literal::new(0.0), + Loc::default(), + )), None, Loc::default(), ); @@ -1665,7 +1703,7 @@ mod print_eqn_proptest { .prop_map(|n| Expr0::Var(RawIdent::new_from_str(n), Loc::default())), prop::sample::select(&[0.0f64, 1.0, 2.5][..]).prop_map(|v| Expr0::Const( format!("{v}"), - v, + Literal::new(v), Loc::default() )), ]; @@ -1981,7 +2019,11 @@ mod ast_tests { // Test if expression with mismatched dimensions let if_expr = Expr1::If( - Box::new(Expr1::Const("1".to_string(), 1.0, Loc::default())), + Box::new(Expr1::Const( + "1".to_string(), + Literal::new(1.0), + Loc::default(), + )), Box::new(Expr1::Var(Ident::new("regional_data"), Loc::default())), Box::new(Expr1::Var(Ident::new("product_data"), Loc::default())), Loc::new(0, 20), diff --git a/src/simlin-engine/src/builtins_visitor.rs b/src/simlin-engine/src/builtins_visitor.rs index 4af8e93cb..aafac368c 100644 --- a/src/simlin-engine/src/builtins_visitor.rs +++ b/src/simlin-engine/src/builtins_visitor.rs @@ -5,7 +5,7 @@ use std::collections::{HashMap, HashSet}; use std::sync::LazyLock; -use crate::ast::{Ast, BinaryOp, Expr0, IndexExpr0, print_eqn}; +use crate::ast::{Ast, BinaryOp, Expr0, IndexExpr0, Literal, print_eqn}; use crate::builtins::{UntypedBuiltinFn, is_builtin_fn}; use crate::common::{ Canonical, CanonicalDimensionName, CanonicalElementName, EquationError, Ident, RawIdent, @@ -73,8 +73,9 @@ pub(crate) fn contains_module_call(expr: &Expr0, macro_registry: &MacroRegistry) fn parse_module_order_arg(expr: &Expr0) -> Option { if let Expr0::Const(_, n, _) = expr { + let n = n.value(); let rounded = n.round(); - if (*n - rounded).abs() < 1e-9 && rounded >= 0.0 { + if (n - rounded).abs() < 1e-9 && rounded >= 0.0 { return Some(rounded as u32); } } @@ -483,7 +484,7 @@ impl<'a> BuiltinVisitor<'a> { // For indexed dimensions, the subscript element is a number // Use it directly as a Const let val: f64 = subscript[i].parse().unwrap_or(0.0); - return Const(subscript[i].clone(), val, loc); + return Const(subscript[i].clone(), Literal::new(val), loc); } Dimension::Named(_, _) => { // For named dimensions, use qualified element (dimension·element). @@ -1040,7 +1041,7 @@ impl<'a> BuiltinVisitor<'a> { } let args = if func == "previous" && args.len() == 1 { let mut args = args; - args.push(Const("0".to_string(), 0.0, loc)); + args.push(Const("0".to_string(), Literal::new(0.0), loc)); args } else { args diff --git a/src/simlin-engine/src/bytecode.rs b/src/simlin-engine/src/bytecode.rs index 7bb7a7c0a..dd6ace017 100644 --- a/src/simlin-engine/src/bytecode.rs +++ b/src/simlin-engine/src/bytecode.rs @@ -1641,6 +1641,14 @@ impl StaticArrayView { /// Context data shared across all bytecode runlists in a module. /// Contains tables that opcodes reference by index. +/// +/// Its graphical-function `f64`s are compared with the DERIVED (IEEE) +/// `PartialEq`, not by bit pattern: a NaN makes this value unequal to a +/// bit-identical rebuild and so unable to backdate, and two tables differing +/// only in a zero's sign compare equal. Both are accepted, knowingly -- see the +/// "Float equality in this crate" section on [`crate::ast::Literal`] for the +/// position, the corrected premise behind GH #642, and what would change the +/// decision. #[cfg_attr(feature = "debug-derive", derive(Debug))] #[derive(Clone, Default, PartialEq)] pub struct ByteCodeContext { @@ -1758,6 +1766,14 @@ impl ByteCodeContext { } } +/// One runlist's concrete opcode stream plus the literal pool it indexes. +/// +/// The pool's `f64`s are compared with the DERIVED (IEEE) `PartialEq`, not by +/// bit pattern: a NaN makes this value unequal to a bit-identical rebuild and +/// so unable to backdate, and two pools differing only in a zero's sign compare +/// equal. Both are accepted, knowingly -- see the "Float equality in this +/// crate" section on [`crate::ast::Literal`] for the position, the corrected +/// premise behind GH #642, and what would change the decision. #[cfg_attr(feature = "debug-derive", derive(Debug))] #[derive(Clone, Default, PartialEq)] pub struct ByteCode { diff --git a/src/simlin-engine/src/compiler/context.rs b/src/simlin-engine/src/compiler/context.rs index 865d237de..77da0020c 100644 --- a/src/simlin-engine/src/compiler/context.rs +++ b/src/simlin-engine/src/compiler/context.rs @@ -1321,7 +1321,7 @@ impl Context<'_> { } // Handle common variants directly (no longer converting to Expr2) - Expr3::Const(_, n, loc) => Ok(Expr::Const(*n, *loc)), + Expr3::Const(_, n, loc) => Ok(Expr::Const(n.value(), *loc)), Expr3::Var(id, _, loc) => { // Check if this identifier is a dimension name @@ -2859,8 +2859,16 @@ fn test_lower() { None, Loc::default(), )), - Box::new(Const("1".to_string(), 1.0, Loc::default())), - Box::new(Const("0".to_string(), 0.0, Loc::default())), + Box::new(Const( + "1".to_string(), + crate::ast::Literal::new(1.0), + Loc::default(), + )), + Box::new(Const( + "0".to_string(), + crate::ast::Literal::new(0.0), + Loc::default(), + )), None, Loc::default(), )) @@ -2966,8 +2974,16 @@ fn test_lower() { None, Loc::default(), )), - Box::new(Const("1".to_string(), 1.0, Loc::default())), - Box::new(Const("0".to_string(), 0.0, Loc::default())), + Box::new(Const( + "1".to_string(), + crate::ast::Literal::new(1.0), + Loc::default(), + )), + Box::new(Const( + "0".to_string(), + crate::ast::Literal::new(0.0), + Loc::default(), + )), None, Loc::default(), )) @@ -3190,7 +3206,11 @@ fn test_positional_fallback_ignores_unrelated_mapping() { ident: Ident::new("source_var"), ast: Some(ast::Ast::ApplyToAll( vec![Dimension::from(&source)], - ast::Expr2::Const("0".to_string(), 0.0, Loc::default()), + ast::Expr2::Const( + "0".to_string(), + crate::ast::Literal::new(0.0), + Loc::default(), + ), )), init_ast: None, eqn: None, diff --git a/src/simlin-engine/src/compiler/mod.rs b/src/simlin-engine/src/compiler/mod.rs index 2a7403435..bfc316cb4 100644 --- a/src/simlin-engine/src/compiler/mod.rs +++ b/src/simlin-engine/src/compiler/mod.rs @@ -652,11 +652,19 @@ fn test_arrayed_default_equation_applies_to_missing_elements() { let mut elements = HashMap::new(); elements.insert( CanonicalElementName::from_raw("a"), - crate::ast::Expr2::Const("1".to_string(), 1.0, Loc::default()), + crate::ast::Expr2::Const( + "1".to_string(), + crate::ast::Literal::new(1.0), + Loc::default(), + ), ); elements.insert( CanonicalElementName::from_raw("b"), - crate::ast::Expr2::Const("2".to_string(), 2.0, Loc::default()), + crate::ast::Expr2::Const( + "2".to_string(), + crate::ast::Literal::new(2.0), + Loc::default(), + ), ); let var = Variable::Var { @@ -666,7 +674,7 @@ fn test_arrayed_default_equation_applies_to_missing_elements() { elements, Some(crate::ast::Expr2::Const( "7".to_string(), - 7.0, + crate::ast::Literal::new(7.0), Loc::default(), )), true, diff --git a/src/simlin-engine/src/compiler/subscript.rs b/src/simlin-engine/src/compiler/subscript.rs index c03560ba4..1024bcece 100644 --- a/src/simlin-engine/src/compiler/subscript.rs +++ b/src/simlin-engine/src/compiler/subscript.rs @@ -150,7 +150,7 @@ pub(crate) fn normalize_subscripts3( match expr { Expr3::Const(_, val, _) => { // Numeric constant - convert from 1-based to 0-based - Some((*val as isize - 1).max(0) as usize) + Some((val.value() as isize - 1).max(0) as usize) } Expr3::Var(ident, _, _) => { // Could be a named dimension element - use O(1) hash lookup @@ -182,7 +182,7 @@ pub(crate) fn normalize_subscripts3( IndexExpr3::Expr(expr) => { match expr { Expr3::Const(_, val, _) => { - let idx = (*val as isize - 1).max(0) as usize; + let idx = (val.value() as isize - 1).max(0) as usize; IndexOp::Single(idx) } Expr3::Var(ident, _, _) => { diff --git a/src/simlin-engine/src/compiler/symbolic.rs b/src/simlin-engine/src/compiler/symbolic.rs index 34c4b6998..f2324323f 100644 --- a/src/simlin-engine/src/compiler/symbolic.rs +++ b/src/simlin-engine/src/compiler/symbolic.rs @@ -294,6 +294,13 @@ pub(crate) enum SymbolicOpcode { /// Symbolic version of `ByteCode`. Contains the literal pool (unchanged) /// and symbolic opcodes. +/// +/// Its `f64`s are compared with the DERIVED (IEEE) `PartialEq`, not by bit +/// pattern: a NaN makes this value unequal to a bit-identical rebuild and so +/// unable to backdate, and two pools differing only in a zero's sign compare +/// equal. Both are accepted, knowingly -- see the "Float equality in this +/// crate" section on [`crate::ast::Literal`] for the position, the corrected +/// premise behind GH #642, and what would change the decision. #[derive(Clone, Debug, Default, PartialEq)] pub(crate) struct SymbolicByteCode { pub literals: Vec, @@ -383,6 +390,13 @@ pub(crate) struct CompiledVarFragment { } /// Bytecodes plus side-channel data for one variable in one phase. +/// +/// This is the value of the TRACKED `db::compile_var_fragment`, read by the +/// TRACKED `db::assemble_module` -- which is why GH #642's "the only consumer is +/// non-tracked" reasoning does not hold here. Its floats (the literal pool +/// inside `symbolic`, and `graphical_functions`) keep the derived IEEE +/// `PartialEq`; see the "Float equality in this crate" section on +/// [`crate::ast::Literal`] for why that is accepted rather than open. #[derive(Clone, Debug, PartialEq)] pub(crate) struct PerVarBytecodes { pub symbolic: SymbolicByteCode, diff --git a/src/simlin-engine/src/conveyor_compile.rs b/src/simlin-engine/src/conveyor_compile.rs index 053d62a9f..4faaefde5 100644 --- a/src/simlin-engine/src/conveyor_compile.rs +++ b/src/simlin-engine/src/conveyor_compile.rs @@ -685,7 +685,7 @@ pub(crate) fn const_scalar_expr(expr: &str) -> Option { use crate::ast::{Expr0, UnaryOp}; fn eval(ast: &Expr0) -> Option { match ast { - Expr0::Const(_, v, _) => Some(*v), + Expr0::Const(_, v, _) => Some(v.value()), Expr0::Op1(UnaryOp::Positive, inner, _) => eval(inner), Expr0::Op1(UnaryOp::Negative, inner, _) => eval(inner).map(|v| -v), _ => None, @@ -1815,8 +1815,9 @@ fn const_slat_index(idx: &crate::ast::IndexExpr0) -> Option { let IndexExpr0::Expr(Expr0::Const(_, val, _)) = idx else { return None; }; - if val.is_finite() && *val >= 0.0 && val.fract() == 0.0 { - Some(*val as usize) + let val = val.value(); + if val.is_finite() && val >= 0.0 && val.fract() == 0.0 { + Some(val as usize) } else { None } diff --git a/src/simlin-engine/src/db/assemble.rs b/src/simlin-engine/src/db/assemble.rs index ad9d1c44d..124100abc 100644 --- a/src/simlin-engine/src/db/assemble.rs +++ b/src/simlin-engine/src/db/assemble.rs @@ -146,7 +146,11 @@ pub(crate) fn build_stub_variable( } else { Some(crate::ast::Ast::ApplyToAll( dims.to_vec(), - crate::ast::Expr2::Const("0".to_string(), 0.0, crate::ast::Loc::default()), + crate::ast::Expr2::Const( + "0".to_string(), + crate::ast::Literal::new(0.0), + crate::ast::Loc::default(), + ), )) }; @@ -286,7 +290,11 @@ pub(crate) fn build_submodel_metadata<'arena>( .collect(); Some(crate::ast::Ast::ApplyToAll( dims, - crate::ast::Expr2::Const("0".to_string(), 0.0, crate::ast::Loc::default()), + crate::ast::Expr2::Const( + "0".to_string(), + crate::ast::Literal::new(0.0), + crate::ast::Loc::default(), + ), )) }; let stub: &'arena crate::variable::Variable = diff --git a/src/simlin-engine/src/db/fragment_compile.rs b/src/simlin-engine/src/db/fragment_compile.rs index 5a18e47a3..6ca3a3185 100644 --- a/src/simlin-engine/src/db/fragment_compile.rs +++ b/src/simlin-engine/src/db/fragment_compile.rs @@ -827,7 +827,11 @@ pub(crate) fn compile_implicit_var_phase_bytecodes( } else { Some(crate::ast::Ast::ApplyToAll( dep_dims, - crate::ast::Expr2::Const("0".to_string(), 0.0, crate::ast::Loc::default()), + crate::ast::Expr2::Const( + "0".to_string(), + crate::ast::Literal::new(0.0), + crate::ast::Loc::default(), + ), )) }; let dep_var = if is_stock { diff --git a/src/simlin-engine/src/db/ltm/compile.rs b/src/simlin-engine/src/db/ltm/compile.rs index 45c933ff5..d6b85a1b6 100644 --- a/src/simlin-engine/src/db/ltm/compile.rs +++ b/src/simlin-engine/src/db/ltm/compile.rs @@ -1367,7 +1367,7 @@ pub(crate) fn compile_ltm_equation_fragment( canonical_dims, crate::ast::Expr2::Const( "0".to_string(), - 0.0, + crate::ast::Literal::new(0.0), crate::ast::Loc::default(), ), )) @@ -1431,7 +1431,7 @@ pub(crate) fn compile_ltm_equation_fragment( canonical_dims, crate::ast::Expr2::Const( "0".to_string(), - 0.0, + crate::ast::Literal::new(0.0), crate::ast::Loc::default(), ), )) @@ -2569,7 +2569,11 @@ mod pass1_gate_tests { use crate::common::{Canonical, Ident}; fn c() -> Box { - Box::new(Expr2::Const("0".to_string(), 0.0, Loc::default())) + Box::new(Expr2::Const( + "0".to_string(), + crate::ast::Literal::new(0.0), + Loc::default(), + )) } fn app(builtin: BuiltinFn) -> Expr2 { diff --git a/src/simlin-engine/src/db/ltm/equation.rs b/src/simlin-engine/src/db/ltm/equation.rs index 784048b1f..8b234d148 100644 --- a/src/simlin-engine/src/db/ltm/equation.rs +++ b/src/simlin-engine/src/db/ltm/equation.rs @@ -360,3 +360,78 @@ impl LtmEquation { } } } + +#[cfg(test)] +mod tests { + use super::{LtmArm, LtmEquation}; + + /// GH #981's exact measurement, inverted. + /// + /// The issue's throwaway probe recorded that two `LtmArm::new("NaN")` were + /// UNEQUAL, that two `LtmEquation::scalar("1 + NaN")` were UNEQUAL, and -- + /// the part that mattered -- that `:: + /// maybe_update` returned CHANGED for identical text, so the backdating + /// mechanism itself was affected and not merely `PartialEq`. Since the + /// literal on `Expr0::Const` is an `ast::Literal` compared by bit pattern, + /// all three are equal/UNCHANGED, and `link_score_equation_text_shaped` can + /// backdate so the expensive `compile_ltm_var_fragment` is reused. + /// + /// The controls (an ordinary equation, and a genuinely edited one) are what + /// keep this from passing by making salsa blind. + #[test] + fn a_nan_bearing_ltm_equation_is_equal_to_itself_and_backdates() { + assert!( + LtmArm::new("NaN".to_string()).expr.is_some(), + "the fixture text must parse, or nothing below measures the AST" + ); + assert_eq!( + LtmArm::new("NaN".to_string()), + LtmArm::new("NaN".to_string()) + ); + assert_eq!( + LtmEquation::scalar("1 + NaN".to_string()), + LtmEquation::scalar("1 + NaN".to_string()) + ); + assert_eq!( + LtmEquation::scalar("1 + 2".to_string()), + LtmEquation::scalar("1 + 2".to_string()), + "control: an ordinary equation was always equal to itself" + ); + assert_ne!( + LtmEquation::scalar("1 + NaN".to_string()), + LtmEquation::scalar("2 + NaN".to_string()), + "control: a genuine difference must still be visible" + ); + + use salsa::plumbing::UpdateDispatch; + let mut slot = LtmEquation::scalar("1 + NaN".to_string()); + let changed = { + // SAFETY (test): `&mut slot` is a valid, owned `LtmEquation` and the + // new value is a fresh owned one, matching the + // `Update::maybe_update` contract. + #[allow(unsafe_code)] + unsafe { + UpdateDispatch::::maybe_update( + &mut slot, + LtmEquation::scalar("1 + NaN".to_string()), + ) + } + }; + assert!( + !changed, + "identical NaN-bearing LTM equation text must backdate (GH #981)" + ); + + let changed = { + // SAFETY (test): as above. + #[allow(unsafe_code)] + unsafe { + UpdateDispatch::::maybe_update( + &mut slot, + LtmEquation::scalar("2 + NaN".to_string()), + ) + } + }; + assert!(changed, "a genuinely edited equation must report CHANGED"); + } +} diff --git a/src/simlin-engine/src/db/stages_tests.rs b/src/simlin-engine/src/db/stages_tests.rs index 14aa3557b..e6d6a756e 100644 --- a/src/simlin-engine/src/db/stages_tests.rs +++ b/src/simlin-engine/src/db/stages_tests.rs @@ -40,12 +40,16 @@ //! which sorted Debug strings and was sound only by the accident that its //! fixtures had no `Equation::Arrayed`. //! -//! The rule: compare with `PartialEq`. If you must reach for Debug text because -//! a NaN literal defeats `PartialEq` (GH #987/#981 -- `ModelStage0` compares -//! parsed `f64` constants, and every SMOOTH/DELAY/TREND template declares -//! `initial_value = NAN`, so such a stage is not even equal to itself), then it -//! is sound ONLY for fixtures with no per-element equations, and the test must -//! say so. +//! The rule: compare with `PartialEq`, always. The one thing that used to force +//! a Debug-text oracle is gone -- a stage carrying a NaN EQUATION LITERAL was +//! not equal even to ITSELF, because `ModelStage0` compares parsed float +//! constants and every SMOOTH/DELAY/TREND template declares +//! `initial_value = NAN`. Since GH #987/#981 those constants are +//! `ast::Literal`, compared by bit pattern, so every stage here -- stdlib +//! templates included -- equals a bit-identical rebuild. One float-bearing +//! field on these memos is still outside that: `variable::Table`'s `Vec` +//! lookup points (see `ast::Literal`'s scope note). No fixture in this file has +//! a graphical function, let alone a NaN in one, so the rule holds here. use super::*; use crate::common::{Canonical, ErrorCode, Ident}; @@ -546,14 +550,19 @@ fn omitting_stdlib_models_from_the_lowering_scope_is_inert_today() { /// `errors` (the duplicate-ident pair), `implicit`, `is_macro` and /// `macro_params`. /// -/// The comparison ranges over the USER models only. The stdlib templates are in -/// the project -- `main` instantiates `stdlib⁚smth1` and both scopes hold all of -/// them -- but SMOOTH/DELAY/TREND bodies declare `initial_value = NAN`, and -/// `ModelStage0` derives `PartialEq`, so a stdlib stage carrying a NaN literal -/// does not even compare equal to ITSELF. Asserting on one would fail for a -/// reason unrelated to what is being tested (GH #987/#981); -/// `cached_stdlib_stage0_equals_implicit_datamodel_build` covers the stdlib side -/// on `npv`, the one template with no NaN. +/// The comparison ranges over EVERY model the sync produced -- the three user +/// models and all nine spliced stdlib templates. It used to be restricted to +/// the user models because five of the nine templates declare +/// `initial_value = NAN` and a stage holding a bare `f64` NaN did not compare +/// equal even to ITSELF, so a stdlib assertion failed for a reason unrelated to +/// what is being tested. `ast::Literal` compares float literals by bit pattern +/// (GH #987/#981), so those stages are now ordinary values and the oracle can +/// cover them. +/// +/// The row-count assertion is not about a newly added stdlib template -- both +/// sides derive their stdlib half from `stdlib::MODEL_NAMES`, so one of those +/// is covered automatically. What it catches is `every_shape_project()` gaining +/// or losing a USER model that the hard-coded list below does not follow. #[test] fn cached_stages_equal_the_datamodel_driven_build_for_every_model_shape() { let db = SimlinDb::default(); @@ -568,7 +577,19 @@ fn cached_stages_equal_the_datamodel_driven_build_for_every_model_shape() { let oracle_s0: HashMap<&Ident, &ModelStage0> = all_s0.iter().map(|m| (&m.ident, m)).collect(); - for name in ["main", "sub", "scaled"] { + let mut names: Vec = vec!["main".to_string(), "sub".to_string(), "scaled".to_string()]; + names.extend( + crate::stdlib::MODEL_NAMES + .iter() + .map(|n| format!("stdlib\u{205A}{n}")), + ); + assert_eq!( + names.len(), + oracle_s0.len(), + "every model the oracle builds must be asserted on" + ); + for name in &names { + let name = name.as_str(); let ident: Ident = Ident::new(name); let source = sync.models[name].source; assert!( @@ -793,11 +814,16 @@ fn model_stage0_parses_a_stdlib_model_under_the_extended_module_context() { /// decisions above (`implicit: true`, and every variable name in the /// module-ident set). /// -/// `npv` is the fixture rather than the more representative `smth1` because -/// `ModelStage0`'s derived `PartialEq` compares the parsed `f64` constants, and -/// every SMOOTH/DELAY/TREND template declares `initial_value = NAN`: `NaN != -/// NaN` makes those stages unequal even to a bit-identical rebuild. `npv` -/// carries no NaN literal, so equality is meaningful there. +/// The fixture is `smth1`, the representative SMOOTH template. It used to have +/// to be `npv` -- the one template with no NaN literal -- because +/// `ModelStage0`'s derived `PartialEq` compared bare parsed `f64` constants and +/// every SMOOTH/DELAY/TREND template declares `initial_value = NAN`, so those +/// stages were unequal even to a bit-identical rebuild. `ast::Literal` compares +/// float literals by bit pattern (GH #987/#981), so a stage whose NaN is an +/// EQUATION LITERAL is equal to itself and the natural fixture works. (A NaN +/// arriving through a graphical function's points is a different field and is +/// still non-reflexive -- see `ast::Literal`'s scope note; no stdlib template +/// has a graphical function.) #[test] fn cached_stdlib_stage0_equals_implicit_datamodel_build() { let db = SimlinDb::default(); @@ -807,15 +833,26 @@ fn cached_stdlib_stage0_equals_implicit_datamodel_build() { let project = x_project(sim_specs_with_units("month"), &[main]); let sync = sync_from_datamodel(&db, &project); - let source = sync.models["stdlib\u{205A}npv"].source; + let source = sync.models["stdlib\u{205A}smth1"].source; let cached = model_stage0(&db, source, sync.project); assert!(cached.implicit, "a stdlib model's Stage0 is implicit"); assert!( cached.variables.len() > 1, "the fixture stdlib model has a body to compare" ); + assert!( + cached.variables.values().any(|v| { + v.ast().is_some_and(|ast| match ast { + crate::ast::Ast::Scalar(e) => { + matches!(e, crate::ast::Expr0::Const(_, n, _) if n.value().is_nan()) + } + _ => false, + }) + }), + "the fixture must actually carry the NaN literal this test is here to cover" + ); - let stdlib_dm = crate::stdlib::get("npv").expect("npv is a stdlib model"); + let stdlib_dm = crate::stdlib::get("smth1").expect("smth1 is a stdlib model"); let oracle = ModelStage0::new( &stdlib_dm, project_datamodel_dims(&db, sync.project), @@ -1593,6 +1630,112 @@ fn an_unrelated_models_edit_invalidates_neither_stage_nor_unit_check() { ); } +/// A model carrying a USER-AUTHORED NaN literal backdates its stage exactly +/// like one without: a revision bump that re-executes `model_stage0` and +/// rebuilds a bit-identical value must NOT re-execute `model_stage1`. +/// +/// This is the reachable half of GH #987's EQUATION-LITERAL path. `ModelStage0` +/// derives `PartialEq` and holds parsed float constants; with a bare `f64` a +/// NaN-bearing stage was never equal to its own rebuild, so salsa could never +/// backdate it and every downstream query in the cone re-ran on every revision +/// bump -- on the interactive diagnostics path, per keystroke. The issue's own +/// reachability argument rests on the stdlib `initial_value = NAN`, but that +/// half is inert (those inputs never change after sync), so the fixture here is +/// a user equation, which is the shape that actually pays. +/// +/// A NaN reaching the same memo through a GRAPHICAL FUNCTION's points is a +/// different, still-unfixed field (`variable::Table`'s `Vec`); this lever +/// measures that one identically, which is how it was confirmed. See +/// `ast::Literal`'s scope note. +/// +/// The lever is the project's UNIT table: `model_stage0` reads +/// `project_units_context` and `model_stage1` does not, so adding an unrelated +/// unit re-executes stage0 and leaves stage1's re-execution decided purely by +/// whether the rebuilt stage compares equal to the old one. +/// +/// Two rows, derived from the one axis the change is about -- whether the +/// model's equations carry a NaN literal -- and they must produce identical +/// counts. The control row is what makes the NaN row attributable: under the +/// mutation probe (bare `f64` equality inside `ast::Literal`) the control stays +/// green and the NaN row reds. +#[test] +fn a_nan_bearing_models_stage_backdates_like_any_other() { + // The control row runs FIRST so that a mutation probe fails on the NaN row + // with the control already green -- attribution, not just a red test. + for (label, eqn) in [("control", "1 + 2"), ("nan literal", "1 + nan")] { + let build = |extra_unit: bool| { + let mut project = x_project( + sim_specs_with_units("month"), + &[x_model("main", vec![x_aux("x", eqn, Some("widget"))])], + ); + if extra_unit { + project.units.push(datamodel::Unit { + name: "gizmo".to_owned(), + equation: None, + disabled: false, + aliases: vec![], + }); + } + project + }; + + let mut db = SimlinDb::default(); + let state1 = sync_from_datamodel_incremental(&mut db, &build(false), None); + let sync1 = state1.to_sync_result(); + let main = sync1.models["main"].source; + let _ = model_stage1(&db, main, sync1.project); + + // Guard against a vacuous pass: the NaN row's stage really does hold a + // NaN literal, so the equality being measured is the one at issue. + let holds_nan = model_stage0(&db, main, sync1.project) + .variables + .values() + .any(|v| { + v.ast().is_some_and(|ast| match ast { + crate::ast::Ast::Scalar(e) => expr0_holds_nan(e), + _ => false, + }) + }); + assert_eq!( + holds_nan, + label == "nan literal", + "{label}: the fixture must carry a NaN literal iff it is the NaN row" + ); + + reset_query_executions(); + let state2 = sync_from_datamodel_incremental(&mut db, &build(true), Some(&state1)); + let sync2 = state2.to_sync_result(); + let _ = model_stage1(&db, sync2.models["main"].source, sync2.project); + assert_eq!( + query_executions(), + QueryExecutions { + stage0: 1, + stage1: 0, + unit_check: 0, + }, + "{label}: the units edit must re-execute stage0 and, because the rebuilt \ + stage is bit-identical, backdate it so stage1 is reused" + ); + } +} + +/// Does this `Expr0` tree contain a NaN literal? Used only by the fixture guard +/// above. +fn expr0_holds_nan(expr: &crate::ast::Expr0) -> bool { + use crate::ast::Expr0; + match expr { + Expr0::Const(_, n, _) => n.value().is_nan(), + Expr0::Var(_, _) => false, + Expr0::App(crate::builtins::UntypedBuiltinFn(_, args), _) => { + args.iter().any(expr0_holds_nan) + } + Expr0::Subscript(_, _, _) => false, + Expr0::Op1(_, inner, _) => expr0_holds_nan(inner), + Expr0::Op2(_, l, r, _) => expr0_holds_nan(l) || expr0_holds_nan(r), + Expr0::If(c, t, f, _) => expr0_holds_nan(c) || expr0_holds_nan(t) || expr0_holds_nan(f), + } +} + /// Editing a model that `main` instantiates DOES re-execute `main`'s lowered /// stage and unit check. /// diff --git a/src/simlin-engine/src/db/tests.rs b/src/simlin-engine/src/db/tests.rs index 90f92846c..76eb1ccc9 100644 --- a/src/simlin-engine/src/db/tests.rs +++ b/src/simlin-engine/src/db/tests.rs @@ -466,7 +466,7 @@ fn test_parse_source_variable_scalar() { let ast = parsed.variable.ast(); assert!(ast.is_some()); if let Some(crate::ast::Ast::Scalar(Expr0::Const(_, val, _))) = ast { - assert_eq!(*val, 100.0); + assert_eq!(*val, crate::ast::Literal::new(100.0)); } else { panic!("Expected Scalar(Const(100.0)), got {:?}", ast); } @@ -655,7 +655,7 @@ fn test_incrementality_unchanged_variable_not_reparsed() { if let Some(crate::ast::Ast::Scalar(crate::ast::Expr0::Const(_, val, _))) = alpha_result_2.variable.ast() { - assert_eq!(*val, 42.0); + assert_eq!(*val, crate::ast::Literal::new(42.0)); } else { panic!( "Expected alpha to parse as Const(42.0), got {:?}", @@ -2643,7 +2643,7 @@ fn test_incremental_sync_preserves_cache_for_unchanged_variable() { if let Some(crate::ast::Ast::Scalar(crate::ast::Expr0::Const(_, val, _))) = alpha_result.variable.ast() { - assert_eq!(*val, 42.0); + assert_eq!(*val, crate::ast::Literal::new(42.0)); } else { panic!( "Expected alpha to parse as Const(42.0), got {:?}", @@ -2818,7 +2818,7 @@ fn test_incremental_sync_successive_patches() { if let Some(crate::ast::Ast::Scalar(crate::ast::Expr0::Const(_, val, _))) = result.variable.ast() { - assert_eq!(*val, 999.0); + assert_eq!(*val, crate::ast::Literal::new(999.0)); } else { panic!("Expected Const(999.0), got {:?}", result.variable.ast()); } diff --git a/src/simlin-engine/src/db/var_fragment.rs b/src/simlin-engine/src/db/var_fragment.rs index e6fd7dd49..42d4954bf 100644 --- a/src/simlin-engine/src/db/var_fragment.rs +++ b/src/simlin-engine/src/db/var_fragment.rs @@ -395,7 +395,11 @@ fn collect_var_dependencies( } else { Some(crate::ast::Ast::ApplyToAll( dep_dims, - crate::ast::Expr2::Const("0".to_string(), 0.0, crate::ast::Loc::default()), + crate::ast::Expr2::Const( + "0".to_string(), + crate::ast::Literal::new(0.0), + crate::ast::Loc::default(), + ), )) }; let dep_var = if meta.is_stock { diff --git a/src/simlin-engine/src/float.rs b/src/simlin-engine/src/float.rs index 737341be8..40b2671ea 100644 --- a/src/simlin-engine/src/float.rs +++ b/src/simlin-engine/src/float.rs @@ -2,7 +2,64 @@ // Use of this source code is governed by the Apache License, // Version 2.0, that can be found in the LICENSE file. -//! Floating-point utility functions for the simulation engine. +//! Floating-point utility functions for the simulation engine, and the two +//! "this number is not ordinary data" conventions the engine works with. +//! +//! Those two are best read together, because they pull in opposite directions: +//! [`NA`] is Vensim's "missing data" sentinel and is FINITE and comparable **on +//! purpose**, while a NaN is a propagating failure signal. If you are asking why +//! this codebase treats NaN the way it does, the answer is below. +//! +//! # What a NaN means here: a diagnostic, not a value +//! +//! **What a practitioner sees, and what it costs them.** A line on a graph that +//! stops. The modeller's next task is *provenance*: search backward through the +//! dependency graph for where the NaN came from, because NaN is absorbing -- +//! every variable downstream of the origin shows the identical symptom, so the +//! graph says that something broke and not what. That backward hunt is the real +//! cost of a NaN, and it is paid by hand. +//! +//! **What it usually is.** Most often the runtime result of a division by zero, +//! which then propagates through everything reading it. Sometimes it is +//! deliberate: a modeller uses it as a sentinel to catch behaviour leaving a +//! sensible range. Either way it means something is broken -- not that a normal +//! computation produced an unusual number. ([`NA`] is the opposite case, and is +//! why it is finite: "missing data" is a value the model tests for, so it has to +//! survive comparison and arithmetic.) +//! +//! **Two consequences for the engine**, which are the part a maintainer needs: +//! +//! 1. *Attributing a NaN to its origin is high-value*, because the alternative +//! is that hand search. Wherever the engine knows STRUCTURALLY that a +//! variable must evaluate to NaN -- an unfilled equation is the clearest case +//! -- a warning naming the variable replaces the entire backward hunt. That +//! is the argument for such diagnostics, and it is worth more than it looks. +//! 2. *A NaN the engine manufactures is noise in a channel debugged by hand.* On +//! the graph it is indistinguishable from the user's own division by zero, so +//! it costs someone a debugging session that ends at our bug. That is what +//! makes GH #975 (a spurious first-step NaN in a generated LTM link score) a +//! real defect rather than a cosmetic one, and the reason to fix that class +//! properly rather than narrowly. +//! +//! **The technical footnote**, which follows from the above rather than +//! motivating it: the engine needs no machinery that treats a NaN as a value +//! with structure worth preserving. +//! +//! A practitioner cannot author a NaN payload. There is exactly one NaN spelling +//! at the language surface -- the lexer's `nan` keyword -- and it yields one +//! canonical `f64::NAN` (`0x7ff8_0000_0000_0000`). The engine can manufacture +//! one too, by folding an arithmetic NaN at compile time +//! (`crate::compiler::fold` on `0/0`) or by computing one at runtime; on +//! x86-64 those carry the hardware default quiet NaN +//! (`0xfff8_0000_0000_0000`) instead. That difference is deterministic and, +//! more to the point, semantically empty: a NaN is a NaN. +//! +//! So comparing NaNs by BIT PATTERN -- which `crate::ast::Literal` does, so a +//! cached value is equal to itself and salsa can backdate it -- draws no +//! distinction anyone can author or observe, while IEEE `==` (`NaN != NaN`) +//! makes a value unequal to its own rebuild. Bit comparison is the right default +//! wherever a float feeds a cache key; `ast::Literal` records where it is +//! implemented, which types are still outside it, and why that is accepted. /// Vensim's `:NA:` ("missing data") sentinel: the *finite* number `-2^109`. /// diff --git a/src/simlin-engine/src/ltm/polarity.rs b/src/simlin-engine/src/ltm/polarity.rs index 44c3df639..b4799a2ae 100644 --- a/src/simlin-engine/src/ltm/polarity.rs +++ b/src/simlin-engine/src/ltm/polarity.rs @@ -189,7 +189,8 @@ pub(super) fn analyze_agg_consumer_polarity( /// Rebuild `expr`, replacing every subtree whose canonical printed form /// equals `target_text` with a bare `Var(replacement)`. Used only by /// [`analyze_agg_consumer_polarity`]; the printed-form comparison mirrors -/// how `enumerate_agg_nodes` keys aggregate nodes (`Expr2` is not `Eq`). +/// how `enumerate_agg_nodes` keys aggregate nodes (`Expr2` is `Eq` but not +/// `Hash`, so the dedup map is keyed by that printed form). fn substitute_subexpr_in_expr2( expr: &Expr2, target_text: &str, @@ -823,7 +824,7 @@ pub(super) fn expr_references_var(expr: &Expr2, var: &Ident) -> bool /// Check if expression is a positive constant pub(super) fn is_positive_constant(expr: &Expr2) -> bool { match expr { - Expr2::Const(_, n, _) => *n > 0.0, + Expr2::Const(_, n, _) => n.value() > 0.0, _ => false, } } @@ -831,7 +832,7 @@ pub(super) fn is_positive_constant(expr: &Expr2) -> bool { /// Check if expression is a negative constant pub(super) fn is_negative_constant(expr: &Expr2) -> bool { match expr { - Expr2::Const(_, n, _) => *n < 0.0, + Expr2::Const(_, n, _) => n.value() < 0.0, _ => false, } } diff --git a/src/simlin-engine/src/ltm/polarity_tests.rs b/src/simlin-engine/src/ltm/polarity_tests.rs new file mode 100644 index 000000000..77785e286 --- /dev/null +++ b/src/simlin-engine/src/ltm/polarity_tests.rs @@ -0,0 +1,971 @@ +// Copyright 2026 The Simlin Authors. All rights reserved. +// Use of this source code is governed by the Apache License, +// Version 2.0, that can be found in the LICENSE file. + +//! Static link-polarity analysis tests: the polarity helpers (`flip_polarity`, +//! the constant-sign predicates), `analyze_link_polarity` / the +//! `analyze_expr_polarity` expression forms (if-then-else, unary NOT, division, +//! array reducers, subscripts), the graphical-function monotonicity classifier, +//! and lookup-table polarity propagation through links. +//! +//! Split out of `ltm/tests.rs` to keep that file under the project line-count +//! lint; included via `#[path]` as a child of it, so `use super::*` resolves +//! that file's imports and private helpers. + +use super::*; + +#[test] +fn test_flip_polarity() { + // Test flip_polarity function (covers lines 1049-1054) + assert_eq!( + flip_polarity(LinkPolarity::Positive), + LinkPolarity::Negative + ); + assert_eq!( + flip_polarity(LinkPolarity::Negative), + LinkPolarity::Positive + ); + assert_eq!(flip_polarity(LinkPolarity::Unknown), LinkPolarity::Unknown); +} + +#[test] +fn test_is_positive_constant() { + // Test is_positive_constant function (covers lines 1058-1062) + use crate::ast::{Expr2, Loc}; + + let pos_const = Expr2::Const( + "5".to_string(), + crate::ast::Literal::new(5.0), + Loc::default(), + ); + assert!(is_positive_constant(&pos_const), "5.0 should be positive"); + + let neg_const = Expr2::Const( + "-5".to_string(), + crate::ast::Literal::new(-5.0), + Loc::default(), + ); + assert!( + !is_positive_constant(&neg_const), + "-5.0 should not be positive" + ); + + let zero_const = Expr2::Const( + "0".to_string(), + crate::ast::Literal::new(0.0), + Loc::default(), + ); + assert!( + !is_positive_constant(&zero_const), + "0.0 should not be positive" + ); + + let var_expr = Expr2::Var(Ident::new("x"), None, Loc::default()); + assert!( + !is_positive_constant(&var_expr), + "Variable should not be positive constant" + ); +} + +#[test] +fn test_is_negative_constant() { + // Test is_negative_constant function (covers lines 1066-1070) + use crate::ast::{Expr2, Loc}; + + let neg_const = Expr2::Const( + "-3".to_string(), + crate::ast::Literal::new(-3.0), + Loc::default(), + ); + assert!(is_negative_constant(&neg_const), "-3.0 should be negative"); + + let pos_const = Expr2::Const( + "3".to_string(), + crate::ast::Literal::new(3.0), + Loc::default(), + ); + assert!( + !is_negative_constant(&pos_const), + "3.0 should not be negative" + ); + + let zero_const = Expr2::Const( + "0".to_string(), + crate::ast::Literal::new(0.0), + Loc::default(), + ); + assert!( + !is_negative_constant(&zero_const), + "0.0 should not be negative" + ); + + let var_expr = Expr2::Var(Ident::new("y"), None, Loc::default()); + assert!( + !is_negative_constant(&var_expr), + "Variable should not be negative constant" + ); +} + +#[test] +fn test_analyze_link_polarity_arrayed() { + // Test analyze_link_polarity with Arrayed AST (covers lines 935-947) + use crate::ast::{Ast, Expr2, Loc}; + use crate::common::CanonicalElementName; + use std::collections::HashMap; + + let x_var = Ident::new("x"); + + // Create arrayed AST with consistent positive polarity + let mut elements = HashMap::new(); + elements.insert( + CanonicalElementName::from_raw("dim1"), + Expr2::Op2( + BinaryOp::Mul, + Box::new(Expr2::Var(x_var.clone(), None, Loc::default())), + Box::new(Expr2::Const( + "2".to_string(), + crate::ast::Literal::new(2.0), + Loc::default(), + )), + None, + Loc::default(), + ), + ); + elements.insert( + CanonicalElementName::from_raw("dim2"), + Expr2::Op2( + BinaryOp::Add, + Box::new(Expr2::Var(x_var.clone(), None, Loc::default())), + Box::new(Expr2::Const( + "10".to_string(), + crate::ast::Literal::new(10.0), + Loc::default(), + )), + None, + Loc::default(), + ), + ); + + let ast = Ast::Arrayed(vec![], elements, None, false); + let empty_vars = HashMap::new(); + let polarity = analyze_link_polarity(&ast, &x_var, &empty_vars); + assert_eq!( + polarity, + LinkPolarity::Positive, + "Consistent positive elements should be positive" + ); + + // Test with mixed polarities + let mut mixed_elements = HashMap::new(); + mixed_elements.insert( + CanonicalElementName::from_raw("dim1"), + Expr2::Var(x_var.clone(), None, Loc::default()), + ); + mixed_elements.insert( + CanonicalElementName::from_raw("dim2"), + Expr2::Op1( + crate::ast::UnaryOp::Negative, + Box::new(Expr2::Var(x_var.clone(), None, Loc::default())), + None, + Loc::default(), + ), + ); + + let mixed_ast = Ast::Arrayed(vec![], mixed_elements, None, false); + let mixed_polarity = analyze_link_polarity(&mixed_ast, &x_var, &empty_vars); + assert_eq!( + mixed_polarity, + LinkPolarity::Unknown, + "Mixed polarities should be Unknown" + ); +} + +#[test] +fn test_analyze_expr_polarity_if_then_else() { + // Test analyze_expr_polarity with If-Then-Else (covers lines 1033-1042) + use crate::ast::{Expr2, Loc}; + + let x_var = Ident::new("x"); + + // If with same polarity in both branches + let if_expr = Expr2::If( + Box::new(Expr2::Const( + "1".to_string(), + crate::ast::Literal::new(1.0), + Loc::default(), + )), + Box::new(Expr2::Var(x_var.clone(), None, Loc::default())), + Box::new(Expr2::Op2( + BinaryOp::Mul, + Box::new(Expr2::Var(x_var.clone(), None, Loc::default())), + Box::new(Expr2::Const( + "2".to_string(), + crate::ast::Literal::new(2.0), + Loc::default(), + )), + None, + Loc::default(), + )), + None, + Loc::default(), + ); + + let polarity = + analyze_expr_polarity_with_context(&if_expr, &x_var, LinkPolarity::Positive, None); + assert_eq!( + polarity, + LinkPolarity::Positive, + "Same polarity branches should return that polarity" + ); + + // If with different polarities in branches + let mixed_if = Expr2::If( + Box::new(Expr2::Const( + "1".to_string(), + crate::ast::Literal::new(1.0), + Loc::default(), + )), + Box::new(Expr2::Var(x_var.clone(), None, Loc::default())), + Box::new(Expr2::Op1( + crate::ast::UnaryOp::Negative, + Box::new(Expr2::Var(x_var.clone(), None, Loc::default())), + None, + Loc::default(), + )), + None, + Loc::default(), + ); + + let mixed_polarity = + analyze_expr_polarity_with_context(&mixed_if, &x_var, LinkPolarity::Positive, None); + assert_eq!( + mixed_polarity, + LinkPolarity::Unknown, + "Different polarity branches should be Unknown" + ); +} + +#[test] +fn test_analyze_expr_polarity_unary_not() { + // Test analyze_expr_polarity with unary NOT operator (covers lines 1026-1031) + use crate::ast::{Expr2, Loc, UnaryOp}; + + let x_var = Ident::new("x"); + + let not_expr = Expr2::Op1( + UnaryOp::Not, + Box::new(Expr2::Var(x_var.clone(), None, Loc::default())), + None, + Loc::default(), + ); + + let polarity = + analyze_expr_polarity_with_context(¬_expr, &x_var, LinkPolarity::Positive, None); + assert_eq!( + polarity, + LinkPolarity::Negative, + "NOT should flip polarity from positive to negative" + ); +} + +#[test] +fn test_analyze_expr_polarity_division_edge_cases() { + // Test division polarity analysis edge cases (covers lines 1013-1022) + use crate::ast::{Expr2, Loc}; + + let x_var = Ident::new("x"); + let y_var = Ident::new("y"); + + // Division with variable in numerator + let div_num = Expr2::Op2( + BinaryOp::Div, + Box::new(Expr2::Var(x_var.clone(), None, Loc::default())), + Box::new(Expr2::Const( + "10".to_string(), + crate::ast::Literal::new(10.0), + Loc::default(), + )), + None, + Loc::default(), + ); + + let pol_num = + analyze_expr_polarity_with_context(&div_num, &x_var, LinkPolarity::Positive, None); + assert_eq!( + pol_num, + LinkPolarity::Positive, + "Variable in numerator should keep polarity" + ); + + // Division with different variable in denominator (not the one we're tracking) + let div_other = Expr2::Op2( + BinaryOp::Div, + Box::new(Expr2::Const( + "100".to_string(), + crate::ast::Literal::new(100.0), + Loc::default(), + )), + Box::new(Expr2::Var(y_var.clone(), None, Loc::default())), + None, + Loc::default(), + ); + + let pol_other = + analyze_expr_polarity_with_context(&div_other, &x_var, LinkPolarity::Positive, None); + assert_eq!( + pol_other, + LinkPolarity::Unknown, + "Unrelated variable should give Unknown" + ); +} + +#[test] +fn test_analyze_expr_polarity_array_reducers() { + // Array reducers SUM, MEAN, MAX (single-arg), MIN (single-arg) are monotone + // in their argument: their polarity equals the inner expression's polarity. + // STDDEV and RANK are not monotone: they must return Unknown even when the + // argument has a known polarity. + use crate::ast::{Expr2, Loc, UnaryOp}; + use crate::builtins::BuiltinFn; + + let x_var = Ident::new("x"); + let pos_inner = || Expr2::Var(x_var.clone(), None, Loc::default()); + let neg_inner = || { + Expr2::Op1( + UnaryOp::Negative, + Box::new(Expr2::Var(x_var.clone(), None, Loc::default())), + None, + Loc::default(), + ) + }; + + // SUM passes through positive polarity. + let sum_pos = Expr2::App(BuiltinFn::Sum(Box::new(pos_inner())), None, Loc::default()); + assert_eq!( + analyze_expr_polarity_with_context(&sum_pos, &x_var, LinkPolarity::Positive, None), + LinkPolarity::Positive, + "SUM(x) should pass through positive polarity", + ); + + // SUM passes through negative polarity (e.g. SUM(-x)). + let sum_neg = Expr2::App(BuiltinFn::Sum(Box::new(neg_inner())), None, Loc::default()); + assert_eq!( + analyze_expr_polarity_with_context(&sum_neg, &x_var, LinkPolarity::Positive, None), + LinkPolarity::Negative, + "SUM(-x) should pass through negative polarity", + ); + + // MEAN with a single (array) argument passes through positive polarity. + let mean_pos = Expr2::App(BuiltinFn::Mean(vec![pos_inner()]), None, Loc::default()); + assert_eq!( + analyze_expr_polarity_with_context(&mean_pos, &x_var, LinkPolarity::Positive, None), + LinkPolarity::Positive, + "MEAN(x) should pass through positive polarity", + ); + + // MEAN with a single (array) argument passes through negative polarity. + let mean_neg = Expr2::App(BuiltinFn::Mean(vec![neg_inner()]), None, Loc::default()); + assert_eq!( + analyze_expr_polarity_with_context(&mean_neg, &x_var, LinkPolarity::Positive, None), + LinkPolarity::Negative, + "MEAN(-x) should pass through negative polarity", + ); + + // Array MAX (no second argument) passes through inner polarity. + let max_array_pos = Expr2::App( + BuiltinFn::Max(Box::new(pos_inner()), None), + None, + Loc::default(), + ); + assert_eq!( + analyze_expr_polarity_with_context(&max_array_pos, &x_var, LinkPolarity::Positive, None), + LinkPolarity::Positive, + "MAX(x) (array form) should pass through positive polarity", + ); + let max_array_neg = Expr2::App( + BuiltinFn::Max(Box::new(neg_inner()), None), + None, + Loc::default(), + ); + assert_eq!( + analyze_expr_polarity_with_context(&max_array_neg, &x_var, LinkPolarity::Positive, None), + LinkPolarity::Negative, + "MAX(-x) (array form) should pass through negative polarity", + ); + + // Array MIN (no second argument) passes through inner polarity. + let min_array_pos = Expr2::App( + BuiltinFn::Min(Box::new(pos_inner()), None), + None, + Loc::default(), + ); + assert_eq!( + analyze_expr_polarity_with_context(&min_array_pos, &x_var, LinkPolarity::Positive, None), + LinkPolarity::Positive, + "MIN(x) (array form) should pass through positive polarity", + ); + let min_array_neg = Expr2::App( + BuiltinFn::Min(Box::new(neg_inner()), None), + None, + Loc::default(), + ); + assert_eq!( + analyze_expr_polarity_with_context(&min_array_neg, &x_var, LinkPolarity::Positive, None), + LinkPolarity::Negative, + "MIN(-x) (array form) should pass through negative polarity", + ); + + // STDDEV is non-monotone: even with a positive-polarity argument, the result + // is Unknown because variance has no fixed sign w.r.t. its inputs. + let stddev = Expr2::App( + BuiltinFn::Stddev(Box::new(pos_inner())), + None, + Loc::default(), + ); + assert_eq!( + analyze_expr_polarity_with_context(&stddev, &x_var, LinkPolarity::Positive, None), + LinkPolarity::Unknown, + "STDDEV must always return Unknown polarity", + ); + + // RANK depends on the rest of the array, so polarity is undefined. + let direction = Expr2::Const( + "1".to_string(), + crate::ast::Literal::new(1.0), + Loc::default(), + ); + let rank = Expr2::App( + BuiltinFn::Rank(Box::new(pos_inner()), Box::new(direction)), + None, + Loc::default(), + ); + assert_eq!( + analyze_expr_polarity_with_context(&rank, &x_var, LinkPolarity::Positive, None), + LinkPolarity::Unknown, + "RANK must always return Unknown polarity", + ); +} + +/// Verify reducer polarity propagates through the actual parsed shape: +/// `SUM(x[*])` lowers to `Sum(Box::new(Subscript(x, [Wildcard], _, _)))`, +/// not `Sum(Box::new(Var(x, ...)))`. Without an `Expr2::Subscript` arm in +/// `analyze_expr_polarity_with_context`, the new reducer arms fall through +/// to Unknown for the production case the issue actually targets. +#[test] +fn test_analyze_expr_polarity_array_reducers_subscript_wildcard() { + use crate::ast::{Expr2, IndexExpr2, Loc, UnaryOp}; + use crate::builtins::BuiltinFn; + use LinkPolarity::{Negative, Positive, Unknown}; + + let x = Ident::new("x"); + let y = Ident::new("y"); + let sub = |id: &Ident| { + Expr2::Subscript( + id.clone(), + vec![IndexExpr2::Wildcard(Loc::default())], + None, + Loc::default(), + ) + }; + let app = |b: BuiltinFn| Expr2::App(b, None, Loc::default()); + let neg = |e: Expr2| Expr2::Op1(UnaryOp::Negative, Box::new(e), None, Loc::default()); + let one = || { + Expr2::Const( + "1".to_string(), + crate::ast::Literal::new(1.0), + Loc::default(), + ) + }; + + // (label, expression, context_polarity, expected_polarity) + let cases: Vec<(&str, Expr2, LinkPolarity, LinkPolarity)> = vec![ + ( + "SUM(x[*]) +", + app(BuiltinFn::Sum(Box::new(sub(&x)))), + Positive, + Positive, + ), + ( + "SUM(x[*]) -", + app(BuiltinFn::Sum(Box::new(sub(&x)))), + Negative, + Negative, + ), + ( + "SUM(-x[*])", + app(BuiltinFn::Sum(Box::new(neg(sub(&x))))), + Positive, + Negative, + ), + ( + "SUM(y[*])", + app(BuiltinFn::Sum(Box::new(sub(&y)))), + Positive, + Unknown, + ), + ( + "MEAN(x[*])", + app(BuiltinFn::Mean(vec![sub(&x)])), + Positive, + Positive, + ), + ( + "MAX(x[*])", + app(BuiltinFn::Max(Box::new(sub(&x)), None)), + Positive, + Positive, + ), + ( + "MIN(x[*])", + app(BuiltinFn::Min(Box::new(sub(&x)), None)), + Positive, + Positive, + ), + ( + "STDDEV(x[*])", + app(BuiltinFn::Stddev(Box::new(sub(&x)))), + Positive, + Unknown, + ), + ( + "RANK(x[*], 1)", + app(BuiltinFn::Rank(Box::new(sub(&x)), Box::new(one()))), + Positive, + Unknown, + ), + ]; + + for (label, expr, ctx, want) in &cases { + assert_eq!( + analyze_expr_polarity_with_context(expr, &x, *ctx, None), + *want, + "{label}", + ); + } +} + +/// The Subscript arm must distinguish between indices that are independent +/// of `from_var` (literal, wildcard, expressions over other variables) and +/// indices that themselves reference `from_var`. In the latter case the +/// relationship between `from_var` and the subscripted result is non-monotone: +/// changing `from_var` shifts both the lookup target AND the index, so no +/// single polarity describes the result. The dominant cases (`SUM(arr[*])`, +/// `arr[Region]`, indices over OTHER variables) keep their original behavior +/// of returning `current_polarity` because their indices don't reference +/// `from_var`. +#[test] +fn test_analyze_expr_polarity_subscript_self_indexing() { + use crate::ast::{Expr2, IndexExpr2, Loc}; + use crate::builtins::BuiltinFn; + use LinkPolarity::{Positive, Unknown}; + + let arr = Ident::new("arr"); + let other = Ident::new("other"); + let i = Ident::new("i"); + + let var = |id: &Ident| Expr2::Var(id.clone(), None, Loc::default()); + let lit = |n: f64| Expr2::Const(format!("{n}"), crate::ast::Literal::new(n), Loc::default()); + + // arr[*] -- wildcard index, no reference to arr in the index. + let arr_wildcard = Expr2::Subscript( + arr.clone(), + vec![IndexExpr2::Wildcard(Loc::default())], + None, + Loc::default(), + ); + assert_eq!( + analyze_expr_polarity_with_context(&arr_wildcard, &arr, Positive, None), + Positive, + "arr[*] preserves current_polarity", + ); + + // arr[3] -- literal index, no reference to arr in the index. + let arr_literal = Expr2::Subscript( + arr.clone(), + vec![IndexExpr2::Expr(lit(3.0))], + None, + Loc::default(), + ); + assert_eq!( + analyze_expr_polarity_with_context(&arr_literal, &arr, Positive, None), + Positive, + "arr[3] preserves current_polarity", + ); + + // arr[i] where i is a different variable -- index references some OTHER + // variable, but not from_var (= arr). Polarity contract still holds. + let arr_other_index = Expr2::Subscript( + arr.clone(), + vec![IndexExpr2::Expr(var(&i))], + None, + Loc::default(), + ); + assert_eq!( + analyze_expr_polarity_with_context(&arr_other_index, &arr, Positive, None), + Positive, + "arr[i] (i != from_var) preserves current_polarity", + ); + + // arr[arr] -- index trivially references arr. Result is non-monotone + // because shifting arr shifts both the lookup target and the index. + let arr_self_var = Expr2::Subscript( + arr.clone(), + vec![IndexExpr2::Expr(var(&arr))], + None, + Loc::default(), + ); + assert_eq!( + analyze_expr_polarity_with_context(&arr_self_var, &arr, Positive, None), + Unknown, + "arr[arr] is non-monotone", + ); + + // arr[INT(arr[i])] -- the canonical self-indexing case. Index references + // arr through a nested subscript; relationship is non-monotone. + let inner = Expr2::Subscript( + arr.clone(), + vec![IndexExpr2::Expr(var(&i))], + None, + Loc::default(), + ); + let int_inner = Expr2::App(BuiltinFn::Int(Box::new(inner)), None, Loc::default()); + let arr_self_nested = Expr2::Subscript( + arr.clone(), + vec![IndexExpr2::Expr(int_inner)], + None, + Loc::default(), + ); + assert_eq!( + analyze_expr_polarity_with_context(&arr_self_nested, &arr, Positive, None), + Unknown, + "arr[INT(arr[i])] is non-monotone", + ); + + // other[*] where from_var is arr -- subscripted array is not from_var. + // Existing behavior: contributes Unknown because the arm conservatively + // can't classify references through other arrays. + let other_wildcard = Expr2::Subscript( + other.clone(), + vec![IndexExpr2::Wildcard(Loc::default())], + None, + Loc::default(), + ); + assert_eq!( + analyze_expr_polarity_with_context(&other_wildcard, &arr, Positive, None), + Unknown, + "other[*] (other != from_var) returns Unknown", + ); +} + +#[test] +fn test_graphical_function_polarity() { + use crate::variable::Table; + + // Test 1: Monotonically increasing function (positive polarity) + let increasing_table = + Table::new_for_test(vec![0.0, 1.0, 2.0, 3.0, 4.0], vec![0.0, 2.0, 4.0, 6.0, 8.0]); + assert_eq!( + analyze_graphical_function_polarity(&increasing_table), + LinkPolarity::Positive, + "Monotonically increasing function should have positive polarity" + ); + + // Test 2: Monotonically decreasing function (negative polarity) + let decreasing_table = Table::new_for_test( + vec![0.0, 1.0, 2.0, 3.0, 4.0], + vec![10.0, 8.0, 6.0, 4.0, 2.0], + ); + assert_eq!( + analyze_graphical_function_polarity(&decreasing_table), + LinkPolarity::Negative, + "Monotonically decreasing function should have negative polarity" + ); + + // Test 3: Non-monotonic function (unknown polarity) + let non_monotonic_table = + Table::new_for_test(vec![0.0, 1.0, 2.0, 3.0, 4.0], vec![0.0, 5.0, 3.0, 7.0, 2.0]); + assert_eq!( + analyze_graphical_function_polarity(&non_monotonic_table), + LinkPolarity::Unknown, + "Non-monotonic function should have unknown polarity" + ); + + // Test 4: Constant function (unknown polarity - no change) + let constant_table = Table::new_for_test(vec![0.0, 1.0, 2.0, 3.0], vec![5.0, 5.0, 5.0, 5.0]); + assert_eq!( + analyze_graphical_function_polarity(&constant_table), + LinkPolarity::Unknown, + "Constant function should have unknown polarity" + ); + + // Test 5: Single point (edge case) + let single_point_table = Table::new_for_test(vec![1.0], vec![2.0]); + assert_eq!( + analyze_graphical_function_polarity(&single_point_table), + LinkPolarity::Unknown, + "Single point should have unknown polarity" + ); + + // Test 6: Nearly constant with small variations (testing tolerance) + let nearly_constant_table = + Table::new_for_test(vec![0.0, 1.0, 2.0, 3.0], vec![5.0, 5.0001, 5.0002, 5.0003]); + assert_eq!( + analyze_graphical_function_polarity(&nearly_constant_table), + LinkPolarity::Positive, + "Nearly constant but increasing should have positive polarity" + ); +} + +#[test] +fn test_graphical_function_polarity_tolerates_import_noise() { + use crate::variable::Table; + + // A table that is monotone non-decreasing modulo round-trip numeric-import + // noise: the second segment dips by ~2e-7 against a 1.5-wide y-range. The + // y-range-relative epsilon (1e-6 * 1.5 = 1.5e-6) absorbs the dip, so the + // table reads as Positive. With the old absolute 1e-10 epsilon this dip + // broke monotonicity and the table read as Unknown (#492). + let import_noise_table = Table::new_for_test( + vec![0.0, 1.0, 2.0, 3.0, 4.0], + vec![0.0, 0.5000001, 0.4999999, 1.0, 1.5], + ); + assert_eq!( + analyze_graphical_function_polarity(&import_noise_table), + LinkPolarity::Positive, + "A monotone-modulo-import-noise table should read as Positive" + ); + + // A genuine ~0.7 reversal against a 1.0 y-range: far larger than the + // relative epsilon (1e-6), so the table is correctly Unknown. + let real_reversal_table = + Table::new_for_test(vec![0.0, 1.0, 2.0, 3.0], vec![0.0, 1.0, 0.3, 0.7]); + assert_eq!( + analyze_graphical_function_polarity(&real_reversal_table), + LinkPolarity::Unknown, + "A genuinely non-monotone table is still Unknown" + ); + + // A perfectly constant table: y_max - y_min == 0 so epsilon clamps to + // 1e-12; every dy == 0 is a plateau (not > 1e-12, not < -1e-12), so the + // table is still classified constant and reads as Unknown. + let constant_table = Table::new_for_test(vec![0.0, 1.0, 2.0], vec![5.0, 5.0, 5.0]); + assert_eq!( + analyze_graphical_function_polarity(&constant_table), + LinkPolarity::Unknown, + "A constant table is still Unknown" + ); +} + +#[test] +fn test_graphical_function_polarity_uses_slope_not_y_delta() { + use crate::variable::Table; + + // Non-uniform x-spacing where the y-delta heuristic and the slope heuristic + // DISAGREE on the verdict (#536). The middle segment is a genuine, steep, + // local DECREASE over a very narrow x-interval (dx = 0.001), so the table is + // NOT monotone -- the correct answer is Unknown. + // + // x = [0, 100, 100.001, 200] + // y = [0, 10, 9.99999, 20] + // + // y-range = 20, so the y-range-relative dy epsilon is 1e-6 * 20 = 2e-5. + // The middle dy is -1e-5, which is *below* that epsilon, so the y-delta + // heuristic SUPPRESSES the dip as a plateau and -- seeing only the two + // surrounding increases -- wrongly classifies the table as Positive. + // + // The middle segment's SLOPE, however, is -1e-5 / 0.001 = -0.01, far + // steeper (in magnitude) than the table's average slope (20/200 = 0.1) + // times the relative tolerance, so the slope heuristic correctly sees a + // real local decrease and returns Unknown. + let narrow_dip_table = Table::new_for_test( + vec![0.0, 100.0, 100.001, 200.0], + vec![0.0, 10.0, 9.99999, 20.0], + ); + assert_eq!( + analyze_graphical_function_polarity(&narrow_dip_table), + LinkPolarity::Unknown, + "a steep narrow local decrease must be caught by the slope heuristic, not \ + smoothed over by the y-delta heuristic" + ); + + // The mirror case: a steep narrow local INCREASE in an otherwise-decreasing + // table must likewise flip a naive Negative to Unknown. + let narrow_bump_table = Table::new_for_test( + vec![0.0, 100.0, 100.001, 200.0], + vec![20.0, 10.0, 10.00001, 0.0], + ); + assert_eq!( + analyze_graphical_function_polarity(&narrow_bump_table), + LinkPolarity::Unknown, + "a steep narrow local increase must flip an otherwise-decreasing table to Unknown" + ); + + // A genuinely monotone table with non-uniform x-spacing (no sign change in + // slope) must still classify correctly: every segment slopes up, so the + // verdict is Positive even though the dx values vary by orders of magnitude. + let monotone_nonuniform_table = + Table::new_for_test(vec![0.0, 0.001, 100.0, 200.0], vec![0.0, 5.0, 10.0, 20.0]); + assert_eq!( + analyze_graphical_function_polarity(&monotone_nonuniform_table), + LinkPolarity::Positive, + "a monotone table with non-uniform x-spacing stays Positive" + ); + + // Degenerate vertical segment (x[i] == x[i-1]) with differing y: two outputs + // for one input is an ambiguous lookup with undefined slope, so bail to + // Unknown. + let vertical_segment_table = + Table::new_for_test(vec![0.0, 1.0, 1.0, 2.0], vec![0.0, 1.0, 5.0, 6.0]); + assert_eq!( + analyze_graphical_function_polarity(&vertical_segment_table), + LinkPolarity::Unknown, + "a vertical (dx == 0, dy != 0) segment has undefined slope and must read as Unknown" + ); + + // A duplicated point (x[i] == x[i-1] AND y[i] == y[i-1]) is redundant, not + // ambiguous: it must be skipped as non-determining and not poison an + // otherwise-monotone verdict. + let duplicate_point_table = + Table::new_for_test(vec![0.0, 1.0, 1.0, 2.0], vec![0.0, 1.0, 1.0, 2.0]); + assert_eq!( + analyze_graphical_function_polarity(&duplicate_point_table), + LinkPolarity::Positive, + "a redundant duplicate point must not flip a monotone-increasing table" + ); +} + +/// GH #536 tolerance regression: a many-point uniformly-spaced monotone table +/// with a single small import-noise dip must still classify Positive even after +/// the slope-based tolerance was introduced (#536). +/// +/// Root cause: the original #536 fix set `slope_epsilon = 1e-6 * (y_max - +/// y_min) / x_span`, which is `1e-6 * avg_slope_mag`. For a uniformly-spaced +/// n-point table `dx = x_span / (n - 1)`, so the effective per-segment dy +/// threshold becomes `slope_epsilon * dx = 1e-6 * (y_max - y_min) / (n - 1)` -- +/// which is `(n - 1)x` tighter than the old #492 threshold `1e-6 * (y_max - +/// y_min)`. A 50-point table therefore has a threshold ~49x tighter, and a +/// noise dip that sat safely inside the #492 window now crosses it and flips the +/// classification to Unknown. +/// +/// The correct fix scales the tolerance by `avg_dx = x_span / (n - 1)`, giving +/// `slope_epsilon = 1e-6 * (y_max - y_min) / avg_dx`. Then for uniform spacing +/// `dx == avg_dx` and the per-segment dy threshold is `slope_epsilon * dx = +/// 1e-6 * (y_max - y_min)` -- exactly the old #492 behavior. For non-uniform +/// spacing the threshold still scales by `dx / avg_dx`, so narrow steep segments +/// still trip correctly (the #536 motivation is preserved). +#[test] +fn test_graphical_function_polarity_uniform_50pt_import_noise() { + use crate::variable::Table; + + // Build a 50-point uniformly-spaced monotone-increasing table on [0, 49] + // with y rising from 0.0 to 1.0. One interior point (index 25) has a + // downward import-noise dip of -5e-8. + // + // With 50 points: y_range = 1.0, x_span = 49.0, avg_dx = 1.0. + // Old #492 threshold: 1e-6 * 1.0 = 1e-6. The dip (-5e-8) is far below that + // threshold in magnitude, so the old code classified Positive. + // + // Buggy #536 slope_epsilon = 1e-6 * 1.0 / 49.0 ≈ 2.04e-8. + // The dip's slope = -5e-8 / 1.0 = -5e-8, which is < -2.04e-8, so the + // buggy code classifies Unknown (the #492 regression). + // + // Fixed #536 slope_epsilon = 1e-6 * 1.0 / avg_dx = 1e-6 * 1.0 / 1.0 = 1e-6. + // The dip's slope (-5e-8) is within [-1e-6, 1e-6], so the fixed code + // correctly classifies Positive. + let n = 50usize; + let x_span = (n - 1) as f64; // 49.0 + let noise_dip_idx = 25usize; + // The noise magnitude must be: + // - within the old #492 threshold: 1e-6 * y_range = 1e-6 * 1.0 = 1e-6 + // - outside the buggy #536 threshold: 1e-6 * y_range / (n-1) = 1e-6/49 ≈ 2.04e-8 + // Choosing 5e-8: between 2.04e-8 and 1e-6, so buggy code fails and correct + // code passes. + let noise_magnitude = 5e-8_f64; + + // Build a strictly-increasing base curve and then pull y[noise_dip_idx] + // BELOW y[noise_dip_idx - 1] by noise_magnitude. This makes the segment + // (noise_dip_idx - 1) -> noise_dip_idx have a genuinely negative slope: + // dy = -noise_magnitude, dx = 1.0, slope = -5e-8. + // The following segment noise_dip_idx -> (noise_dip_idx + 1) is then + // extra-positive (+2/49 + noise_magnitude), so the table overall is nearly + // monotone. + let xs: Vec = (0..n).map(|i| i as f64).collect(); + let base_y: Vec = (0..n).map(|i| i as f64 / x_span).collect(); + let ys: Vec = (0..n) + .map(|i| { + if i == noise_dip_idx { + // pull this point below its predecessor: slope (i-1)->i is -5e-8 + base_y[noise_dip_idx - 1] - noise_magnitude + } else { + base_y[i] + } + }) + .collect(); + + let table = Table::new_for_test(xs, ys); + assert_eq!( + analyze_graphical_function_polarity(&table), + LinkPolarity::Positive, + "a 50-point uniformly-spaced monotone table with a single {noise_magnitude:.0e} \ + import-noise dip (within the #492 tolerance 1e-6 * y_range = 1e-6) must \ + classify Positive; slope_epsilon must scale by avg_dx so uniform-spacing \ + behavior matches the pre-#536 y-delta threshold (GH #536 regression)" + ); +} + +#[test] +fn test_lookup_table_polarity_in_links() { + use crate::datamodel; + + // Create a model with a lookup table. The flow scales the lookup result + // by a positive CONSTANT (not by `water` itself): with `water` appearing + // in both factors of a product, the partial `L(w) + w*L'(w)` is + // sign-indefinite without value reasoning, so the sound answer would be + // Unknown -- this test isolates the table-monotonicity propagation. + let mut model_vars = vec![ + x_stock("water", "100", &[], &["outflow"], None), + x_flow("outflow", "0.5 * lookup(lookup, water)", None), + ]; + + // Create the lookup table auxiliary + let mut lookup_var = x_aux("lookup", "0", None); + if let datamodel::Variable::Aux(aux) = &mut lookup_var { + aux.gf = Some(datamodel::GraphicalFunction { + kind: datamodel::GraphicalFunctionKind::Continuous, + x_points: Some(vec![0.0, 50.0, 100.0, 150.0]), + y_points: vec![0.1, 0.2, 0.3, 0.4], + x_scale: datamodel::GraphicalFunctionScale { + min: 0.0, + max: 150.0, + }, + y_scale: datamodel::GraphicalFunctionScale { min: 0.1, max: 0.4 }, + }); + } + model_vars.push(lookup_var); + + let model = x_model("main", model_vars); + let sim_specs = sim_specs_with_units("months"); + let datamodel_project = x_project(sim_specs, &[model]); + let db = SimlinDb::default(); + let result = sync_from_datamodel(&db, &datamodel_project); + let model = result.models["main"].source; + + // Check per-link polarity via compute_link_polarities + let polarities = compute_link_polarities(&db, model, result.project); + let water_to_outflow_key = ("water".to_string(), "outflow".to_string()); + assert_eq!( + polarities[&water_to_outflow_key], + LinkPolarity::Positive, + "Monotonically increasing lookup table should preserve positive polarity" + ); + + // Verify loop polarity via model_detected_loops + let detected = model_detected_loops(&db, model, result.project); + assert_eq!(detected.loops.len(), 1, "Should have one loop"); + // water -> outflow: Positive (increasing lookup), outflow -> water: Negative (outflow) + assert_eq!( + detected.loops[0].polarity, + DetectedLoopPolarity::Balancing, + "Loop with one negative link should be balancing" + ); +} diff --git a/src/simlin-engine/src/ltm/tests.rs b/src/simlin-engine/src/ltm/tests.rs index bdae54a53..b45c2f08a 100644 --- a/src/simlin-engine/src/ltm/tests.rs +++ b/src/simlin-engine/src/ltm/tests.rs @@ -358,7 +358,7 @@ fn test_link_polarity_detection() { Box::new(Expr2::Var(x_var.clone(), None, crate::ast::Loc::default())), Box::new(Expr2::Const( "2".to_string(), - 2.0, + crate::ast::Literal::new(2.0), crate::ast::Loc::default(), )), None, @@ -374,7 +374,7 @@ fn test_link_polarity_detection() { BinaryOp::Sub, Box::new(Expr2::Const( "0".to_string(), - 0.0, + crate::ast::Literal::new(0.0), crate::ast::Loc::default(), )), Box::new(Expr2::Var(x_var.clone(), None, crate::ast::Loc::default())), @@ -391,7 +391,7 @@ fn test_link_polarity_detection() { Box::new(Expr2::Var(x_var.clone(), None, crate::ast::Loc::default())), Box::new(Expr2::Const( "-3".to_string(), - -3.0, + crate::ast::Literal::new(-3.0), crate::ast::Loc::default(), )), None, @@ -415,7 +415,7 @@ fn test_polarity_mul_both_operands_referencing_source_is_unknown() { let loc = crate::ast::Loc::default(); let x_var: Ident = Ident::new("x"); let var = |n: &str| Expr2::Var(Ident::new(n), None, loc); - let cnst = |v: f64| Expr2::Const(format!("{v}"), v, loc); + let cnst = |v: f64| Expr2::Const(format!("{v}"), crate::ast::Literal::new(v), loc); let op2 = |op: BinaryOp, l: Expr2, r: Expr2| Expr2::Op2(op, Box::new(l), Box::new(r), None, loc); let empty_vars = HashMap::new(); @@ -455,7 +455,7 @@ fn test_polarity_div_numerator_value_sign_guards_denominator_flip() { let loc = crate::ast::Loc::default(); let y_var: Ident = Ident::new("y"); let var = |n: &str| Expr2::Var(Ident::new(n), None, loc); - let cnst = |v: f64| Expr2::Const(format!("{v}"), v, loc); + let cnst = |v: f64| Expr2::Const(format!("{v}"), crate::ast::Literal::new(v), loc); let op2 = |op: BinaryOp, l: Expr2, r: Expr2| Expr2::Op2(op, Box::new(l), Box::new(r), None, loc); let empty_vars = HashMap::new(); @@ -497,7 +497,7 @@ fn test_polarity_div_denominator_value_sign_guards_numerator_polarity() { let loc = crate::ast::Loc::default(); let x_var: Ident = Ident::new("x"); let var = |n: &str| Expr2::Var(Ident::new(n), None, loc); - let cnst = |v: f64| Expr2::Const(format!("{v}"), v, loc); + let cnst = |v: f64| Expr2::Const(format!("{v}"), crate::ast::Literal::new(v), loc); let op2 = |op: BinaryOp, l: Expr2, r: Expr2| Expr2::Op2(op, Box::new(l), Box::new(r), None, loc); let empty_vars = HashMap::new(); @@ -566,7 +566,7 @@ fn test_polarity_div_both_sides_referencing_source_is_unknown() { let loc = crate::ast::Loc::default(); let x_var: Ident = Ident::new("x"); let var = |n: &str| Expr2::Var(Ident::new(n), None, loc); - let cnst = |v: f64| Expr2::Const(format!("{v}"), v, loc); + let cnst = |v: f64| Expr2::Const(format!("{v}"), crate::ast::Literal::new(v), loc); let op2 = |op: BinaryOp, l: Expr2, r: Expr2| Expr2::Op2(op, Box::new(l), Box::new(r), None, loc); let empty_vars = HashMap::new(); @@ -662,900 +662,6 @@ fn test_get_variable_dependencies_no_ast() { ); } -#[test] -fn test_flip_polarity() { - // Test flip_polarity function (covers lines 1049-1054) - assert_eq!( - flip_polarity(LinkPolarity::Positive), - LinkPolarity::Negative - ); - assert_eq!( - flip_polarity(LinkPolarity::Negative), - LinkPolarity::Positive - ); - assert_eq!(flip_polarity(LinkPolarity::Unknown), LinkPolarity::Unknown); -} - -#[test] -fn test_is_positive_constant() { - // Test is_positive_constant function (covers lines 1058-1062) - use crate::ast::{Expr2, Loc}; - - let pos_const = Expr2::Const("5".to_string(), 5.0, Loc::default()); - assert!(is_positive_constant(&pos_const), "5.0 should be positive"); - - let neg_const = Expr2::Const("-5".to_string(), -5.0, Loc::default()); - assert!( - !is_positive_constant(&neg_const), - "-5.0 should not be positive" - ); - - let zero_const = Expr2::Const("0".to_string(), 0.0, Loc::default()); - assert!( - !is_positive_constant(&zero_const), - "0.0 should not be positive" - ); - - let var_expr = Expr2::Var(Ident::new("x"), None, Loc::default()); - assert!( - !is_positive_constant(&var_expr), - "Variable should not be positive constant" - ); -} - -#[test] -fn test_is_negative_constant() { - // Test is_negative_constant function (covers lines 1066-1070) - use crate::ast::{Expr2, Loc}; - - let neg_const = Expr2::Const("-3".to_string(), -3.0, Loc::default()); - assert!(is_negative_constant(&neg_const), "-3.0 should be negative"); - - let pos_const = Expr2::Const("3".to_string(), 3.0, Loc::default()); - assert!( - !is_negative_constant(&pos_const), - "3.0 should not be negative" - ); - - let zero_const = Expr2::Const("0".to_string(), 0.0, Loc::default()); - assert!( - !is_negative_constant(&zero_const), - "0.0 should not be negative" - ); - - let var_expr = Expr2::Var(Ident::new("y"), None, Loc::default()); - assert!( - !is_negative_constant(&var_expr), - "Variable should not be negative constant" - ); -} - -#[test] -fn test_analyze_link_polarity_arrayed() { - // Test analyze_link_polarity with Arrayed AST (covers lines 935-947) - use crate::ast::{Ast, Expr2, Loc}; - use crate::common::CanonicalElementName; - use std::collections::HashMap; - - let x_var = Ident::new("x"); - - // Create arrayed AST with consistent positive polarity - let mut elements = HashMap::new(); - elements.insert( - CanonicalElementName::from_raw("dim1"), - Expr2::Op2( - BinaryOp::Mul, - Box::new(Expr2::Var(x_var.clone(), None, Loc::default())), - Box::new(Expr2::Const("2".to_string(), 2.0, Loc::default())), - None, - Loc::default(), - ), - ); - elements.insert( - CanonicalElementName::from_raw("dim2"), - Expr2::Op2( - BinaryOp::Add, - Box::new(Expr2::Var(x_var.clone(), None, Loc::default())), - Box::new(Expr2::Const("10".to_string(), 10.0, Loc::default())), - None, - Loc::default(), - ), - ); - - let ast = Ast::Arrayed(vec![], elements, None, false); - let empty_vars = HashMap::new(); - let polarity = analyze_link_polarity(&ast, &x_var, &empty_vars); - assert_eq!( - polarity, - LinkPolarity::Positive, - "Consistent positive elements should be positive" - ); - - // Test with mixed polarities - let mut mixed_elements = HashMap::new(); - mixed_elements.insert( - CanonicalElementName::from_raw("dim1"), - Expr2::Var(x_var.clone(), None, Loc::default()), - ); - mixed_elements.insert( - CanonicalElementName::from_raw("dim2"), - Expr2::Op1( - crate::ast::UnaryOp::Negative, - Box::new(Expr2::Var(x_var.clone(), None, Loc::default())), - None, - Loc::default(), - ), - ); - - let mixed_ast = Ast::Arrayed(vec![], mixed_elements, None, false); - let mixed_polarity = analyze_link_polarity(&mixed_ast, &x_var, &empty_vars); - assert_eq!( - mixed_polarity, - LinkPolarity::Unknown, - "Mixed polarities should be Unknown" - ); -} - -#[test] -fn test_analyze_expr_polarity_if_then_else() { - // Test analyze_expr_polarity with If-Then-Else (covers lines 1033-1042) - use crate::ast::{Expr2, Loc}; - - let x_var = Ident::new("x"); - - // If with same polarity in both branches - let if_expr = Expr2::If( - Box::new(Expr2::Const("1".to_string(), 1.0, Loc::default())), - Box::new(Expr2::Var(x_var.clone(), None, Loc::default())), - Box::new(Expr2::Op2( - BinaryOp::Mul, - Box::new(Expr2::Var(x_var.clone(), None, Loc::default())), - Box::new(Expr2::Const("2".to_string(), 2.0, Loc::default())), - None, - Loc::default(), - )), - None, - Loc::default(), - ); - - let polarity = - analyze_expr_polarity_with_context(&if_expr, &x_var, LinkPolarity::Positive, None); - assert_eq!( - polarity, - LinkPolarity::Positive, - "Same polarity branches should return that polarity" - ); - - // If with different polarities in branches - let mixed_if = Expr2::If( - Box::new(Expr2::Const("1".to_string(), 1.0, Loc::default())), - Box::new(Expr2::Var(x_var.clone(), None, Loc::default())), - Box::new(Expr2::Op1( - crate::ast::UnaryOp::Negative, - Box::new(Expr2::Var(x_var.clone(), None, Loc::default())), - None, - Loc::default(), - )), - None, - Loc::default(), - ); - - let mixed_polarity = - analyze_expr_polarity_with_context(&mixed_if, &x_var, LinkPolarity::Positive, None); - assert_eq!( - mixed_polarity, - LinkPolarity::Unknown, - "Different polarity branches should be Unknown" - ); -} - -#[test] -fn test_analyze_expr_polarity_unary_not() { - // Test analyze_expr_polarity with unary NOT operator (covers lines 1026-1031) - use crate::ast::{Expr2, Loc, UnaryOp}; - - let x_var = Ident::new("x"); - - let not_expr = Expr2::Op1( - UnaryOp::Not, - Box::new(Expr2::Var(x_var.clone(), None, Loc::default())), - None, - Loc::default(), - ); - - let polarity = - analyze_expr_polarity_with_context(¬_expr, &x_var, LinkPolarity::Positive, None); - assert_eq!( - polarity, - LinkPolarity::Negative, - "NOT should flip polarity from positive to negative" - ); -} - -#[test] -fn test_analyze_expr_polarity_division_edge_cases() { - // Test division polarity analysis edge cases (covers lines 1013-1022) - use crate::ast::{Expr2, Loc}; - - let x_var = Ident::new("x"); - let y_var = Ident::new("y"); - - // Division with variable in numerator - let div_num = Expr2::Op2( - BinaryOp::Div, - Box::new(Expr2::Var(x_var.clone(), None, Loc::default())), - Box::new(Expr2::Const("10".to_string(), 10.0, Loc::default())), - None, - Loc::default(), - ); - - let pol_num = - analyze_expr_polarity_with_context(&div_num, &x_var, LinkPolarity::Positive, None); - assert_eq!( - pol_num, - LinkPolarity::Positive, - "Variable in numerator should keep polarity" - ); - - // Division with different variable in denominator (not the one we're tracking) - let div_other = Expr2::Op2( - BinaryOp::Div, - Box::new(Expr2::Const("100".to_string(), 100.0, Loc::default())), - Box::new(Expr2::Var(y_var.clone(), None, Loc::default())), - None, - Loc::default(), - ); - - let pol_other = - analyze_expr_polarity_with_context(&div_other, &x_var, LinkPolarity::Positive, None); - assert_eq!( - pol_other, - LinkPolarity::Unknown, - "Unrelated variable should give Unknown" - ); -} - -#[test] -fn test_analyze_expr_polarity_array_reducers() { - // Array reducers SUM, MEAN, MAX (single-arg), MIN (single-arg) are monotone - // in their argument: their polarity equals the inner expression's polarity. - // STDDEV and RANK are not monotone: they must return Unknown even when the - // argument has a known polarity. - use crate::ast::{Expr2, Loc, UnaryOp}; - use crate::builtins::BuiltinFn; - - let x_var = Ident::new("x"); - let pos_inner = || Expr2::Var(x_var.clone(), None, Loc::default()); - let neg_inner = || { - Expr2::Op1( - UnaryOp::Negative, - Box::new(Expr2::Var(x_var.clone(), None, Loc::default())), - None, - Loc::default(), - ) - }; - - // SUM passes through positive polarity. - let sum_pos = Expr2::App(BuiltinFn::Sum(Box::new(pos_inner())), None, Loc::default()); - assert_eq!( - analyze_expr_polarity_with_context(&sum_pos, &x_var, LinkPolarity::Positive, None), - LinkPolarity::Positive, - "SUM(x) should pass through positive polarity", - ); - - // SUM passes through negative polarity (e.g. SUM(-x)). - let sum_neg = Expr2::App(BuiltinFn::Sum(Box::new(neg_inner())), None, Loc::default()); - assert_eq!( - analyze_expr_polarity_with_context(&sum_neg, &x_var, LinkPolarity::Positive, None), - LinkPolarity::Negative, - "SUM(-x) should pass through negative polarity", - ); - - // MEAN with a single (array) argument passes through positive polarity. - let mean_pos = Expr2::App(BuiltinFn::Mean(vec![pos_inner()]), None, Loc::default()); - assert_eq!( - analyze_expr_polarity_with_context(&mean_pos, &x_var, LinkPolarity::Positive, None), - LinkPolarity::Positive, - "MEAN(x) should pass through positive polarity", - ); - - // MEAN with a single (array) argument passes through negative polarity. - let mean_neg = Expr2::App(BuiltinFn::Mean(vec![neg_inner()]), None, Loc::default()); - assert_eq!( - analyze_expr_polarity_with_context(&mean_neg, &x_var, LinkPolarity::Positive, None), - LinkPolarity::Negative, - "MEAN(-x) should pass through negative polarity", - ); - - // Array MAX (no second argument) passes through inner polarity. - let max_array_pos = Expr2::App( - BuiltinFn::Max(Box::new(pos_inner()), None), - None, - Loc::default(), - ); - assert_eq!( - analyze_expr_polarity_with_context(&max_array_pos, &x_var, LinkPolarity::Positive, None), - LinkPolarity::Positive, - "MAX(x) (array form) should pass through positive polarity", - ); - let max_array_neg = Expr2::App( - BuiltinFn::Max(Box::new(neg_inner()), None), - None, - Loc::default(), - ); - assert_eq!( - analyze_expr_polarity_with_context(&max_array_neg, &x_var, LinkPolarity::Positive, None), - LinkPolarity::Negative, - "MAX(-x) (array form) should pass through negative polarity", - ); - - // Array MIN (no second argument) passes through inner polarity. - let min_array_pos = Expr2::App( - BuiltinFn::Min(Box::new(pos_inner()), None), - None, - Loc::default(), - ); - assert_eq!( - analyze_expr_polarity_with_context(&min_array_pos, &x_var, LinkPolarity::Positive, None), - LinkPolarity::Positive, - "MIN(x) (array form) should pass through positive polarity", - ); - let min_array_neg = Expr2::App( - BuiltinFn::Min(Box::new(neg_inner()), None), - None, - Loc::default(), - ); - assert_eq!( - analyze_expr_polarity_with_context(&min_array_neg, &x_var, LinkPolarity::Positive, None), - LinkPolarity::Negative, - "MIN(-x) (array form) should pass through negative polarity", - ); - - // STDDEV is non-monotone: even with a positive-polarity argument, the result - // is Unknown because variance has no fixed sign w.r.t. its inputs. - let stddev = Expr2::App( - BuiltinFn::Stddev(Box::new(pos_inner())), - None, - Loc::default(), - ); - assert_eq!( - analyze_expr_polarity_with_context(&stddev, &x_var, LinkPolarity::Positive, None), - LinkPolarity::Unknown, - "STDDEV must always return Unknown polarity", - ); - - // RANK depends on the rest of the array, so polarity is undefined. - let direction = Expr2::Const("1".to_string(), 1.0, Loc::default()); - let rank = Expr2::App( - BuiltinFn::Rank(Box::new(pos_inner()), Box::new(direction)), - None, - Loc::default(), - ); - assert_eq!( - analyze_expr_polarity_with_context(&rank, &x_var, LinkPolarity::Positive, None), - LinkPolarity::Unknown, - "RANK must always return Unknown polarity", - ); -} - -/// Verify reducer polarity propagates through the actual parsed shape: -/// `SUM(x[*])` lowers to `Sum(Box::new(Subscript(x, [Wildcard], _, _)))`, -/// not `Sum(Box::new(Var(x, ...)))`. Without an `Expr2::Subscript` arm in -/// `analyze_expr_polarity_with_context`, the new reducer arms fall through -/// to Unknown for the production case the issue actually targets. -#[test] -fn test_analyze_expr_polarity_array_reducers_subscript_wildcard() { - use crate::ast::{Expr2, IndexExpr2, Loc, UnaryOp}; - use crate::builtins::BuiltinFn; - use LinkPolarity::{Negative, Positive, Unknown}; - - let x = Ident::new("x"); - let y = Ident::new("y"); - let sub = |id: &Ident| { - Expr2::Subscript( - id.clone(), - vec![IndexExpr2::Wildcard(Loc::default())], - None, - Loc::default(), - ) - }; - let app = |b: BuiltinFn| Expr2::App(b, None, Loc::default()); - let neg = |e: Expr2| Expr2::Op1(UnaryOp::Negative, Box::new(e), None, Loc::default()); - let one = || Expr2::Const("1".to_string(), 1.0, Loc::default()); - - // (label, expression, context_polarity, expected_polarity) - let cases: Vec<(&str, Expr2, LinkPolarity, LinkPolarity)> = vec![ - ( - "SUM(x[*]) +", - app(BuiltinFn::Sum(Box::new(sub(&x)))), - Positive, - Positive, - ), - ( - "SUM(x[*]) -", - app(BuiltinFn::Sum(Box::new(sub(&x)))), - Negative, - Negative, - ), - ( - "SUM(-x[*])", - app(BuiltinFn::Sum(Box::new(neg(sub(&x))))), - Positive, - Negative, - ), - ( - "SUM(y[*])", - app(BuiltinFn::Sum(Box::new(sub(&y)))), - Positive, - Unknown, - ), - ( - "MEAN(x[*])", - app(BuiltinFn::Mean(vec![sub(&x)])), - Positive, - Positive, - ), - ( - "MAX(x[*])", - app(BuiltinFn::Max(Box::new(sub(&x)), None)), - Positive, - Positive, - ), - ( - "MIN(x[*])", - app(BuiltinFn::Min(Box::new(sub(&x)), None)), - Positive, - Positive, - ), - ( - "STDDEV(x[*])", - app(BuiltinFn::Stddev(Box::new(sub(&x)))), - Positive, - Unknown, - ), - ( - "RANK(x[*], 1)", - app(BuiltinFn::Rank(Box::new(sub(&x)), Box::new(one()))), - Positive, - Unknown, - ), - ]; - - for (label, expr, ctx, want) in &cases { - assert_eq!( - analyze_expr_polarity_with_context(expr, &x, *ctx, None), - *want, - "{label}", - ); - } -} - -/// The Subscript arm must distinguish between indices that are independent -/// of `from_var` (literal, wildcard, expressions over other variables) and -/// indices that themselves reference `from_var`. In the latter case the -/// relationship between `from_var` and the subscripted result is non-monotone: -/// changing `from_var` shifts both the lookup target AND the index, so no -/// single polarity describes the result. The dominant cases (`SUM(arr[*])`, -/// `arr[Region]`, indices over OTHER variables) keep their original behavior -/// of returning `current_polarity` because their indices don't reference -/// `from_var`. -#[test] -fn test_analyze_expr_polarity_subscript_self_indexing() { - use crate::ast::{Expr2, IndexExpr2, Loc}; - use crate::builtins::BuiltinFn; - use LinkPolarity::{Positive, Unknown}; - - let arr = Ident::new("arr"); - let other = Ident::new("other"); - let i = Ident::new("i"); - - let var = |id: &Ident| Expr2::Var(id.clone(), None, Loc::default()); - let lit = |n: f64| Expr2::Const(format!("{n}"), n, Loc::default()); - - // arr[*] -- wildcard index, no reference to arr in the index. - let arr_wildcard = Expr2::Subscript( - arr.clone(), - vec![IndexExpr2::Wildcard(Loc::default())], - None, - Loc::default(), - ); - assert_eq!( - analyze_expr_polarity_with_context(&arr_wildcard, &arr, Positive, None), - Positive, - "arr[*] preserves current_polarity", - ); - - // arr[3] -- literal index, no reference to arr in the index. - let arr_literal = Expr2::Subscript( - arr.clone(), - vec![IndexExpr2::Expr(lit(3.0))], - None, - Loc::default(), - ); - assert_eq!( - analyze_expr_polarity_with_context(&arr_literal, &arr, Positive, None), - Positive, - "arr[3] preserves current_polarity", - ); - - // arr[i] where i is a different variable -- index references some OTHER - // variable, but not from_var (= arr). Polarity contract still holds. - let arr_other_index = Expr2::Subscript( - arr.clone(), - vec![IndexExpr2::Expr(var(&i))], - None, - Loc::default(), - ); - assert_eq!( - analyze_expr_polarity_with_context(&arr_other_index, &arr, Positive, None), - Positive, - "arr[i] (i != from_var) preserves current_polarity", - ); - - // arr[arr] -- index trivially references arr. Result is non-monotone - // because shifting arr shifts both the lookup target and the index. - let arr_self_var = Expr2::Subscript( - arr.clone(), - vec![IndexExpr2::Expr(var(&arr))], - None, - Loc::default(), - ); - assert_eq!( - analyze_expr_polarity_with_context(&arr_self_var, &arr, Positive, None), - Unknown, - "arr[arr] is non-monotone", - ); - - // arr[INT(arr[i])] -- the canonical self-indexing case. Index references - // arr through a nested subscript; relationship is non-monotone. - let inner = Expr2::Subscript( - arr.clone(), - vec![IndexExpr2::Expr(var(&i))], - None, - Loc::default(), - ); - let int_inner = Expr2::App(BuiltinFn::Int(Box::new(inner)), None, Loc::default()); - let arr_self_nested = Expr2::Subscript( - arr.clone(), - vec![IndexExpr2::Expr(int_inner)], - None, - Loc::default(), - ); - assert_eq!( - analyze_expr_polarity_with_context(&arr_self_nested, &arr, Positive, None), - Unknown, - "arr[INT(arr[i])] is non-monotone", - ); - - // other[*] where from_var is arr -- subscripted array is not from_var. - // Existing behavior: contributes Unknown because the arm conservatively - // can't classify references through other arrays. - let other_wildcard = Expr2::Subscript( - other.clone(), - vec![IndexExpr2::Wildcard(Loc::default())], - None, - Loc::default(), - ); - assert_eq!( - analyze_expr_polarity_with_context(&other_wildcard, &arr, Positive, None), - Unknown, - "other[*] (other != from_var) returns Unknown", - ); -} - -#[test] -fn test_graphical_function_polarity() { - use crate::variable::Table; - - // Test 1: Monotonically increasing function (positive polarity) - let increasing_table = - Table::new_for_test(vec![0.0, 1.0, 2.0, 3.0, 4.0], vec![0.0, 2.0, 4.0, 6.0, 8.0]); - assert_eq!( - analyze_graphical_function_polarity(&increasing_table), - LinkPolarity::Positive, - "Monotonically increasing function should have positive polarity" - ); - - // Test 2: Monotonically decreasing function (negative polarity) - let decreasing_table = Table::new_for_test( - vec![0.0, 1.0, 2.0, 3.0, 4.0], - vec![10.0, 8.0, 6.0, 4.0, 2.0], - ); - assert_eq!( - analyze_graphical_function_polarity(&decreasing_table), - LinkPolarity::Negative, - "Monotonically decreasing function should have negative polarity" - ); - - // Test 3: Non-monotonic function (unknown polarity) - let non_monotonic_table = - Table::new_for_test(vec![0.0, 1.0, 2.0, 3.0, 4.0], vec![0.0, 5.0, 3.0, 7.0, 2.0]); - assert_eq!( - analyze_graphical_function_polarity(&non_monotonic_table), - LinkPolarity::Unknown, - "Non-monotonic function should have unknown polarity" - ); - - // Test 4: Constant function (unknown polarity - no change) - let constant_table = Table::new_for_test(vec![0.0, 1.0, 2.0, 3.0], vec![5.0, 5.0, 5.0, 5.0]); - assert_eq!( - analyze_graphical_function_polarity(&constant_table), - LinkPolarity::Unknown, - "Constant function should have unknown polarity" - ); - - // Test 5: Single point (edge case) - let single_point_table = Table::new_for_test(vec![1.0], vec![2.0]); - assert_eq!( - analyze_graphical_function_polarity(&single_point_table), - LinkPolarity::Unknown, - "Single point should have unknown polarity" - ); - - // Test 6: Nearly constant with small variations (testing tolerance) - let nearly_constant_table = - Table::new_for_test(vec![0.0, 1.0, 2.0, 3.0], vec![5.0, 5.0001, 5.0002, 5.0003]); - assert_eq!( - analyze_graphical_function_polarity(&nearly_constant_table), - LinkPolarity::Positive, - "Nearly constant but increasing should have positive polarity" - ); -} - -#[test] -fn test_graphical_function_polarity_tolerates_import_noise() { - use crate::variable::Table; - - // A table that is monotone non-decreasing modulo round-trip numeric-import - // noise: the second segment dips by ~2e-7 against a 1.5-wide y-range. The - // y-range-relative epsilon (1e-6 * 1.5 = 1.5e-6) absorbs the dip, so the - // table reads as Positive. With the old absolute 1e-10 epsilon this dip - // broke monotonicity and the table read as Unknown (#492). - let import_noise_table = Table::new_for_test( - vec![0.0, 1.0, 2.0, 3.0, 4.0], - vec![0.0, 0.5000001, 0.4999999, 1.0, 1.5], - ); - assert_eq!( - analyze_graphical_function_polarity(&import_noise_table), - LinkPolarity::Positive, - "A monotone-modulo-import-noise table should read as Positive" - ); - - // A genuine ~0.7 reversal against a 1.0 y-range: far larger than the - // relative epsilon (1e-6), so the table is correctly Unknown. - let real_reversal_table = - Table::new_for_test(vec![0.0, 1.0, 2.0, 3.0], vec![0.0, 1.0, 0.3, 0.7]); - assert_eq!( - analyze_graphical_function_polarity(&real_reversal_table), - LinkPolarity::Unknown, - "A genuinely non-monotone table is still Unknown" - ); - - // A perfectly constant table: y_max - y_min == 0 so epsilon clamps to - // 1e-12; every dy == 0 is a plateau (not > 1e-12, not < -1e-12), so the - // table is still classified constant and reads as Unknown. - let constant_table = Table::new_for_test(vec![0.0, 1.0, 2.0], vec![5.0, 5.0, 5.0]); - assert_eq!( - analyze_graphical_function_polarity(&constant_table), - LinkPolarity::Unknown, - "A constant table is still Unknown" - ); -} - -#[test] -fn test_graphical_function_polarity_uses_slope_not_y_delta() { - use crate::variable::Table; - - // Non-uniform x-spacing where the y-delta heuristic and the slope heuristic - // DISAGREE on the verdict (#536). The middle segment is a genuine, steep, - // local DECREASE over a very narrow x-interval (dx = 0.001), so the table is - // NOT monotone -- the correct answer is Unknown. - // - // x = [0, 100, 100.001, 200] - // y = [0, 10, 9.99999, 20] - // - // y-range = 20, so the y-range-relative dy epsilon is 1e-6 * 20 = 2e-5. - // The middle dy is -1e-5, which is *below* that epsilon, so the y-delta - // heuristic SUPPRESSES the dip as a plateau and -- seeing only the two - // surrounding increases -- wrongly classifies the table as Positive. - // - // The middle segment's SLOPE, however, is -1e-5 / 0.001 = -0.01, far - // steeper (in magnitude) than the table's average slope (20/200 = 0.1) - // times the relative tolerance, so the slope heuristic correctly sees a - // real local decrease and returns Unknown. - let narrow_dip_table = Table::new_for_test( - vec![0.0, 100.0, 100.001, 200.0], - vec![0.0, 10.0, 9.99999, 20.0], - ); - assert_eq!( - analyze_graphical_function_polarity(&narrow_dip_table), - LinkPolarity::Unknown, - "a steep narrow local decrease must be caught by the slope heuristic, not \ - smoothed over by the y-delta heuristic" - ); - - // The mirror case: a steep narrow local INCREASE in an otherwise-decreasing - // table must likewise flip a naive Negative to Unknown. - let narrow_bump_table = Table::new_for_test( - vec![0.0, 100.0, 100.001, 200.0], - vec![20.0, 10.0, 10.00001, 0.0], - ); - assert_eq!( - analyze_graphical_function_polarity(&narrow_bump_table), - LinkPolarity::Unknown, - "a steep narrow local increase must flip an otherwise-decreasing table to Unknown" - ); - - // A genuinely monotone table with non-uniform x-spacing (no sign change in - // slope) must still classify correctly: every segment slopes up, so the - // verdict is Positive even though the dx values vary by orders of magnitude. - let monotone_nonuniform_table = - Table::new_for_test(vec![0.0, 0.001, 100.0, 200.0], vec![0.0, 5.0, 10.0, 20.0]); - assert_eq!( - analyze_graphical_function_polarity(&monotone_nonuniform_table), - LinkPolarity::Positive, - "a monotone table with non-uniform x-spacing stays Positive" - ); - - // Degenerate vertical segment (x[i] == x[i-1]) with differing y: two outputs - // for one input is an ambiguous lookup with undefined slope, so bail to - // Unknown. - let vertical_segment_table = - Table::new_for_test(vec![0.0, 1.0, 1.0, 2.0], vec![0.0, 1.0, 5.0, 6.0]); - assert_eq!( - analyze_graphical_function_polarity(&vertical_segment_table), - LinkPolarity::Unknown, - "a vertical (dx == 0, dy != 0) segment has undefined slope and must read as Unknown" - ); - - // A duplicated point (x[i] == x[i-1] AND y[i] == y[i-1]) is redundant, not - // ambiguous: it must be skipped as non-determining and not poison an - // otherwise-monotone verdict. - let duplicate_point_table = - Table::new_for_test(vec![0.0, 1.0, 1.0, 2.0], vec![0.0, 1.0, 1.0, 2.0]); - assert_eq!( - analyze_graphical_function_polarity(&duplicate_point_table), - LinkPolarity::Positive, - "a redundant duplicate point must not flip a monotone-increasing table" - ); -} - -/// GH #536 tolerance regression: a many-point uniformly-spaced monotone table -/// with a single small import-noise dip must still classify Positive even after -/// the slope-based tolerance was introduced (#536). -/// -/// Root cause: the original #536 fix set `slope_epsilon = 1e-6 * (y_max - -/// y_min) / x_span`, which is `1e-6 * avg_slope_mag`. For a uniformly-spaced -/// n-point table `dx = x_span / (n - 1)`, so the effective per-segment dy -/// threshold becomes `slope_epsilon * dx = 1e-6 * (y_max - y_min) / (n - 1)` -- -/// which is `(n - 1)x` tighter than the old #492 threshold `1e-6 * (y_max - -/// y_min)`. A 50-point table therefore has a threshold ~49x tighter, and a -/// noise dip that sat safely inside the #492 window now crosses it and flips the -/// classification to Unknown. -/// -/// The correct fix scales the tolerance by `avg_dx = x_span / (n - 1)`, giving -/// `slope_epsilon = 1e-6 * (y_max - y_min) / avg_dx`. Then for uniform spacing -/// `dx == avg_dx` and the per-segment dy threshold is `slope_epsilon * dx = -/// 1e-6 * (y_max - y_min)` -- exactly the old #492 behavior. For non-uniform -/// spacing the threshold still scales by `dx / avg_dx`, so narrow steep segments -/// still trip correctly (the #536 motivation is preserved). -#[test] -fn test_graphical_function_polarity_uniform_50pt_import_noise() { - use crate::variable::Table; - - // Build a 50-point uniformly-spaced monotone-increasing table on [0, 49] - // with y rising from 0.0 to 1.0. One interior point (index 25) has a - // downward import-noise dip of -5e-8. - // - // With 50 points: y_range = 1.0, x_span = 49.0, avg_dx = 1.0. - // Old #492 threshold: 1e-6 * 1.0 = 1e-6. The dip (-5e-8) is far below that - // threshold in magnitude, so the old code classified Positive. - // - // Buggy #536 slope_epsilon = 1e-6 * 1.0 / 49.0 ≈ 2.04e-8. - // The dip's slope = -5e-8 / 1.0 = -5e-8, which is < -2.04e-8, so the - // buggy code classifies Unknown (the #492 regression). - // - // Fixed #536 slope_epsilon = 1e-6 * 1.0 / avg_dx = 1e-6 * 1.0 / 1.0 = 1e-6. - // The dip's slope (-5e-8) is within [-1e-6, 1e-6], so the fixed code - // correctly classifies Positive. - let n = 50usize; - let x_span = (n - 1) as f64; // 49.0 - let noise_dip_idx = 25usize; - // The noise magnitude must be: - // - within the old #492 threshold: 1e-6 * y_range = 1e-6 * 1.0 = 1e-6 - // - outside the buggy #536 threshold: 1e-6 * y_range / (n-1) = 1e-6/49 ≈ 2.04e-8 - // Choosing 5e-8: between 2.04e-8 and 1e-6, so buggy code fails and correct - // code passes. - let noise_magnitude = 5e-8_f64; - - // Build a strictly-increasing base curve and then pull y[noise_dip_idx] - // BELOW y[noise_dip_idx - 1] by noise_magnitude. This makes the segment - // (noise_dip_idx - 1) -> noise_dip_idx have a genuinely negative slope: - // dy = -noise_magnitude, dx = 1.0, slope = -5e-8. - // The following segment noise_dip_idx -> (noise_dip_idx + 1) is then - // extra-positive (+2/49 + noise_magnitude), so the table overall is nearly - // monotone. - let xs: Vec = (0..n).map(|i| i as f64).collect(); - let base_y: Vec = (0..n).map(|i| i as f64 / x_span).collect(); - let ys: Vec = (0..n) - .map(|i| { - if i == noise_dip_idx { - // pull this point below its predecessor: slope (i-1)->i is -5e-8 - base_y[noise_dip_idx - 1] - noise_magnitude - } else { - base_y[i] - } - }) - .collect(); - - let table = Table::new_for_test(xs, ys); - assert_eq!( - analyze_graphical_function_polarity(&table), - LinkPolarity::Positive, - "a 50-point uniformly-spaced monotone table with a single {noise_magnitude:.0e} \ - import-noise dip (within the #492 tolerance 1e-6 * y_range = 1e-6) must \ - classify Positive; slope_epsilon must scale by avg_dx so uniform-spacing \ - behavior matches the pre-#536 y-delta threshold (GH #536 regression)" - ); -} - -#[test] -fn test_lookup_table_polarity_in_links() { - use crate::datamodel; - - // Create a model with a lookup table. The flow scales the lookup result - // by a positive CONSTANT (not by `water` itself): with `water` appearing - // in both factors of a product, the partial `L(w) + w*L'(w)` is - // sign-indefinite without value reasoning, so the sound answer would be - // Unknown -- this test isolates the table-monotonicity propagation. - let mut model_vars = vec![ - x_stock("water", "100", &[], &["outflow"], None), - x_flow("outflow", "0.5 * lookup(lookup, water)", None), - ]; - - // Create the lookup table auxiliary - let mut lookup_var = x_aux("lookup", "0", None); - if let datamodel::Variable::Aux(aux) = &mut lookup_var { - aux.gf = Some(datamodel::GraphicalFunction { - kind: datamodel::GraphicalFunctionKind::Continuous, - x_points: Some(vec![0.0, 50.0, 100.0, 150.0]), - y_points: vec![0.1, 0.2, 0.3, 0.4], - x_scale: datamodel::GraphicalFunctionScale { - min: 0.0, - max: 150.0, - }, - y_scale: datamodel::GraphicalFunctionScale { min: 0.1, max: 0.4 }, - }); - } - model_vars.push(lookup_var); - - let model = x_model("main", model_vars); - let sim_specs = sim_specs_with_units("months"); - let datamodel_project = x_project(sim_specs, &[model]); - let db = SimlinDb::default(); - let result = sync_from_datamodel(&db, &datamodel_project); - let model = result.models["main"].source; - - // Check per-link polarity via compute_link_polarities - let polarities = compute_link_polarities(&db, model, result.project); - let water_to_outflow_key = ("water".to_string(), "outflow".to_string()); - assert_eq!( - polarities[&water_to_outflow_key], - LinkPolarity::Positive, - "Monotonically increasing lookup table should preserve positive polarity" - ); - - // Verify loop polarity via model_detected_loops - let detected = model_detected_loops(&db, model, result.project); - assert_eq!(detected.loops.len(), 1, "Should have one loop"); - // water -> outflow: Positive (increasing lookup), outflow -> water: Negative (outflow) - assert_eq!( - detected.loops[0].polarity, - DetectedLoopPolarity::Balancing, - "Loop with one negative link should be balancing" - ); -} - /// A continuous graphical function over x = [0, 1, 2, ...] with the given /// y-points; used to build per-element GF fixtures for the #502 tests. #[cfg(test)] @@ -1929,7 +1035,11 @@ fn gf_var_for_test( ident: Ident::new(ident), ast: Some(Ast::ApplyToAll( dims, - Expr2::Const("0".to_string(), 0.0, Loc::default()), + Expr2::Const( + "0".to_string(), + crate::ast::Literal::new(0.0), + Loc::default(), + ), )), init_ast: None, eqn: None, @@ -1994,7 +1104,13 @@ fn test_lookup_table_polarity_defensive_subscript_branches() { )) }; let var = |id: &str| Expr2::Var(Ident::new(id), None, Loc::default()); - let int_const = |n: i64| Expr2::Const(n.to_string(), n as f64, Loc::default()); + let int_const = |n: i64| { + Expr2::Const( + n.to_string(), + crate::ast::Literal::new(n as f64), + Loc::default(), + ) + }; // (a) `curve[1]` -- a 1-based integer literal into the single dimension. // Picks NYC's (offset 0) decreasing table -> Negative. `curve[2]` picks @@ -2711,7 +1827,13 @@ fn test_builtin_polarity_max_min_scalar() { let x_var = Ident::new("x"); let x_expr = || Box::new(Expr2::Var(x_var.clone(), None, Loc::default())); - let const_5 = || Box::new(Expr2::Const("5".to_string(), 5.0, Loc::default())); + let const_5 = || { + Box::new(Expr2::Const( + "5".to_string(), + crate::ast::Literal::new(5.0), + Loc::default(), + )) + }; let empty_vars = HashMap::new(); // Max(x, 5): only x depends on from_var -> propagate x's polarity @@ -2751,7 +1873,13 @@ fn test_expr_references_var() { let x_var = Ident::new("x"); let y_var = Ident::new("y"); let x_expr = || Expr2::Var(x_var.clone(), None, Loc::default()); - let const_5 = || Expr2::Const("5".to_string(), 5.0, Loc::default()); + let const_5 = || { + Expr2::Const( + "5".to_string(), + crate::ast::Literal::new(5.0), + Loc::default(), + ) + }; assert!(!expr_references_var(&const_5(), &x_var)); assert!(expr_references_var(&x_expr(), &x_var)); @@ -2836,7 +1964,13 @@ fn test_max_min_polarity_with_non_monotone_arg() { let x_var = Ident::new("x"); let x_expr = || Box::new(Expr2::Var(x_var.clone(), None, Loc::default())); let abs_x = || Box::new(Expr2::App(BuiltinFn::Abs(x_expr()), None, Loc::default())); - let const_5 = || Box::new(Expr2::Const("5".to_string(), 5.0, Loc::default())); + let const_5 = || { + Box::new(Expr2::Const( + "5".to_string(), + crate::ast::Literal::new(5.0), + Loc::default(), + )) + }; let empty_vars = HashMap::new(); // MAX(ABS(x), x): ABS(x) is non-monotonically dependent on x, @@ -3001,7 +2135,13 @@ fn test_add_sub_div_polarity_with_non_monotone_arg() { let x_var = Ident::new("x"); let x_expr = || Box::new(Expr2::Var(x_var.clone(), None, Loc::default())); let abs_x = || Box::new(Expr2::App(BuiltinFn::Abs(x_expr()), None, Loc::default())); - let const_5 = || Box::new(Expr2::Const("5".to_string(), 5.0, Loc::default())); + let const_5 = || { + Box::new(Expr2::Const( + "5".to_string(), + crate::ast::Literal::new(5.0), + Loc::default(), + )) + }; let empty_vars = HashMap::new(); // x + ABS(x): ABS(x) non-monotonically depends on x → Unknown @@ -5912,7 +5052,7 @@ fn test_source_to_agg_polarity_discriminates_body_sign() { let loc = crate::ast::Loc::default(); let scale: Ident = Ident::new("scale"); let var = |n: &str| Expr2::Var(Ident::new(n), None, loc); - let cnst = |v: f64| Expr2::Const(format!("{v}"), v, loc); + let cnst = |v: f64| Expr2::Const(format!("{v}"), crate::ast::Literal::new(v), loc); let op2 = |op: BinaryOp, l: Expr2, r: Expr2| Expr2::Op2(op, Box::new(l), Box::new(r), None, loc); let pop_wild = || { @@ -5991,6 +5131,12 @@ fn test_source_to_agg_polarity_discriminates_body_sign() { ); } +/// Static link-polarity analysis tests, in a sibling file to keep this one +/// under the project line-count lint. A child module, so `use super::*` +/// reaches this file's imports and private helpers. +#[path = "polarity_tests.rs"] +mod polarity; + /// Implicit WITH-LOOKUP link-polarity tests (GH #910), in a sibling file to /// keep this one under the project line-count lint. A child module, so /// `use super::*` reaches this file's private helpers. diff --git a/src/simlin-engine/src/ltm_agg.rs b/src/simlin-engine/src/ltm_agg.rs index 1ca04cd0a..dde94a24e 100644 --- a/src/simlin-engine/src/ltm_agg.rs +++ b/src/simlin-engine/src/ltm_agg.rs @@ -23,7 +23,8 @@ //! synthetic names are assigned `$⁚ltm⁚agg⁚0`, `1`, ... in first-encounter //! order. AST-identical *synthetic* reducer subexpressions dedupe to a single //! agg node (canonicalization is via printed equation text, since `Expr2` is -//! not `Eq`). Variable-backed aggs are never deduped (see below). +//! not `Hash` and so cannot key a map directly). Variable-backed aggs are never +//! deduped (see below). //! //! Two kinds of aggregate node: //! - **Synthetic** (`is_synthetic == true`): the reducer is a *sub-expression* @@ -531,7 +532,7 @@ pub struct AggSource { /// One aggregate node: the stand-in for a maximal reducer subexpression. #[cfg_attr(feature = "debug-derive", derive(Debug))] -#[derive(Clone, PartialEq, salsa::Update)] +#[derive(Clone, PartialEq, Eq, salsa::Update)] pub struct AggNode { /// The aggregate node's name. For a synthetic agg this is /// `$⁚ltm⁚agg⁚{n}`; for a variable-backed agg this is the owning @@ -594,9 +595,10 @@ pub struct AggNode { /// /// # What the stored form does and does not normalize /// - /// Stored in [`Expr2::strip_loc_and_bounds`] form, which removes TWO of the - /// three ways this field can make the salsa-cached `AggNodesResult` - /// compare unequal to an identical rebuild. It does not remove the third. + /// Stored in [`Expr2::strip_loc_and_bounds`] form, which removes two of the + /// three ways this field could make the salsa-cached `AggNodesResult` + /// compare unequal to an identical rebuild; the third is closed at the root + /// instead. /// /// * `Loc` -- removed, and load-bearing. Two AST-identical occurrences in /// different equations carry different `Loc`s, so storing them raw would @@ -615,20 +617,17 @@ pub struct AggNode { /// equation order, and the cached value would become sensitive to /// unrelated edits elsewhere in the owning equation. Neither reader looks /// at one either -- `expr2_to_expr0` drops them outright. - /// * The `f64` on `Expr2::Const` -- **NOT removed, and not removable.** A - /// derived `PartialEq` over an `f64` is not reflexive on NaN, so a model - /// whose hoisted reducer contains a `nan` literal - /// (`out = 1 + SUM(pop[*] * nan)`) yields two enumerations that are - /// structurally identical and compare UNEQUAL -- permanently defeating - /// this query's backdating, so every revision bump re-executes - /// `model_element_causal_edges` / `model_ltm_reference_sites` / - /// `model_ltm_variables`. No normalization can fix it here: dropping a - /// NaN literal would change what the equation means. It is the GH - /// #987/#981 class (the same mechanism that forces `db::stages_tests`' - /// stdlib oracle onto `npv`), and it is fixed at the ROOT by making AST - /// float literals compare by bit pattern. Today's broken behaviour is - /// pinned by `a_nan_literal_in_a_reducer_defeats_agg_backdating`, which - /// that fix must invert. + /// * The float literal on `Expr2::Const` -- **not normalized here, and not + /// normalizable here:** dropping a `nan` literal would change what the + /// equation means. It is closed at the ROOT instead. The literal is an + /// [`crate::ast::Literal`], compared by BIT PATTERN, so a model whose + /// hoisted reducer contains a `nan` (`out = 1 + SUM(pop[*] * nan)`) + /// enumerates to a value equal to an identical rebuild and backdates like + /// any other. With a bare `f64` it could not (`NaN != NaN`), and every + /// revision bump re-executed `model_element_causal_edges` / + /// `model_ltm_reference_sites` / `model_ltm_variables` -- the GH #987/#981 + /// class. Pinned by + /// `a_nan_literal_in_a_reducer_does_not_defeat_agg_backdating`. pub reducer: BuiltinFn, } @@ -732,7 +731,7 @@ impl AggNode { /// filtered out by the `is_synthetic` checks downstream, leaving the inline /// reducer on the conservative direct-scoring path -- a name-ordering bug). #[cfg_attr(feature = "debug-derive", derive(Debug))] -#[derive(Clone, PartialEq, Default, salsa::Update)] +#[derive(Clone, PartialEq, Eq, Default, salsa::Update)] pub struct AggNodesResult { /// Aggregate nodes in first-encounter (deterministic) order. pub aggs: Vec, @@ -5754,10 +5753,11 @@ mod tests { /// leaves the salsa-cached `AggNodesResult` equal. /// /// **This is one of the two ways the backdating claim can fail, and it - /// measures only this one.** The other is `f64` non-reflexivity on a NaN - /// literal, which normalization cannot touch; - /// [`a_nan_literal_in_a_reducer_defeats_agg_backdating`] pins that arm - /// (today: broken) and names the root fix. + /// measures only this one.** The other is float non-reflexivity on a NaN + /// literal, which normalization cannot touch and which is closed at the + /// root by [`crate::ast::Literal`]'s bit-pattern equality; + /// [`a_nan_literal_in_a_reducer_does_not_defeat_agg_backdating`] pins that + /// arm. /// /// The two spellings differ ONLY in a leading term that mints no /// aggregate node of its own -- a `SIZE` call, which is recognized as a @@ -5819,40 +5819,27 @@ mod tests { ); } - // CHARACTERIZATION, NOT A CONTRACT: this asserts today's BROKEN behaviour - // so the fix has something to turn red. + // The third channel by which `AggNode::reducer` (GH #983) could make the + // salsa-cached, `PartialEq`-compared `AggNodesResult` unequal to an + // identical rebuild: the `f64` on `Expr2::Const`, which + // `strip_loc_and_bounds` cannot normalize away (there is no rewrite that + // removes a NaN literal without changing what the equation means). // - // `AggNode::reducer` (GH #983) puts an `Expr2` inside the salsa-cached, - // `PartialEq`-compared `AggNodesResult`, and `Expr2::Const` holds a bare - // `f64`. A derived `PartialEq` over an `f64` is not reflexive on NaN, so a - // model whose hoisted reducer contains a `nan` literal enumerates to a - // value that is structurally identical to an independent rebuild and - // compares UNEQUAL to it. Salsa's backdating criterion is exactly that - // equality, so such a model can never backdate `enumerate_agg_nodes`: - // every revision bump, from any unrelated edit anywhere in the project, - // re-executes `model_element_causal_edges`, `model_ltm_reference_sites` - // and `model_ltm_variables` -- the whole LTM compile. + // It is closed at the ROOT rather than here: `ast::Literal` compares float + // literals by BIT PATTERN, so `nan == nan` and a bit-identical AST is + // equal to itself (GH #987/#981). Salsa's backdating criterion is exactly + // that equality, so without the root fix every revision bump -- from any + // unrelated edit anywhere in the project -- re-executed + // `model_element_causal_edges`, `model_ltm_reference_sites` and + // `model_ltm_variables` for any model with a `nan` in a hoisted reducer. // - // `strip_loc_and_bounds` closes the `Loc` and `ArrayBounds` channels (see - // `the_carried_reducer_is_normalized_so_offset_only_edits_backdate`); it - // cannot close this one, because there is no normalization that removes a - // NaN literal without changing what the equation means. This is the GH - // #987/#981 class -- the same mechanism that forces `db::stages_tests`' - // stdlib value oracle onto `npv` rather than `smth1` -- and the root fix is - // to make AST float literals compare by BIT PATTERN. - // - // When that lands, this test MUST be inverted, not deleted: switch to - // `assert_eq`, rename the test (it asserts the opposite of - // `defeats_agg_backdating` afterwards) along with the - // `nan_backdating_is_broken` project name, and update the three rustdocs - // that cite it by name -- `AggNode::reducer`, - // `the_carried_reducer_is_normalized_so_offset_only_edits_backdate`, and - // the `ltm_agg.rs` bullet in `simlin-engine/CLAUDE.md`. Its failure is the - // signal that the root fix reached this query. + // This test was written inverted (`assert_ne!`) as the characterization of + // that defect; flipping it is how the root fix demonstrates it reached a + // real consumer rather than only its own unit test. #[test] - fn a_nan_literal_in_a_reducer_defeats_agg_backdating() { + fn a_nan_literal_in_a_reducer_does_not_defeat_agg_backdating() { let build = || { - TestProject::new("nan_backdating_is_broken") + TestProject::new("nan_backdating") .named_dimension("Region", &["nyc", "boston"]) .array_aux("pop[Region]", "1") .scalar_aux("out", "1 + SUM(pop[*] * nan)") @@ -5868,13 +5855,12 @@ mod tests { "the NaN-bearing reducer must still be hoisted for this to measure anything" ); - assert_ne!( + assert_eq!( agg_nodes(&build()), agg_nodes(&build()), - "TODAY'S BEHAVIOUR, not the desired one: two enumerations of an \ - identical NaN-bearing model compare unequal, so `enumerate_agg_nodes` \ - can never backdate for this model. Making AST float literals compare \ - by bit pattern (GH #987/#981) must flip this to `assert_eq!`." + "two enumerations of an identical NaN-bearing model must compare \ + equal, so `enumerate_agg_nodes` backdates for this model like any \ + other" ); } } diff --git a/src/simlin-engine/src/ltm_augment_tests.rs b/src/simlin-engine/src/ltm_augment_tests.rs index d192928a1..6c465ec65 100644 --- a/src/simlin-engine/src/ltm_augment_tests.rs +++ b/src/simlin-engine/src/ltm_augment_tests.rs @@ -79,7 +79,7 @@ fn classify_expr0_rejects_out_of_range_integer_literal() { let dims = vec![vec!["1".to_string(), "2".to_string()]]; let indices = vec![IndexExpr0::Expr(Expr0::Const( "999".to_string(), - 999.0, + crate::ast::Literal::new(999.0), Loc::default(), ))]; @@ -100,7 +100,7 @@ fn classify_expr0_rejects_out_of_range_integer_literal() { // Sanity: the same classifier still accepts an in-range integer. let in_range = vec![IndexExpr0::Expr(Expr0::Const( "1".to_string(), - 1.0, + crate::ast::Literal::new(1.0), Loc::default(), ))]; let in_range_shape = classify_expr0_subscript_shape(&in_range, &dims, None, None); @@ -135,7 +135,7 @@ fn classify_expr0_canonicalizes_integer_literal_subscript() { ]; let indices = vec![IndexExpr0::Expr(Expr0::Const( "01".to_string(), - 1.0, + crate::ast::Literal::new(1.0), Loc::default(), ))]; @@ -496,7 +496,11 @@ fn test_classify_reducer_stddev() { #[test] fn test_classify_reducer_rank() { let inner = subscript_wildcard("population"); - let direction = Expr2::Const("1".to_string(), 1.0, Loc::default()); + let direction = Expr2::Const( + "1".to_string(), + crate::ast::Literal::new(1.0), + Loc::default(), + ); let expr = Expr2::App( BuiltinFn::Rank(Box::new(inner), Box::new(direction)), None, @@ -553,8 +557,16 @@ fn test_classify_reducer_nested_in_expression() { // Reducer is NOT at the top level, so is_bare should be false. let inner = subscript_wildcard("population"); let sum_expr = Expr2::App(BuiltinFn::Sum(Box::new(inner)), None, Loc::default()); - let two = Expr2::Const("2".to_string(), 2.0, Loc::default()); - let one = Expr2::Const("1".to_string(), 1.0, Loc::default()); + let two = Expr2::Const( + "2".to_string(), + crate::ast::Literal::new(2.0), + Loc::default(), + ); + let one = Expr2::Const( + "1".to_string(), + crate::ast::Literal::new(1.0), + Loc::default(), + ); let mul = Expr2::Op2( crate::ast::BinaryOp::Mul, Box::new(two), @@ -583,7 +595,11 @@ fn test_classify_reducer_nested_in_scalar_max() { // The SUM is nested inside a non-reducer App, so is_bare should be false. let inner = subscript_wildcard("population"); let sum_expr = Expr2::App(BuiltinFn::Sum(Box::new(inner)), None, Loc::default()); - let zero = Expr2::Const("0".to_string(), 0.0, Loc::default()); + let zero = Expr2::Const( + "0".to_string(), + crate::ast::Literal::new(0.0), + Loc::default(), + ); let expr = Expr2::App( BuiltinFn::Max(Box::new(sum_expr), Some(Box::new(zero))), None, @@ -1514,7 +1530,11 @@ fn test_body_aware_nested_reducer_falls_back_to_delta_ratio() { fn test_classify_reducer_returns_body_ast() { let pop = subscript_wildcard("pop"); let weight = subscript_wildcard("weight"); - let one = Expr2::Const("1".to_string(), 1.0, Loc::default()); + let one = Expr2::Const( + "1".to_string(), + crate::ast::Literal::new(1.0), + Loc::default(), + ); let coeff = Expr2::Op2( crate::ast::BinaryOp::Sub, Box::new(one), diff --git a/src/simlin-engine/src/mdl/writer.rs b/src/simlin-engine/src/mdl/writer.rs index 7debfe0df..07f7f12e1 100644 --- a/src/simlin-engine/src/mdl/writer.rs +++ b/src/simlin-engine/src/mdl/writer.rs @@ -775,13 +775,15 @@ fn is_var(expr: &Expr0, name: &str) -> bool { /// Returns true when `expr` is `Const(_, v, _)` with value exactly `v`. fn is_const(expr: &Expr0, v: f64) -> bool { - matches!(expr, Expr0::Const(_, n, _) if (*n - v).abs() < f64::EPSILON) + matches!(expr, Expr0::Const(_, n, _) if (n.value() - v).abs() < f64::EPSILON) } /// Structurally compare two Expr0 trees, ignoring source locations. fn exprs_equal(a: &Expr0, b: &Expr0) -> bool { match (a, b) { - (Expr0::Const(_, av, _), Expr0::Const(_, bv, _)) => (av - bv).abs() < f64::EPSILON, + (Expr0::Const(_, av, _), Expr0::Const(_, bv, _)) => { + (av.value() - bv.value()).abs() < f64::EPSILON + } (Expr0::Var(aid, _), Expr0::Var(bid, _)) => aid == bid, (Expr0::App(UntypedBuiltinFn(af, aa), _), Expr0::App(UntypedBuiltinFn(bf, ba), _)) => { af == bf && aa.len() == ba.len() && aa.iter().zip(ba).all(|(x, y)| exprs_equal(x, y)) diff --git a/src/simlin-engine/src/mdl/writer_proptest.rs b/src/simlin-engine/src/mdl/writer_proptest.rs index 363367fe0..0c6289c86 100644 --- a/src/simlin-engine/src/mdl/writer_proptest.rs +++ b/src/simlin-engine/src/mdl/writer_proptest.rs @@ -205,11 +205,31 @@ const GRAMMAR_VARS: &[&str] = &["a", "b", "c"]; fn const_leaf() -> impl Strategy { prop_oneof![ - (0i64..1000).prop_map(|n| Expr0::Const(n.to_string(), n as f64, Loc::default())), - Just(Expr0::Const("0.5".to_owned(), 0.5, Loc::default())), - Just(Expr0::Const("1.5".to_owned(), 1.5, Loc::default())), - Just(Expr0::Const("3.5".to_owned(), 3.5, Loc::default())), - Just(Expr0::Const("2.25".to_owned(), 2.25, Loc::default())), + (0i64..1000).prop_map(|n| Expr0::Const( + n.to_string(), + crate::ast::Literal::new(n as f64), + Loc::default() + )), + Just(Expr0::Const( + "0.5".to_owned(), + crate::ast::Literal::new(0.5), + Loc::default() + )), + Just(Expr0::Const( + "1.5".to_owned(), + crate::ast::Literal::new(1.5), + Loc::default() + )), + Just(Expr0::Const( + "3.5".to_owned(), + crate::ast::Literal::new(3.5), + Loc::default() + )), + Just(Expr0::Const( + "2.25".to_owned(), + crate::ast::Literal::new(2.25), + Loc::default() + )), ] } diff --git a/src/simlin-engine/src/parser/mod.rs b/src/simlin-engine/src/parser/mod.rs index ff13a0575..1fd8a4195 100644 --- a/src/simlin-engine/src/parser/mod.rs +++ b/src/simlin-engine/src/parser/mod.rs @@ -7,7 +7,7 @@ //! This parser replaces the LALRPOP-generated parser with equivalent functionality. //! It uses the existing lexer and produces the same AST types (Expr0, IndexExpr0). -use crate::ast::{BinaryOp, Expr0, IndexExpr0, UnaryOp}; +use crate::ast::{BinaryOp, Expr0, IndexExpr0, Literal, UnaryOp}; use crate::builtins::{Loc, UntypedBuiltinFn}; use crate::common::{EquationError, ErrorCode, RawIdent}; use crate::lexer::{Lexer, LexerType, Spanned, Token}; @@ -564,7 +564,11 @@ impl<'input> Parser<'input> { let (lpos, tok, rpos) = *self.advance().unwrap(); if let Token::Num(s) = tok { match s.parse::() { - Ok(n) => Ok(Expr0::Const(s.to_string(), n, Loc::new(lpos, rpos))), + Ok(n) => Ok(Expr0::Const( + s.to_string(), + Literal::new(n), + Loc::new(lpos, rpos), + )), Err(_) => Err(EquationError { start: lpos as u16, end: rpos as u16, @@ -579,7 +583,7 @@ impl<'input> Parser<'input> { let (lpos, _, rpos) = *self.advance().unwrap(); Ok(Expr0::Const( "NaN".to_string(), - f64::NAN, + Literal::new(f64::NAN), Loc::new(lpos, rpos), )) } diff --git a/src/simlin-engine/src/parser/tests.rs b/src/simlin-engine/src/parser/tests.rs index 191c5727d..e8cf0a847 100644 --- a/src/simlin-engine/src/parser/tests.rs +++ b/src/simlin-engine/src/parser/tests.rs @@ -19,19 +19,21 @@ fn parse_eq(input: &str) -> Result, Vec> { #[test] fn test_parse_number() { let ast = parse_eq("42").unwrap().unwrap(); - assert!(matches!(ast, Expr0::Const(s, n, _) if s == "42" && n == 42.0)); + assert!(matches!(ast, Expr0::Const(s, n, _) if s == "42" && n == Literal::new(42.0))); } #[test] fn test_parse_float() { let ast = parse_eq("2.75").unwrap().unwrap(); - assert!(matches!(ast, Expr0::Const(s, n, _) if s == "2.75" && (n - 2.75).abs() < 0.001)); + assert!( + matches!(ast, Expr0::Const(s, n, _) if s == "2.75" && (n.value() - 2.75).abs() < 0.001) + ); } #[test] fn test_parse_scientific_notation() { let ast = parse_eq("1e10").unwrap().unwrap(); - assert!(matches!(ast, Expr0::Const(s, n, _) if s == "1e10" && n == 1e10)); + assert!(matches!(ast, Expr0::Const(s, n, _) if s == "1e10" && n == Literal::new(1e10))); } #[test] @@ -39,7 +41,7 @@ fn test_parse_nan() { let ast = parse_eq("NaN").unwrap().unwrap(); if let Expr0::Const(s, n, _) = ast { assert_eq!(s, "NaN"); - assert!(n.is_nan()); + assert!(n.value().is_nan()); } else { panic!("Expected Const"); } @@ -60,7 +62,7 @@ fn test_parse_quoted_identifier() { #[test] fn test_parse_parenthesized() { let ast = parse_eq("(42)").unwrap().unwrap().strip_loc(); - let expected = Expr0::Const("42".to_string(), 42.0, Loc::default()); + let expected = Expr0::Const("42".to_string(), Literal::new(42.0), Loc::default()); assert_eq!(ast, expected); } @@ -93,7 +95,7 @@ fn test_parse_subscript_simple() { RawIdent::new_from_str("a"), vec![IndexExpr0::Expr(Expr0::Const( "1".to_string(), - 1.0, + Literal::new(1.0), Loc::default(), ))], Loc::default(), @@ -107,8 +109,16 @@ fn test_parse_subscript_multiple() { let expected = Expr0::Subscript( RawIdent::new_from_str("a"), vec![ - IndexExpr0::Expr(Expr0::Const("1".to_string(), 1.0, Loc::default())), - IndexExpr0::Expr(Expr0::Const("2".to_string(), 2.0, Loc::default())), + IndexExpr0::Expr(Expr0::Const( + "1".to_string(), + Literal::new(1.0), + Loc::default(), + )), + IndexExpr0::Expr(Expr0::Const( + "2".to_string(), + Literal::new(2.0), + Loc::default(), + )), ], Loc::default(), ); @@ -174,8 +184,8 @@ fn test_parse_subscript_range() { let expected = Expr0::Subscript( RawIdent::new_from_str("a"), vec![IndexExpr0::Range( - Expr0::Const("1".to_string(), 1.0, Loc::default()), - Expr0::Const("2".to_string(), 2.0, Loc::default()), + Expr0::Const("1".to_string(), Literal::new(1.0), Loc::default()), + Expr0::Const("2".to_string(), Literal::new(2.0), Loc::default()), Loc::default(), )], Loc::default(), @@ -231,7 +241,7 @@ fn test_parse_subscript_trailing_comma() { RawIdent::new_from_str("a"), vec![IndexExpr0::Expr(Expr0::Const( "1".to_string(), - 1.0, + Literal::new(1.0), Loc::default(), ))], Loc::default(), @@ -285,7 +295,11 @@ fn test_parse_subscript_transpose() { RawIdent::new_from_str("matrix"), vec![ IndexExpr0::Wildcard(Loc::default()), - IndexExpr0::Expr(Expr0::Const("1".to_string(), 1.0, Loc::default())), + IndexExpr0::Expr(Expr0::Const( + "1".to_string(), + Literal::new(1.0), + Loc::default(), + )), ], Loc::default(), )), @@ -474,11 +488,23 @@ fn test_parse_exponentiation_right_associative() { let ast = parse_eq("2^3^4").unwrap().unwrap().strip_loc(); let expected = Expr0::Op2( BinaryOp::Exp, - Box::new(Expr0::Const("2".to_string(), 2.0, Loc::default())), + Box::new(Expr0::Const( + "2".to_string(), + Literal::new(2.0), + Loc::default(), + )), Box::new(Expr0::Op2( BinaryOp::Exp, - Box::new(Expr0::Const("3".to_string(), 3.0, Loc::default())), - Box::new(Expr0::Const("4".to_string(), 4.0, Loc::default())), + Box::new(Expr0::Const( + "3".to_string(), + Literal::new(3.0), + Loc::default(), + )), + Box::new(Expr0::Const( + "4".to_string(), + Literal::new(4.0), + Loc::default(), + )), Loc::default(), )), Loc::default(), @@ -683,7 +709,7 @@ fn var(name: &str) -> Expr0 { } fn num(s: &str, v: f64) -> Expr0 { - Expr0::Const(s.to_string(), v, Loc::default()) + Expr0::Const(s.to_string(), Literal::new(v), Loc::default()) } /// A stacked unary minus is legal Vensim (`x = - -3`), and the MDL importer's @@ -968,9 +994,21 @@ fn test_exponent_does_not_swallow_multiplicative() { fn test_parse_if_simple() { let ast = parse_eq("if 1 then 2 else 3").unwrap().unwrap().strip_loc(); let expected = Expr0::If( - Box::new(Expr0::Const("1".to_string(), 1.0, Loc::default())), - Box::new(Expr0::Const("2".to_string(), 2.0, Loc::default())), - Box::new(Expr0::Const("3".to_string(), 3.0, Loc::default())), + Box::new(Expr0::Const( + "1".to_string(), + Literal::new(1.0), + Loc::default(), + )), + Box::new(Expr0::Const( + "2".to_string(), + Literal::new(2.0), + Loc::default(), + )), + Box::new(Expr0::Const( + "3".to_string(), + Literal::new(3.0), + Loc::default(), + )), Loc::default(), ); assert_eq!(ast, expected); @@ -989,8 +1027,16 @@ fn test_parse_if_with_condition() { Box::new(Expr0::Var(RawIdent::new_from_str("b"), Loc::default())), Loc::default(), )), - Box::new(Expr0::Const("1".to_string(), 1.0, Loc::default())), - Box::new(Expr0::Const("0".to_string(), 0.0, Loc::default())), + Box::new(Expr0::Const( + "1".to_string(), + Literal::new(1.0), + Loc::default(), + )), + Box::new(Expr0::Const( + "0".to_string(), + Literal::new(0.0), + Loc::default(), + )), Loc::default(), ); assert_eq!(ast, expected); @@ -1003,9 +1049,21 @@ fn test_parse_if_parenthesized() { .unwrap() .strip_loc(); let expected = Expr0::If( - Box::new(Expr0::Const("1".to_string(), 1.0, Loc::default())), - Box::new(Expr0::Const("2".to_string(), 2.0, Loc::default())), - Box::new(Expr0::Const("3".to_string(), 3.0, Loc::default())), + Box::new(Expr0::Const( + "1".to_string(), + Literal::new(1.0), + Loc::default(), + )), + Box::new(Expr0::Const( + "2".to_string(), + Literal::new(2.0), + Loc::default(), + )), + Box::new(Expr0::Const( + "3".to_string(), + Literal::new(3.0), + Loc::default(), + )), Loc::default(), ); assert_eq!(ast, expected); @@ -1024,8 +1082,16 @@ fn test_parse_if_with_logical() { Box::new(Expr0::Var(RawIdent::new_from_str("b"), Loc::default())), Loc::default(), )), - Box::new(Expr0::Const("1".to_string(), 1.0, Loc::default())), - Box::new(Expr0::Const("0".to_string(), 0.0, Loc::default())), + Box::new(Expr0::Const( + "1".to_string(), + Literal::new(1.0), + Loc::default(), + )), + Box::new(Expr0::Const( + "0".to_string(), + Literal::new(0.0), + Loc::default(), + )), Loc::default(), ); assert_eq!(ast, expected); @@ -1158,13 +1224,21 @@ fn test_complex_time_subscript() { vec![Expr0::Op2( BinaryOp::Mod, Box::new(Expr0::Var(RawIdent::new_from_str("TIME"), Loc::default())), - Box::new(Expr0::Const("5".to_string(), 5.0, Loc::default())), + Box::new(Expr0::Const( + "5".to_string(), + Literal::new(5.0), + Loc::default(), + )), Loc::default(), )], ), Loc::default(), )), - Box::new(Expr0::Const("1".to_string(), 1.0, Loc::default())), + Box::new(Expr0::Const( + "1".to_string(), + Literal::new(1.0), + Loc::default(), + )), Loc::default(), ))], Loc::default(), @@ -1621,7 +1695,11 @@ fn test_subscript_with_expression() { vec![IndexExpr0::Expr(Expr0::Op2( BinaryOp::Add, Box::new(Expr0::Var(RawIdent::new_from_str("b"), Loc::default())), - Box::new(Expr0::Const("1".to_string(), 1.0, Loc::default())), + Box::new(Expr0::Const( + "1".to_string(), + Literal::new(1.0), + Loc::default(), + )), Loc::default(), ))], Loc::default(), @@ -1646,11 +1724,23 @@ fn test_deeply_nested_if() { Box::new(Expr0::Var(RawIdent::new_from_str("a"), Loc::default())), Box::new(Expr0::If( Box::new(Expr0::Var(RawIdent::new_from_str("b"), Loc::default())), - Box::new(Expr0::Const("1".to_string(), 1.0, Loc::default())), - Box::new(Expr0::Const("2".to_string(), 2.0, Loc::default())), + Box::new(Expr0::Const( + "1".to_string(), + Literal::new(1.0), + Loc::default(), + )), + Box::new(Expr0::Const( + "2".to_string(), + Literal::new(2.0), + Loc::default(), + )), + Loc::default(), + )), + Box::new(Expr0::Const( + "3".to_string(), + Literal::new(3.0), Loc::default(), )), - Box::new(Expr0::Const("3".to_string(), 3.0, Loc::default())), Loc::default(), ); assert_eq!(ast, expected); @@ -1770,13 +1860,15 @@ fn test_safe_div_chain() { #[test] fn test_scientific_negative_exponent() { let ast = parse_eq("1.5e-3").unwrap().unwrap(); - assert!(matches!(ast, Expr0::Const(s, n, _) if s == "1.5e-3" && (n - 0.0015).abs() < 1e-10)); + assert!( + matches!(ast, Expr0::Const(s, n, _) if s == "1.5e-3" && (n.value() - 0.0015).abs() < 1e-10) + ); } #[test] fn test_leading_decimal() { let ast = parse_eq(".5").unwrap().unwrap(); - assert!(matches!(ast, Expr0::Const(s, n, _) if s == ".5" && (n - 0.5).abs() < 1e-10)); + assert!(matches!(ast, Expr0::Const(s, n, _) if s == ".5" && (n.value() - 0.5).abs() < 1e-10)); } #[test] @@ -1802,8 +1894,16 @@ fn test_if_with_complex_condition() { )), Loc::default(), )), - Box::new(Expr0::Const("1".to_string(), 1.0, Loc::default())), - Box::new(Expr0::Const("0".to_string(), 0.0, Loc::default())), + Box::new(Expr0::Const( + "1".to_string(), + Literal::new(1.0), + Loc::default(), + )), + Box::new(Expr0::Const( + "0".to_string(), + Literal::new(0.0), + Loc::default(), + )), Loc::default(), ); assert_eq!(ast, expected); @@ -1907,13 +2007,13 @@ fn test_multiple_subscripts_with_ranges() { RawIdent::new_from_str("a"), vec![ IndexExpr0::Range( - Expr0::Const("1".to_string(), 1.0, Loc::default()), - Expr0::Const("2".to_string(), 2.0, Loc::default()), + Expr0::Const("1".to_string(), Literal::new(1.0), Loc::default()), + Expr0::Const("2".to_string(), Literal::new(2.0), Loc::default()), Loc::default(), ), IndexExpr0::Range( - Expr0::Const("3".to_string(), 3.0, Loc::default()), - Expr0::Const("4".to_string(), 4.0, Loc::default()), + Expr0::Const("3".to_string(), Literal::new(3.0), Loc::default()), + Expr0::Const("4".to_string(), Literal::new(4.0), Loc::default()), Loc::default(), ), ], diff --git a/src/simlin-engine/src/patch.rs b/src/simlin-engine/src/patch.rs index ebee75a97..3be8e9ad9 100644 --- a/src/simlin-engine/src/patch.rs +++ b/src/simlin-engine/src/patch.rs @@ -1920,7 +1920,11 @@ mod tests { let mut exprs = std::collections::HashMap::new(); exprs.insert( CanonicalElementName::from_raw("north"), - Expr2::Const("2".to_string(), 2.0, crate::ast::Loc::default()), + Expr2::Const( + "2".to_string(), + crate::ast::Literal::new(2.0), + crate::ast::Loc::default(), + ), ); let ast = Ast::Arrayed(vec![], exprs, None, false); diff --git a/src/simlin-engine/src/results.rs b/src/simlin-engine/src/results.rs index ebfbd8a5b..3b220425c 100644 --- a/src/simlin-engine/src/results.rs +++ b/src/simlin-engine/src/results.rs @@ -17,6 +17,11 @@ pub enum Method { RungeKutta4, } +/// Its `f64` time specs are compared with the DERIVED (IEEE) `PartialEq`; see +/// the "Float equality in this crate" section on `crate::ast::Literal` for the +/// project's position on float equality in cache keys (GH #642). A NaN here is +/// not reachable from a valid `SimSpecs`, so only the signed-zero direction +/// applies, and it is inert for a start/stop/dt. #[cfg_attr(feature = "debug-derive", derive(Debug))] #[derive(Clone, PartialEq)] pub struct Specs { diff --git a/src/simlin-engine/src/units.rs b/src/simlin-engine/src/units.rs index d415999cf..729465526 100644 --- a/src/simlin-engine/src/units.rs +++ b/src/simlin-engine/src/units.rs @@ -277,7 +277,8 @@ impl Context { fn const_int_eval(ast: &Expr0) -> EquationResult { match ast { Expr0::Const(_, n, loc) => { - if approx_eq(*n, n.round()) { + let n = n.value(); + if approx_eq(n, n.round()) { Ok(n.round() as i32) } else { eqn_err!(ExpectedInteger, loc.start, loc.end) diff --git a/src/simlin-engine/src/variable.rs b/src/simlin-engine/src/variable.rs index 7b23ae585..8795f9b60 100644 --- a/src/simlin-engine/src/variable.rs +++ b/src/simlin-engine/src/variable.rs @@ -22,6 +22,15 @@ use crate::module_functions::MacroRegistry; use crate::units::parse_units; use crate::{ErrorCode, eqn_err, units}; +/// A graphical function's points, as the compiler and the VM read them. +/// +/// The `f64`s keep the derived (IEEE) `PartialEq`, so a lookup table holding a +/// NaN y-point makes this -- and every `ModelStage0` / `ModelStage1` / +/// `db::query::ParsedVariableResult` carrying it -- unequal to a bit-identical +/// rebuild, defeating salsa backdating. The XMILE reader admits one, since +/// `f64::from_str` accepts `"NaN"` in a `` list unvalidated. Accepted +/// knowingly, on the same terms as the bytecode types: see the "Float equality +/// in this crate" section on [`crate::ast::Literal`]. #[cfg_attr(feature = "debug-derive", derive(Debug))] #[derive(Clone, PartialEq, salsa::Update)] pub struct Table { @@ -1256,8 +1265,8 @@ fn test_classify_dependencies_matrix() { } let loc = Loc::new(0, 1); - let const_one = Expr2::Const("1".to_string(), 1.0, loc); - let const_zero = Expr2::Const("0".to_string(), 0.0, loc); + let const_one = Expr2::Const("1".to_string(), crate::ast::Literal::new(1.0), loc); + let const_zero = Expr2::Const("0".to_string(), crate::ast::Literal::new(0.0), loc); let module_inputs_with_input: BTreeSet> = [Ident::new("input")].into_iter().collect(); @@ -1950,7 +1959,7 @@ fn test_tables() { ident: Ident::new("lookup_function_table"), ast: Some(Ast::Scalar(Expr0::Const( "0".to_string(), - 0.0, + crate::ast::Literal::new(0.0), Loc::new(0, 1), ))), init_ast: None, From 68866e3f25c1930eaab8743de214987b30fa77ed Mon Sep 17 00:00:00 2001 From: Bobby Powers Date: Mon, 27 Jul 2026 20:38:00 -0700 Subject: [PATCH 07/17] engine: per-element pin and wrap correctness Three silent-wrong-answer defects in the LTM per-element machinery, each of which produced a degraded link score indistinguishable from a real one. GH #974 -- a per-element link-score partial pinned every arrayed dep of the target with the target's FULL element tuple, which is only right when the dep declares exactly the target's dimensions in the target's order. A dep declaring a strict subset got an over-arity subscript whose fragment failed to compile, so the score read a constant 0 behind four Assembly warnings; a dep declaring the same dimensions in another order got a subscript that COMPILED and read the transposed element, with no diagnostic at all. A bare reference to the live source had the mirror defect: it projected by dimension name only, so a positionally-mapped target found no coordinate, left the reference bare, and froze it into a multi-slot PREVIOUS that cannot compile in a scalar fragment. Both arms now go through one projection, which asks the existing single row derivation which element each of the DEP's own axes reads -- so an identity axis and a positionally-mapped one are one arm, and the rule accepts exactly the mapped pairs the occurrence-driven pin does. A dep whose every axis resolves may subscript a bare reference; a dep with an unresolved axis still gets the dimension-name indices of an already-subscripted reference substituted (that only needs the axes the reference spells) but keeps its bare references bare, which fails loudly rather than reading a wrong element. The pin table also replaces deriving the dimension names by parsing the separator out of our own printed element tuple. ----- The ceteris-paribus wrap held a LOOKUP's table argument verbatim, which is right for the table HEAD -- a graphical-function table has no value slot, so wrapping it cannot compile and zeroes the whole fragment -- but wrong for its subscript INDEX expressions, which codegen evaluates at the current step to select the element. Every partial of such a target therefore varied with the index's own movement, attributing it to whichever source the partial isolates. The indices now go through the wrap's own index pass, which already carries the element and dimension-name guards, so only genuine value reads change spelling. That pass freezes an ident only when it is in the wrap's dep set, and the dependency extractor never walks a table expression -- so an index variable referenced ONLY inside a table argument is not a dependency and would have been left live. The descent therefore widens its own dep set with the idents under these indices, scoped to that argument, with the element and dimension-name guards running first so a selector still cannot wrap. Leaving the extractor itself alone is deliberate: an index dropped from a variable's dependencies is a runlist-ordering question rather than an LTM one. Because the freeze now covers any index ident, and an enclosing freeze covers the descents the arm does not reach, the per-element path's separate REFUSAL of a runtime table index is deleted along with its frozen plumbing, one verdict variant and the static-index predicate it needed -- a refusal that also threw away the scoreable sites on the same edge. The descent uses an empty occurrence lookup because the IR records nothing under a table argument by design, so the desync guard's premise does not hold there. Widening the dep set also made an existing gap reachable: an already-qualified dim-element index had no standalone guard on the paths that suppress qualification, so it froze; the guard now sits beside the element and dimension-name ones. Note for anyone re-reading this later: a table argument carrying a runtime index does not compile on this engine at all -- the target's own fragment fails with DoesNotExist first -- so the wrong score this prevents is not reachable today. Five constructions were tried and all five fail that way, which is why the model-level fixture asserts generated text rather than a score series, and asserts that upstream refusal alongside it. ----- The LTM per-axis classifier asked whether a subscript index named one of the target's iterated dimensions BEFORE asking whether the axis declares it as an element, the opposite of what the compiler does. XMILE lets a dimension declare an element whose name is also a dimension name, and there the orders disagree: without a mapping the classifier declined and the reference fell to the conservative cross-product, but WITH one it described the axis as iterated over a dimension the compiler never iterates there, so read slices and element edges named rows the simulation does not read. The precedence moves into one shared function that both the classifier and the per-element pin rule read, so a colliding name cannot be resolved two ways; the pin rule's rustdoc no longer documents the divergence because there is none. Element-first is what the engine executes, and the binding evidence is a run rather than the spec: a new oracle test compiles the collision and asserts the VM reads the element, then asserts the reference-site IR describes that same read. The spec points the same way but settles only the adjacent variable-vs-element pair outright, and the shared function's rustdoc says which passages are a rule and which are an argument from the namespace rule. ltm_agg.rs crossed the per-file line cap, so its test module moves to a #[path]-mounted sibling, matching ltm_augment's four. --- src/simlin-engine/CLAUDE.md | 6 +- src/simlin-engine/src/db/ltm/link_scores.rs | 263 +- src/simlin-engine/src/db/ltm_char_tests.rs | 387 ++ src/simlin-engine/src/dimensions.rs | 92 + src/simlin-engine/src/ltm_agg.rs | 2896 +-------------- src/simlin-engine/src/ltm_agg_tests.rs | 3205 +++++++++++++++++ src/simlin-engine/src/ltm_augment.rs | 452 ++- .../src/ltm_augment_occurrence.rs | 26 + .../src/ltm_augment_pin_tests.rs | 422 ++- .../src/ltm_augment_post_transform.rs | 488 +-- src/simlin-engine/src/ltm_augment_tests.rs | 220 +- src/simlin-engine/src/variable.rs | 2 +- 12 files changed, 4983 insertions(+), 3476 deletions(-) create mode 100644 src/simlin-engine/src/ltm_agg_tests.rs diff --git a/src/simlin-engine/CLAUDE.md b/src/simlin-engine/CLAUDE.md index 3dc56f97c..28d153135 100644 --- a/src/simlin-engine/CLAUDE.md +++ b/src/simlin-engine/CLAUDE.md @@ -91,7 +91,7 @@ Diagnostics (`collect_all_diagnostics`) run on the user's `SourceProject`, so sy - **`src/errors.rs`** - Human-readable error formatting: `FormattedError`/`FormattedErrors`, `FormattedErrorKind`, `UnitErrorKind`. `format_diagnostic()` converts a salsa `Diagnostic` to `FormattedError`; `format_diagnostic_with_datamodel()` adds source snippets from the datamodel. `collect_formatted_errors()` is the bulk entry point that aggregates all diagnostics into a `FormattedErrors` value. `FormattedError.message` is TERMINAL-formatted (source snippet + `~~~` underline + model/variable summary line); the separate `details` field carries just the bare reason for unit errors (e.g. "computed units 'x' don't match specified units") so GUI consumers (via libsimlin's `SimlinErrorDetail.details`) can render it without the snippet noise. `FormattedError.severity` carries the originating `db::DiagnosticSeverity`, and the summary line's severity word is rendered FROM it (`severity_word`) rather than hardcoded (GH #919): a `Warning` -- the conveyor/queue LTM-degraded advisory, the conveyor spec advisories, a unit *consistency*/*inference* mismatch -- reads "warning in model ..." / "units warning in model ...", while an `Error` (an equation error, a unit *definition* syntax error) keeps "error in model ...". The word therefore tracks the diagnostic's severity, not the arm it came from, so no consumer that renders `message` verbatim can present an advisory as a compilation failure. `FormattedErrors::push` is the SOLE place `has_model_errors`/`has_variable_errors` are set, and it counts `Error` severity only -- those flags gate failure-shaped decisions (the CLI's redundant-`NotSimulatable` suppression), so a warning must never flip them. `format_simulation_error` has no `Diagnostic` behind it and is unconditionally `Error`. Canonical implementation shared by both `simlin-mcp` and `libsimlin` (which re-exports from here). - **`src/datamodel.rs`** - Core structures: `Project`, `Model`, `Variable`, `Equation` (including `Arrayed` variant with `default_equation` for EXCEPT semantics and `has_except_default` bool flag), `Dimension` (with `mappings: Vec` replacing the old `maps_to` field, and `parent: Option` for indexed subdimension relationships), `DimensionMapping`, `DataSource`/`DataSourceKind`, `UnitMap`, `MacroSpec` (a macro-marked `Model`'s calling convention: `parameters`/`primary_output`/`additional_outputs`; `Model.macro_spec` is `None` for every ordinary model; `salsa::Update` so it rides directly on the `SourceModel` input). `Model::new_macro(name, params, additional_outputs, body_variables)` is the shared port-synthesis + `MacroSpec`-construction step used by *both* the MDL converter and the XMILE reader: it sets `can_be_module_input` on each formal-parameter body variable (synthesizing a `Flow`/`Aux` placeholder port when absent) so `collect_module_idents` treats the macro as an ordinary sub-model. View element types (`Aux`, `Stock`, `Flow`, `Alias`, `Cloud`) carry an optional `ViewElementCompat` with original Vensim sketch dimensions/bits for MDL roundtrip fidelity. `StockFlow` has an optional `font` string for the Vensim default font spec. - **`src/variable.rs`** - Variable variants (`Stock`, `Flow`, `Aux`, `Module`), `ModuleInput`, `Table` (graphical functions). `classify_dependencies()` is the primary API for extracting dependency categories from an AST in a single walk, returning a `DepClassification` with five sets: `all` (every referenced ident), `init_referenced`, `previous_referenced`, `previous_only` (idents only inside PREVIOUS), and `init_only` (idents only inside INIT/PREVIOUS). `parse_var_with_module_context` accepts a `module_idents` set so `PREVIOUS(module_var)` rewrites through a scalar helper aux instead of `LoadPrev`. Per-element graphical functions: `build_tables` materializes one `Table` per element of an arrayed GF and `reorder_arrayed_element_tables` places each at the element's *flat row-major declared-dimension index*, NOT its `Equation::Arrayed` `elems` Vec position -- because the runtime selects a per-element table by the row-major dimension offset (`vm.rs` `Lookup`/`LookupArray`: `graphical_functions[base_gf + element_offset]`). Elements lacking a GF get an empty placeholder so the index stays aligned. The salsa dependency-table path mirrors this in `db.rs::extract_tables_from_source_var`. -- **`src/dimensions.rs`** - `DimensionsContext` for dimension matching, subdimension detection, and element-level mappings. Supports indexed subdimensions via `parent` field (child maps to first N elements of parent). `has_mapping_to()` checks for element-level dimension mappings between two dimensions. `mapped_element_correspondence(iterated_dim, source_dim)` (GH #527) is the reusable element-level correspondence for a mapped dimension pair -- per iterated element, the source element the executed simulation reads (both declaration directions, single-hop only, POSITIONAL mappings only; an explicit element map returns `None` because the executed A2A lowering resolves positionally and ignores it -- GH #753 and the tracked positional-vs-element-map execution inconsistency (GH #756) are the gate for re-enabling; `None` ⇒ callers keep their conservative broadcast, a superset of the true edges). It is what keeps the LTM element graph (`expand_same_element`'s mapped diagonal) and link-score dimensions (`link_score_dimensions` -- whose mapped arm is additionally gated on the edge having a `Bare`-classified site via `model_edge_shapes`, so the arrayed retarget fires exactly when the element graph expands the diagonal rather than the element-mapped `DynamicIndex` cross-product) in lockstep with the classifier (`classify_iterated_dim_shape`, whose mapped arm gates on this same correspondence in BOTH declaration directions since GH #757 -- via `ltm_agg::classify_axis_access` / `iterated_axis_slot_elements` -- so a reverse-declared positional pair classifies `Bare` and gets the diagonal) and -- for positional mappings -- the simulation; the agg machinery's mapped sliced reducers (GH #534) consume it through `ltm_agg::iterated_axis_slot_elements` (the per-source-element preimage inversion), so the same gate governs hoisting and the emitters' slot remap. `SubdimensionRelation` caches parent-child offset mappings for both named (element containment) and indexed (declared parent) dimensions +- **`src/dimensions.rs`** - `DimensionsContext` for dimension matching, subdimension detection, and element-level mappings. Supports indexed subdimensions via `parent` field (child maps to first N elements of parent). `has_mapping_to()` checks for element-level dimension mappings between two dimensions. `resolve_axis_index_name(name, axis_dim, target_iterates)` is the engine's SINGLE element-vs-dimension-name precedence for a bare-identifier subscript index -- the axis's own declared elements first, then a dimension the enclosing equation iterates -- matching `compiler::subscript::normalize_subscripts3`; both LTM callers (`ltm_agg::classify_axis_access` and the pin rule) read it, so a colliding element name cannot be resolved two ways (GH #986). `mapped_element_correspondence(iterated_dim, source_dim)` (GH #527) is the reusable element-level correspondence for a mapped dimension pair -- per iterated element, the source element the executed simulation reads (both declaration directions, single-hop only, POSITIONAL mappings only; an explicit element map returns `None` because the executed A2A lowering resolves positionally and ignores it -- GH #753 and the tracked positional-vs-element-map execution inconsistency (GH #756) are the gate for re-enabling; `None` ⇒ callers keep their conservative broadcast, a superset of the true edges). It is what keeps the LTM element graph (`expand_same_element`'s mapped diagonal) and link-score dimensions (`link_score_dimensions` -- whose mapped arm is additionally gated on the edge having a `Bare`-classified site via `model_edge_shapes`, so the arrayed retarget fires exactly when the element graph expands the diagonal rather than the element-mapped `DynamicIndex` cross-product) in lockstep with the classifier (`classify_iterated_dim_shape`, whose mapped arm gates on this same correspondence in BOTH declaration directions since GH #757 -- via `ltm_agg::classify_axis_access` / `iterated_axis_slot_elements` -- so a reverse-declared positional pair classifies `Bare` and gets the diagonal) and -- for positional mappings -- the simulation; the agg machinery's mapped sliced reducers (GH #534) consume it through `ltm_agg::iterated_axis_slot_elements` (the per-source-element preimage inversion), so the same gate governs hoisting and the emitters' slot remap. `SubdimensionRelation` caches parent-child offset mappings for both named (element containment) and indexed (declared parent) dimensions - **`src/model.rs`** - Model compilation stages (`ModelStage0` -> `ModelStage1` -> `ModuleStage2`). It performs NO dependency analysis of its own: `ModelStage1::set_dependencies` reads the production dependency graph (`db::dep_graph::model_dependency_graph`) once per module instantiation and copies its runlists into `ModuleStage2`. Until GH #568 it ran a second, independent walk here -- its own transitive closure (`all_deps`), its own cross-model output resolution (`module_output_deps`), its own `topo_sort` runlists and its own `CircularDependency` -- and that second gate genuinely disagreed with production's, rejecting the element-acyclic recurrence SCCs `resolve_recurrence_sccs` resolves. There is one gate now, pinned by `project.rs`'s `the_circular_dependency_gate_is_the_production_one` in both the resolves and the rejects direction. `collect_module_idents` pre-scans datamodel variables to identify which names will expand to modules (preventing incorrect `LoadPrev` compilation). Unit checking uses salsa tracked functions in `db.rs`. The two `#[cfg(test)]` `ModelStage0` constructors are the salsa-FREE twin of `db::stages::model_stage0`: `new_in_project(project_models, x_model, ..)` builds a stage from a `datamodel::Model` with no database, resolving module-function calls against the whole project's `MacroRegistry` (which is what makes it a faithful oracle for a model that CALLS a macro defined in a sibling model), and `new(x_model, ..)` is the single-model wrapper the many one-model fixtures use. They derive the module-ident set, the macro registry and the duplicate-ident errors along completely different routes than the query, which is exactly why `db::stages_tests` can use them as an oracle. The former salsa-cached `ModelStage0::new_cached` -- a test-only third copy of the query's construction -- was deleted; its coverage now points at the live query. `enumerate_modules_inner` records a model's instantiation BEFORE descending into it, so that a module cycle terminates: the "have I seen this model" test is the recursion guard, so a model still being walked has to count as seen. (A cycle through `main` already terminated, because `enumerate_modules` records `main` up front; one below `main` did not.) This matches its salsa twin `db::assemble::enumerate_module_instances_inner`. Recording early cannot LOSE an instantiation because the insert is unconditional at every module site -- only the recursion is guarded; the set-of-input-sets value is a separate fact, and it is why the ORDER instantiations arrive in is unobservable. - **`src/project.rs`** - `Project` struct aggregating models. `from_salsa(datamodel, db, source_project, cb)` builds a Project from a pre-synced salsa database: it READS `db::stages::model_stage1` for every project model and clones each memo (the clone is mandatory, not incidental -- `model_deps.take()`, `set_dependencies` and the `model_cb` all mutate it, while a `returns(ref)` memo is shared with every other reader). Each stage is kept paired with the `SourceModel` handle it came from, because `set_dependencies` reads the production dependency graph and that query is keyed on the handle, not on the model's name. It used to build its own whole-project `ModelStage0` map and lower it inline, a second salsa-native copy that had silently drifted from `db::stages` on three fields. Its model order is deterministic because the `topo_sort` seed is sorted first -- `topo_sort` breaks ties by visit order and the seed came from a `HashMap`'s keys (the `Project`-path twin of the GH #595 fix in `db::dep_graph`); the Initials runlists inherit the dependency graph's own determinism since GH #568 unified the two gates. `compiler::Module::new` is the sole reader of `ModelStage1::instantiations`; its callers are `#[cfg(test)] TestProject::build_module` and three direct calls in the `compiler`'s dimension tests, all `#[cfg(test)]`, while production `src/db/assemble.rs` never builds a `compiler::Module` at all. It REFUSES a model whose dependency graph carries a resolved recurrence SCC (`ModelStage1::set_dependencies` records a `NotSimulatable` for one): GH #568 unified the cycle GATE but not the EMITTER, and the per-element interleave `db::assemble::combine_scc_fragment` performs has no monolithic equivalent, so emitting the members whole would read a co-member's element before assigning it -- a silent wrong answer where the second gate used to give an accidental refusal. A model that can REACH a module cycle is gated out of dependency resolution first, exactly as the production entry points gate it (`db::diagnostic::module_cycle_diagnostic`, GH #806): the dependency graph's recurrence-SCC refinement descends into the recursive `model_module_map`, and salsa turns that into an unrecoverable dependency-graph panic -- a process abort under `panic = abort`, reachable from the public `From`. Such a model records the cycle as its error and keeps empty runlists. `from_datamodel(datamodel)` is a convenience wrapper that creates a local DB and syncs. **Neither is reachable from production**: inside the crate every caller is a test, and the only in-repo user of the public `From` impl is the engine's own `tests/integration/simulate_ltm.rs`, feeding the `ltm_finding::discover_loops` convenience wrapper (the shipped analysis path, libsimlin's `simlin_analyze_discover_loops`, goes through `discover_loops_with_graph` and builds no `Project`). Treat it as a monolith kept honest as a test oracle, not as live code. Production compiles via `db::compile_project_incremental` with `ltm_enabled`/`ltm_discovery_mode` on `SourceProject`. - **`src/results.rs`** - `Results` (variable offsets + timeseries data), `Specs` (time/integration config) @@ -181,8 +181,8 @@ The unit subsystem is partial-result throughout: a single bad declaration or one - `graph.rs` - public `CausalGraph` type owning model adjacency, stock identity, an optional variable AST map for polarity, and optional recursively-built sub-graphs for referenced sub-models. `CausalGraph::from_model` populates both for **every** referenced sub-model (DynamicModule and stockless passthrough alike, since GH #698 -- the passthrough's sub-graph drives the discovery-mode per-exit-port pathway recompute; a pathless module's sub-graph enumerates no pathways and is harmless). The lightweight edge-only `db::analysis` constructors (`causal_graph_from_edges` / `causal_graph_from_element_edges`) leave both EMPTY; the production discovery path enriches the element-level graph via `causal_graph_from_element_edges_with_modules` (sharing `model_variables_and_module_graphs` with `causal_graph_with_modules`). The `variables()` / `module_graph()` accessors expose the variable map and a module instance's sub-graph for that recompute. Drives `find_loops_with_limit` / `compute_cycle_partitions` / `all_link_polarities` / `enumerate_pathways_to_outputs`. `Loop` struct carries `id`, `links`, `stocks`, `polarity`, and `dimensions` (non-empty for A2A loops where per-element evaluation is needed; empty for scalar or mixed loops). - `tests.rs` - integration-style tests spanning all submodules; preserved as a single file to avoid splitting shared test fixtures. - **`src/ltm_finding.rs`** - Strongest-path loop discovery algorithm (Eberlein & Schoenberg 2020). Post-processes simulation results containing link score synthetic variables to find the most important loops at each timestep. `discover_loops_with_graph(results, causal_graph, stocks, ltm_vars, dims, expansion, sub_model_output_ports, budget)` is the primary entry point: `ltm_vars` and `dims` enable A2A link score expansion (per-element edges from `parse_link_offsets`); when empty, all link scores are treated as scalar. A Bare A2A link score is dimensioned over the TARGET's dims, so `expand_a2a_link_offsets` subscripts the TARGET node per element but PROJECTS the SOURCE node onto `from`'s OWN declared dims -- bare for a scalar feeder (`scale → growth`, GH #790), the same-element diagonal for an equal-dim feeder, the broadcast/partial-collapse form for a lower-dim feeder, and the positionally-mapped diagonal for a `State→Region` pair (GH #527) -- by reusing the element graph's own `db::expand_same_element` rule, so the discovery search graph's node names match `model_element_causal_edges` node-for-node (GH #754; before this, subscripting BOTH endpoints with the score's full tuple minted phantom from-nodes like `scale[a]`/`boost[r,a]`/`x[s]` that named no real node, so every loop through such a feeder dangled and was silently undiscoverable). The from-var declared dims + dimension-mapping context ride in a `LinkExpansionContext` (`declared_dims` + `dim_ctx`) that `analyze_model` builds via the public `analysis::build_link_expansion_context` (the SAME `variable_dimensions` / `project_dimensions_context` queries the element graph reads); the db-less `discover_loops(&Results, &Project)` convenience path passes `LinkExpansionContext::default()` (no A2A expansion runs there). Element-mapped (non-positional) pairs never reach the projection: `db::ltm::link_score_dimensions` declines to retarget them (no Bare A2A score; the GH #758 loud skip fires instead) under the GH #756 positional-only gate. `SearchGraph` provides DFS-based strongest-path traversal from stock nodes. The post-simulation score recompute (path → `FoundLoop`) applies the **per-exit-port pathway selection** (`recompute_module_input_edge_series`, GH #698): a loop edge `x → m` into a module is recomputed by max-abs-selecting over the sub-model's `m·$⁚ltm⁚path⁚{entry}⁚{idx}` pathway scores that end at the exit port the loop reads (recovered from the next link `m → y`), instead of reading the module *composite* (which max-abs-selects across ALL ports and so can pick a wrong-signed port for a multi-output module -- flipping the loop polarity vs exhaustive). The pathway indices come from the module's sub-graph via the same `enumerate_pathways_to_outputs_with_truncation` over the same sorted output-port set the sub-model emitted against, so they match index-for-index (the discovery-mode equivalent of exhaustive's `compute_module_link_overrides`). That port set is NOT re-derived parent-scoped (which would shift indices when another project model reads an extra output port -- GH #698 / PR #705 r3353097150): `discover_loops_with_graph` takes a `SubModelOutputPorts` map (sub-model canonical name -> emitted sorted port set) that `analyze_model` builds from the SAME emission decision via `analysis::build_sub_model_output_ports` -> `db::ltm::sub_model_output_ports` (identity-by-construction); the db-less `discover_loops(&Results, &Project)` convenience path reconstructs the same project-wide-union + stdlib-`output` semantics in `project_sub_model_output_ports`. Falls back to the base composite offset for single-output / pathless / indeterminate-port / sub-model-absent-from-map edges. Because the discovery graph is element-level, the recompute `strip_subscript`s `link.from`/`link.to`/`next.from`/`next.to` before its name matches (arrayed loop nodes carry `[elem]`; mirrors the exhaustive twin's stripping -- PR #705 r3353758167); this is currently latent parity defense, since an arrayed loop through a multi-output module is not yet discoverable end-to-end (a scalar module output feeding an arrayed reader scores a constant-0 link, dropping the loop -- GH #716). Returns a `DiscoveryResult` whose `FoundLoop`s carry per-timestep link/loop/pathway scores, ranked **competitive-first** (`rank_and_filter`): loops sharing their cycle partition with at least one other discovered loop come first, ordered by mean partition-relative importance; loops trivially ALONE in their partition (relative score exactly +/-1 by construction -- zero information, e.g. C-LEARN's isolated gas-uptake decay loops) sort after all competing loops and are dropped first under the `MAX_LOOPS` cap. Each `FoundLoop` carries a result-scoped dense `partition` index into `DiscoveryResult::partitions` (`DiscoveredPartition { stocks, loop_count }`, first-appearance order; NOT stable across runs/edits -- key on the stock set for durable identity), threaded through `analysis::ModelAnalysis::partitions` / `LoopSummary::partition`, the FFI `SimlinDiscoveredPartition` / `SimlinDiscoveredLoop.partition`, and pysimlin `Analysis.partitions` / `Loop.partition`. Also hosts the link-set synthetic-node collapse: `trim_synthetic_aggs_from_loop_links` collapses `$⁚ltm⁚agg⁚{n}` nodes out of a single loop's link *cycle*, and the public `collapse_synthetic_links(Vec)` generalizes that to ALL synthetic/macro/module-internal nodes (`ltm::is_synthetic_node_name`) over an arbitrary link *set* -- each chain `X -> $⁚…internal… -> Y` collapses to one composite edge `X -> Y` with product polarity and the per-timestep max-magnitude path score (the composite link score, LTM ref 6.3/6.4); a purely-internal cycle/source is dropped. `CollapsibleLink { from, to, polarity, score: Option> }` is the abstract shape, so structural-only callers (`score = None`) and LTM-run callers share one impl. libsimlin's `simlin_analyze_get_links(.., include_internal=false)` collapses the `get_links()` set through this; `include_internal=true` returns the raw graph. The `#[cfg(test)]` tests live in the sibling `src/ltm_finding_tests.rs` (mounted via `#[cfg(test)] #[path]`, split out for the per-file line cap). -- **`src/ltm_agg.rs`** - Aggregate-node enumeration for LTM. `enumerate_agg_nodes` (salsa-tracked) walks every variable's `Expr2` AST left-to-right depth-first and identifies each maximal array-reducer subexpression (`SUM`/`MEAN`/`MIN`/`MAX`/`STDDEV`/`RANK`/`SIZE`); AST-identical subexpressions (keyed by canonical printed equation text) dedupe to one `AggNode`. Two kinds: **synthetic** (`is_synthetic == true`, the reducer is a sub-expression of a larger equation -- a `$⁚ltm⁚agg⁚{n}` aux is minted) and **variable-backed** (`is_synthetic == false`, the reducer is the entire dt-equation of a scalar/A2A variable like `total_population = SUM(pop[*])` -- the variable itself is the agg; EXCEPT a whole-RHS reducer whose shape the variable-backed machinery cannot express -- a MAPPED iterated axis (GH #534: the name-based link-score path cannot remap, so its `Wildcard` partial would silently stub to 0) or NON-ALIGNED result dims (GH #764 / shape-expressiveness T4: a broadcast over extra owner dims `out[D1,D3] = SUM(matrix[D1,*])` or permuted axes `out[D2,D1] = SUM(cube[D1,D2,*])`, where a per-`(row, slot)` slot cannot name a complete owner element) -- which mints a synthetic agg instead (`variable_backed_shape_is_expressible`, the ONE minting condition), riding the two-half emitters + the GH #528 projection). Each `AggNode` carries a `sources: Vec` -- one entry per source variable, SORTED by canonical name and deduped (salsa cache-equality + emission-order determinism; T2 of the shape-expressiveness design), each with its OWN `read_slice: Vec` (one `AxisRead ∈ {Pinned(elem), Iterated{dim, source_dim}, Reduced{subset}}` per THAT source's axes -- which rows of it the reducer actually reads; a scalar feeder like `scale` in `SUM(pop[*] * scale)` carries an empty slice; `Iterated` carries the (target, source) canonical dim pair, equal for the literal case; `Reduced.subset` is `None` for the full extent or the proper-subdimension element subset for a `SUM(arr[*:Sub])` StarRange, GH #766, decided per axis by `classify_axis_access` -- the single per-axis classifier of the shape-expressiveness design) -- and a `result_dims` (the `Iterated` axes' TARGET dims, datamodel-cased -- empty for a whole-extent or pinned-slice reduce). Under the I1 acceptance (`accept_source_slices`, T5 of the shape-expressiveness design / GH #767) every arrayed CO-SOURCE (`Reduced`-bearing slice) carries the identical *canonical* slice (`AggNode::canonical_read_slice` -- the first `Reduced`-bearing source slice, falling back to the first non-empty for the degenerate no-co-source agg), while an ITERATED-DIM PROJECTION FEEDER -- a source whose slice is all-`Iterated` over exactly the canonical slice's iterated target dims, in order, unmapped (`frac[D1]` in `SUM(matrix[D1,*] * frac[D1])`, per-result-slot constant) -- is accepted with ITS OWN slice (`AggNode::source_is_projection_feeder` is the discriminator); per-source consumers (`emit_agg_routed_edges`, `emit_source_to_agg_link_scores`) read `AggNode::source_read_slice(from)`, and name-keyed consumers use `AggNode::reads_var`. A feeder's link-score half is the per-`(row, slot)` CHANGED-LAST equation (`ltm_augment::generate_iterated_feeder_to_agg_equation`, emitted via `link_scores::iterated_feeder_row_scores` from both the synthetic source-half and `try_cross_dimensional_link_scores`' variable-backed feeder branch): the reducer text pinned to the slot with only the feeder frozen -- the arrayed generalization of the GH #737 scalar-feeder convention, exactly complementary per slot to the co-source rows' changed-first numerators for a bilinear body; the co-source rows' changed-first body partial pins the mismatched-arity feeder dep BY DIM NAME (`pin_body_to_row`'s GH #767 extension) so `PREVIOUS(frac[d1·r])` is held frozen at the row instead of bailing to the delta-ratio fallback. `compute_read_slice` decides hoistability: a whole-extent reduce (`SUM(pop[*])` ⇒ all-`Reduced`), a sliced one (`SUM(pop[NYC,*])` ⇒ `[Pinned(nyc), Reduced]`, `SUM(matrix[D1,*])` over an A2A-`D1` body ⇒ `[Iterated(d1,d1), Reduced]` → an arrayed agg over `D1`), a mixed one (`SUM(matrix3d[D1,NYC,*])` over an A2A-`D1` body ⇒ `[Iterated(d1,d1), Pinned(nyc), Reduced]`), or a positionally-MAPPED sliced one (GH #534: `SUM(matrix[State,*])` over `matrix[Region,D2]` with a positional `State→Region` mapping ⇒ `[Iterated{state, region}, Reduced]`, `result_dims = [State]` -- the agg is arrayed over the TARGET's iterated dim and the three `Iterated`-axis consumers remap each source row to the slot of its positionally-corresponding target element via `iterated_axis_slot_elements`, the preimage inversion of `mapped_element_correspondence`, so the positional-only/element-map gate is inherited) is hoisted; the carve-outs are (a) a reducer over a *dynamic index* (`SUM(pop[idx,*])`, `idx` non-literal ⇒ not statically describable ⇒ not hoisted, reference stays on the conservative path -- `db::ltm_ir` reclassifies it as `DynamicIndex`), (b) an ELEMENT-mapped sliced reducer the correspondence declines (execution resolves positionally and ignores the map, GH #756; a POSITIONAL mapping is accepted in either declaration direction since GH #757) ⇒ `compute_read_slice` returns `None`, conservative -- (c) a multi-source slice combination outside the I1 acceptance -- co-sources with differing slices, one variable read with two different slices (I3b), or a no-`Reduced` source that is not the pure iterated projection (a Pinned-axis mix like `SUM(matrix[D1,*] * w[D1, c1])`, a dim-subset/permuted feeder, or any mapped Iterated axis in a feeder combination) -- `combined_read_slice`/`accept_source_slices` return `None`, (d) a StarRange naming a NON-subdimension of its axis (a mid-edit inconsistency; declined rather than silently widened to the full extent), and (e) `RANK` (GH #771: array-valued, so `reducer_is_hoistable` requires `reducer_collapses_to_scalar` and RANK references stay `Direct` -- a bare arg classifies `Bare`, scored by the GH #742 arrayed-capture path; loops through the rank ORDERING are a documented residual). A multi-source reducer whose arrayed args *agree* (`SUM(a[*] + b[*])`, `a`, `b` over the same dim) mints one agg with one `AggSource` per variable, each carrying the shared canonical slice. `AggNodesResult` exposes `aggs` (first-encounter order), `agg_for_key` (by canonical reducer text), and `aggs_in_var` (which aggs occur in a variable's equation, so the element-graph reroute can ask "which agg of `to` reads `from`?"). A variable-backed reduce with a NON-TRIVIAL statically-describable slice is gated by `variable_backed_reduce_agg` (GH #752, generalized by GH #765 / T3 of the shape-expressiveness design) -- the SAME gate `model_element_causal_edges`' dispatch, `build_element_level_loops`' per-circuit routing, and `try_cross_dimensional_link_scores`' row derivation consume, so edges, loop routing, and scores always cover the identical read rows (all three derive from `read_slice_rows`, invariant I4). Accepted: an ALIGNED partial reduce (`row_sum[D1] = SUM(matrix[D1,*])`, `result_dims` equal to the variable's own dims in order -- Pinned-mixed `outf[D1] = MEAN(cube[D1,x,*])` and subset `out[D1] = MEAN(matrix[D1,*:Sub])` slices included: the divisor is the true read count and unread rows get neither edges nor scores) gets read-slice element edges straight onto the variable's element nodes and per-circuit scalar loops whose both-subscripted `matrix[d1,d2]→row_sum[d1]` links resolve the per-`(row, slot)` scores; a scalar-result Pinned/subset slice on a SCALAR owner (`total = SUM(pop[nyc,*])`, `total = SUM(arr[*:Sub])`) routes the read rows into the bare `to` node with matching per-read-row scores; and the ARRAYED-owner scalar-result Pinned/subset BROADCAST slice (`share[Region] = SUM(pop[nyc,*])`, no `Iterated` axis -- GH #777) fans each read row across the FULL target element set (`emit_agg_routed_edges`' broadcast arm emits `pop[nyc,d2] → share[e]` for every `e`; `emit_broadcast_reduce_link_scores` emits the matching per-(read-row, full-target-element) scalar scores `pop[nyc,d2]→share[e]`, the section-3 `PerElement` rule applied to a variable-backed reducer owner), with loop circuits routed to the per-circuit scalar path via `is_broadcast_reduce_edge` -- the read rows are independent of `to`'s dims, so the RELATED-dim (`share[Region]`) and DISJOINT-dim (`share[D9]`) spellings emit identically. Declined: a pure full-extent variable-backed agg keeps the normal reference walker's edges (the true reads for that shape, inert skip). The GH #764 broadcast/permuted result shapes never reach this gate since T4 -- they mint synthetic aggs at enumeration -- so the gate's Iterated-arm alignment check is defense-in-depth. `scalar_feeder_of_variable_backed_agg` (GH #790) is the scalar sibling of `source_is_projection_feeder`: it composes `variable_backed_reduce_agg` with an empty-slice + genuine-`Reduced`-canonical check to recognize a SCALAR FEEDER of a whole-RHS variable-backed reduce (`scale` in `growth[D1] = SUM(matrix[D1,*] * scale)`), so `try_scalar_to_arrayed_link_scores` can route it to the single Bare A2A changed-last feeder score instead of the uncompilable per-target-element partials. Two predicates here answer what LOOKS like one question -- "is this reference inside a reducer?" -- and their answers are inverted on exactly `SIZE` and `RANK`; the inversion is the DEFINITION of the difference, not a disagreement (GH #982, assessed and left as two predicates). `reducer_collapses_to_scalar` is about the reducer's RESULT TYPE (does the subtree fit in a scalar slot?) and is read by the two freeze/capture gates plus the GH #779 bare-reducer-feeder decline: `SIZE` is a count so it collapses, `RANK` is array-valued so it does not. `builtin_routes_through_agg` is about LTM ROUTING (did `enumerate_agg_nodes` mint a node for this call?) and sets `db::ltm_ir::OccurrenceSite::in_reducer`: `SIZE` is `Constant` and is never hoisted, `RANK` gets an array-valued agg. Both read the ONE `reducer_kind_from_name` table, and `builtin_routes_through_agg` is the disjunction of the enumerator's own two minting branches (`reducer_is_hoistable` and `array_valued_rank_arg`) rather than a restatement of them. The `#[cfg(test)]` `REDUCER_DECISION_TABLE` pins all three derived predicates row by row over every arm of the kind table -- including the agreement gate's name-keyed twin of the routing predicate -- so no cell can move silently. `is_synthetic_agg_name` / `synthetic_agg_name` are the `$⁚ltm⁚agg⁚{n}` name helpers. Each node also carries `reducer: BuiltinFn` -- the reducer call the enumerator classified when it decided the hoist, of which `equation_text` is the printed rendering (GH #983). It is what makes the link-score and polarity emitters parse-free: `ltm_augment::classify_reducer_in_builtin` reads the kind/name/body off it and `ltm::CausalGraph::source_to_agg_polarity` analyses it directly, where both used to print `equation_text`, re-parse it, re-lower it against a freshly built scope and re-derive the classification -- per (agg, source) pair, with both fallible steps returning early and silently zeroing the agg's loop score. It is stored in `Expr2::strip_loc_and_bounds` form, which removes two of the three ways this field could make the salsa-cached `AggNodesResult` compare unequal to an identical rebuild: `Loc` (load-bearing -- two AST-identical occurrences differ in byte offsets, so raw storage would make the dedup winner observable and would stop `enumerate_agg_nodes` backdating across an offset-only edit) and `ArrayBounds` (a guard -- inert today, since `reconstruct_model_variables` lowers against an empty model scope and allocates no bound). Neither reader looks at either. It cannot remove the third -- dropping a `nan` literal would change what the equation means -- so that one is closed at the ROOT instead: `Expr2::Const` holds an `ast::Literal`, compared by BIT PATTERN, so a model whose hoisted reducer contains a `nan` literal backdates like any other (GH #987/#981; with a bare `f64` it never could, since `NaN != NaN`). Pinned by `a_nan_literal_in_a_reducer_does_not_defeat_agg_backdating`. The stored builtin is read only for SYNTHETIC aggs (both readers filter to those); the variable-backed arm's copy is unread today, kept so `AggNode` has one shape. `AggNode`/`AggNodesResult` derive `Eq` (reflexivity is a compile-checked property now that the literal is not a bare `f64`) and `Debug` only under `debug-derive`. -- **`src/ltm_augment.rs`** - Equation generators for LTM synthetic variables: `generate_link_score_equation_for_link` (ceteris-paribus link scores; takes `RefShape` and source dimension elements to drive per-shape PREVIOUS wrapping), `generate_loop_score_variables` (emits one `loop_score` per loop as a dimension-shaped `datamodel::Equation`: `Scalar` for scalar loops, `ApplyToAll` for dimensioned loops whose links resolve through Bare A2A names, and per-slot `Equation::Arrayed` for dimensioned loops backed by per-element circuits via `Loop.slot_links` -- GH #653; relative loop scores are computed post-simulation in `ltm_post.rs`), `build_partial_equation_shaped` (the `#[cfg(test)]` TEXT entry point for the ceteris-paribus wrap; arrayed-per-element-equation (`Ast::Arrayed`) targets get one partial per element assembled into an `Equation::Arrayed`). **Production never parses a target equation**: `wrap_changed_first_ast` takes an `Expr0` lowered straight from the target's `Expr2` by `patch::expr2_to_expr0` (which is what `expr2_to_string` prints, so the former print->reparse was a parse of our own output), and every per-occurrence decision -- access shape, the GH #526 other-dep verdict, the literal-element index guard, and the `PerElement` row pinning -- is a lookup into the `db::ltm_ir` occurrence IR by the structural child-index path the wrap tracks, which equals the occurrence's `SiteId` BY CONSTRUCTION now that both walk the same tree. A subscripted source reference the IR did NOT record -- reachable under a `LOOKUP` TABLE argument, which the walker skips as static data ("not a causal edge", which is right about ATTRIBUTION) -- still has to COMPILE, so the pin-only descent discharges it by NAME (`pin_dimension_name_indices`: an index naming one of the TARGET's iterated dimensions becomes the source element this target element reads on that axis, the same structural substitution `pin_bare_source_ref` performs for a bare `Var`). That is a lowering-completeness rule, not a second classifier: it consults no occurrence, infers no shape, and never makes the reference live-selectable -- it asks the SHARED row derivation `per_element_row_for_target` (hence `DimensionsContext::mapped_element_correspondence`) which element an axis reads, so the identity axis and a positionally-MAPPED one (`effect[State, old]` over an `effect[Region, Age]` source, either declaration direction) are one arm and it accepts exactly the mapped pairs `ltm_agg::classify_axis_access` accepts. An index the source's axis DECLARES as an element is resolved BEFORE that dimension-name reading, matching `compiler::subscript`'s `normalize_subscripts3` ("First check if it's a named dimension element (takes priority)") -- the two readings collide when a dimension declares an element whose name is also a dimension the target iterates, and there the element must win or the pin spells a row the simulation never reads. `classify_axis_access` still has the opposite order (GH #986: a precision gap today, since its dimension-first reading finds no mapping and declines). An index that already selects a FIXED element is left verbatim, needing no pin and reading nothing at the current step: a numeric literal, arithmetic over numeric literals (`index_expr_selects_a_fixed_element`, deliberately the narrowest sound predicate -- `compiler::invariance` is the engine's run-invariance derivation but consumes lowered `Expr` and answers a wider question), or an `@N` position index, which `compiler::context` resolves to a concrete element offset in scalar context (spelling `@N` out here would be a second implementation of position syntax). A 0-arity BUILTIN index is where that widening deliberately stops: `TIME` and `PI` are indistinguishable inside `Expr0::App` without a fourth copy of the builtin classification, so the arm stays conservatively loud. What the SHARED derivation declines -- an unmapped or element-mapped pair (GH #756), a transposition, a dimension the target does not iterate -- is declined LOUDLY (`WrapOutcome::missing_occurrence` -> warned skip) rather than emitted with its dimension-name subscript intact, which would not resolve in a scalar fragment. The whole two-verdict space (`Unspellable`, a compilability verdict loud everywhere; `RuntimeRead`, a ceteris-paribus verdict loud only in a bare table argument) is ENUMERATED cell by cell in `ltm_augment_pin_tests.rs`'s two verdict enumerations rather than sampled. There is ONE access-shape classifier family, on `Expr2`; the Expr0 mirror is test-support only and lives in `ltm_augment_wrap_test_support.rs` (kept because three wrap unit tests -- unparseable text, empty text, and the Fig. 2 Q4 `SUM(w[from]) + from` shape the engine REJECTS as a model -- cannot be db-backed fixtures), with `ltm_classifier_agreement_tests.rs` proving it matches the IR field for field (`SiteId` path, `shape`, `axes`, `in_reducer`) corpus-wide, `link_score_var_name` (synthetic name helper: Bare gets the canonical `{from}\u{2192}{to}` form, FixedIndex prepends `[elem]` to from; the obsolete per-shape `\u{205A}wildcard`/`\u{205A}dynamic` Wildcard/DynamicIndex suffixes were retired -- those shapes now collapse onto the Bare name, since *every statically-describable* inlined reducer (whole-extent or sliced) is hoisted into a `$⁚ltm⁚agg⁚{n}` node and only a `DynamicIndex` reference -- `arr[i+1]`, a range, or the not-hoistable dynamic-index reducer carve-out `SUM(pop[idx,*])` -- a whole-RHS variable-backed reducer's `Wildcard` argument, or a de-hoisted array-valued reducer's `Wildcard` arg (`RANK(pop[*], 1)`, GH #771) reach this function), `quote_ident` (identifier quoting for equations). Array support: `classify_reducer` (walks target Expr2 AST to identify reducing builtins -- Linear for SUM/MEAN, Nonlinear for MIN/MAX/STDDEV/RANK, Constant for SIZE -- a thin reader of `ltm_agg::reducer_kind`; it also hands back the reducer's array-argument AST as `ClassifiedReducer::body`, lowered from `Expr2` rather than printed, so the body-aware row partials never re-parse it), `generate_element_to_scalar_equation` (per-element link score equations for arrayed-to-scalar edges, used by both the variable-backed-reducer path and the `source[d] → $⁚ltm⁚agg⁚{n}` half) which dispatches on `ReducerKind` -- `generate_linear_partial` (SUM/MEAN algebraic shortcut), `generate_nonlinear_partial` (MIN/MAX nested binary calls; STDDEV the unrolled population-variance `sqrt` ceteris-paribus partial -- divisor `N`, matching `vm.rs::Opcode::ArrayStddev`, with the mean string-inlined; RANK the documented delta-ratio stand-in pinned by `test_generate_rank_keeps_delta_ratio` -- an order statistic, non-differentiable and unreachable via a real model RHS), `generate_scalar_to_element_equation` (per-element link score for the `$⁚ltm⁚agg⁚{n} → target[e]` half; takes a `source_ref_override: Option<&str>` so a multi-slot arrayed agg's `Δsource` denominator carries the projected `agg[]` subscript instead of the bare agg name, which wouldn't compile as a scalar), `substitute_reducers_in_expr0` (textually replaces a recognized reducer subexpression in an `Expr0` with its agg name, for the `$⁚ltm⁚agg⁚{n} → target` link score), `resolve_link_score_name_for_loop` (picks the Bare-or-FixedIndex link-score name a loop-score reference should target). Module link score formulas (black-box delta-ratio and composite-ref) are inlined directly into `module_link_score_equation` in `db.rs` (called by the per-shape `link_score_equation_text_shaped`). The partial-equation builders (`build_partial_equation_shaped`/`_with_live_ref`, `subscript_idents_at_element`) and every link-score equation generator return `Result<_, PartialEquationError>`: a parse failure (genuine `Err`, or an empty `Ok(None)` equation) has no AST to PREVIOUS-wrap, so emitting the unwrapped input would silently produce a non-ceteris-paribus "partial" identical to the full equation (link score magnitude constant |Δz/Δz| = 1) -- a hidden attribution error that compiles cleanly (GH #311). The db-bearing callers (`link_score_equation_text_shaped` and the `src/db/ltm/link_scores.rs` emitters) convert the error into a `Warning` (`emit_ltm_partial_equation_warning`, naming the variable + offending equation text) and skip the variable -- distinct from `model_ltm_fragment_diagnostics`, which only catches *compile* failures. The failure is effectively unreachable in production (the text is always a `print_eqn` re-print; an empty equation is rejected as an `EmptyEquation` Error upstream), so this is defense-in-depth. The `#[cfg(test)]` tests live in the sibling `src/ltm_augment_tests.rs` (split out for the per-file line cap). Four more siblings are `#[path]`-mounted into `ltm_augment` purely for that cap, so every caller still names their items `crate::ltm_augment::*`: **`ltm_augment_occurrence.rs`** (the wrap's read side of the occurrence IR -- `SlotOccurrences` groups a target's stream by slot ONCE and is the only way to obtain an `OccurrenceLookup`, so the borrow forces callers to hoist it out of their per-element loop), **`ltm_augment_post_transform.rs`** (the concrete-form lowerings: the agg-name substitution, and the `PerElement` row pinning the wrap calls AS IT DESCENDS -- the wrap is the only place that knows both the occurrence and whether it is about to FREEZE the reference, which is what picks the bare row for the live occurrence over the qualified row for every other one), **`ltm_augment_with_lookup.rs`** (the GH #910 implicit-WITH-LOOKUP rules), and **`ltm_augment_wrap_test_support.rs`** (the `#[cfg(test)]` occurrence reconstruction + the Expr0 classifier mirror described above). +- **`src/ltm_agg.rs`** - Aggregate-node enumeration for LTM. `enumerate_agg_nodes` (salsa-tracked) walks every variable's `Expr2` AST left-to-right depth-first and identifies each maximal array-reducer subexpression (`SUM`/`MEAN`/`MIN`/`MAX`/`STDDEV`/`RANK`/`SIZE`); AST-identical subexpressions (keyed by canonical printed equation text) dedupe to one `AggNode`. Two kinds: **synthetic** (`is_synthetic == true`, the reducer is a sub-expression of a larger equation -- a `$⁚ltm⁚agg⁚{n}` aux is minted) and **variable-backed** (`is_synthetic == false`, the reducer is the entire dt-equation of a scalar/A2A variable like `total_population = SUM(pop[*])` -- the variable itself is the agg; EXCEPT a whole-RHS reducer whose shape the variable-backed machinery cannot express -- a MAPPED iterated axis (GH #534: the name-based link-score path cannot remap, so its `Wildcard` partial would silently stub to 0) or NON-ALIGNED result dims (GH #764 / shape-expressiveness T4: a broadcast over extra owner dims `out[D1,D3] = SUM(matrix[D1,*])` or permuted axes `out[D2,D1] = SUM(cube[D1,D2,*])`, where a per-`(row, slot)` slot cannot name a complete owner element) -- which mints a synthetic agg instead (`variable_backed_shape_is_expressible`, the ONE minting condition), riding the two-half emitters + the GH #528 projection). Each `AggNode` carries a `sources: Vec` -- one entry per source variable, SORTED by canonical name and deduped (salsa cache-equality + emission-order determinism; T2 of the shape-expressiveness design), each with its OWN `read_slice: Vec` (one `AxisRead ∈ {Pinned(elem), Iterated{dim, source_dim}, Reduced{subset}}` per THAT source's axes -- which rows of it the reducer actually reads; a scalar feeder like `scale` in `SUM(pop[*] * scale)` carries an empty slice; `Iterated` carries the (target, source) canonical dim pair, equal for the literal case; `Reduced.subset` is `None` for the full extent or the proper-subdimension element subset for a `SUM(arr[*:Sub])` StarRange, GH #766, decided per axis by `classify_axis_access` -- the single per-axis classifier of the shape-expressiveness design) -- and a `result_dims` (the `Iterated` axes' TARGET dims, datamodel-cased -- empty for a whole-extent or pinned-slice reduce). Under the I1 acceptance (`accept_source_slices`, T5 of the shape-expressiveness design / GH #767) every arrayed CO-SOURCE (`Reduced`-bearing slice) carries the identical *canonical* slice (`AggNode::canonical_read_slice` -- the first `Reduced`-bearing source slice, falling back to the first non-empty for the degenerate no-co-source agg), while an ITERATED-DIM PROJECTION FEEDER -- a source whose slice is all-`Iterated` over exactly the canonical slice's iterated target dims, in order, unmapped (`frac[D1]` in `SUM(matrix[D1,*] * frac[D1])`, per-result-slot constant) -- is accepted with ITS OWN slice (`AggNode::source_is_projection_feeder` is the discriminator); per-source consumers (`emit_agg_routed_edges`, `emit_source_to_agg_link_scores`) read `AggNode::source_read_slice(from)`, and name-keyed consumers use `AggNode::reads_var`. A feeder's link-score half is the per-`(row, slot)` CHANGED-LAST equation (`ltm_augment::generate_iterated_feeder_to_agg_equation`, emitted via `link_scores::iterated_feeder_row_scores` from both the synthetic source-half and `try_cross_dimensional_link_scores`' variable-backed feeder branch): the reducer text pinned to the slot with only the feeder frozen -- the arrayed generalization of the GH #737 scalar-feeder convention, exactly complementary per slot to the co-source rows' changed-first numerators for a bilinear body; the co-source rows' changed-first body partial pins the mismatched-arity feeder dep BY DIM NAME (`pin_body_to_row`'s GH #767 extension) so `PREVIOUS(frac[d1·r])` is held frozen at the row instead of bailing to the delta-ratio fallback. `compute_read_slice` decides hoistability: a whole-extent reduce (`SUM(pop[*])` ⇒ all-`Reduced`), a sliced one (`SUM(pop[NYC,*])` ⇒ `[Pinned(nyc), Reduced]`, `SUM(matrix[D1,*])` over an A2A-`D1` body ⇒ `[Iterated(d1,d1), Reduced]` → an arrayed agg over `D1`), a mixed one (`SUM(matrix3d[D1,NYC,*])` over an A2A-`D1` body ⇒ `[Iterated(d1,d1), Pinned(nyc), Reduced]`), or a positionally-MAPPED sliced one (GH #534: `SUM(matrix[State,*])` over `matrix[Region,D2]` with a positional `State→Region` mapping ⇒ `[Iterated{state, region}, Reduced]`, `result_dims = [State]` -- the agg is arrayed over the TARGET's iterated dim and the three `Iterated`-axis consumers remap each source row to the slot of its positionally-corresponding target element via `iterated_axis_slot_elements`, the preimage inversion of `mapped_element_correspondence`, so the positional-only/element-map gate is inherited) is hoisted; the carve-outs are (a) a reducer over a *dynamic index* (`SUM(pop[idx,*])`, `idx` non-literal ⇒ not statically describable ⇒ not hoisted, reference stays on the conservative path -- `db::ltm_ir` reclassifies it as `DynamicIndex`), (b) an ELEMENT-mapped sliced reducer the correspondence declines (execution resolves positionally and ignores the map, GH #756; a POSITIONAL mapping is accepted in either declaration direction since GH #757) ⇒ `compute_read_slice` returns `None`, conservative -- (c) a multi-source slice combination outside the I1 acceptance -- co-sources with differing slices, one variable read with two different slices (I3b), or a no-`Reduced` source that is not the pure iterated projection (a Pinned-axis mix like `SUM(matrix[D1,*] * w[D1, c1])`, a dim-subset/permuted feeder, or any mapped Iterated axis in a feeder combination) -- `combined_read_slice`/`accept_source_slices` return `None`, (d) a StarRange naming a NON-subdimension of its axis (a mid-edit inconsistency; declined rather than silently widened to the full extent), and (e) `RANK` (GH #771: array-valued, so `reducer_is_hoistable` requires `reducer_collapses_to_scalar` and RANK references stay `Direct` -- a bare arg classifies `Bare`, scored by the GH #742 arrayed-capture path; loops through the rank ORDERING are a documented residual). A multi-source reducer whose arrayed args *agree* (`SUM(a[*] + b[*])`, `a`, `b` over the same dim) mints one agg with one `AggSource` per variable, each carrying the shared canonical slice. `AggNodesResult` exposes `aggs` (first-encounter order), `agg_for_key` (by canonical reducer text), and `aggs_in_var` (which aggs occur in a variable's equation, so the element-graph reroute can ask "which agg of `to` reads `from`?"). A variable-backed reduce with a NON-TRIVIAL statically-describable slice is gated by `variable_backed_reduce_agg` (GH #752, generalized by GH #765 / T3 of the shape-expressiveness design) -- the SAME gate `model_element_causal_edges`' dispatch, `build_element_level_loops`' per-circuit routing, and `try_cross_dimensional_link_scores`' row derivation consume, so edges, loop routing, and scores always cover the identical read rows (all three derive from `read_slice_rows`, invariant I4). Accepted: an ALIGNED partial reduce (`row_sum[D1] = SUM(matrix[D1,*])`, `result_dims` equal to the variable's own dims in order -- Pinned-mixed `outf[D1] = MEAN(cube[D1,x,*])` and subset `out[D1] = MEAN(matrix[D1,*:Sub])` slices included: the divisor is the true read count and unread rows get neither edges nor scores) gets read-slice element edges straight onto the variable's element nodes and per-circuit scalar loops whose both-subscripted `matrix[d1,d2]→row_sum[d1]` links resolve the per-`(row, slot)` scores; a scalar-result Pinned/subset slice on a SCALAR owner (`total = SUM(pop[nyc,*])`, `total = SUM(arr[*:Sub])`) routes the read rows into the bare `to` node with matching per-read-row scores; and the ARRAYED-owner scalar-result Pinned/subset BROADCAST slice (`share[Region] = SUM(pop[nyc,*])`, no `Iterated` axis -- GH #777) fans each read row across the FULL target element set (`emit_agg_routed_edges`' broadcast arm emits `pop[nyc,d2] → share[e]` for every `e`; `emit_broadcast_reduce_link_scores` emits the matching per-(read-row, full-target-element) scalar scores `pop[nyc,d2]→share[e]`, the section-3 `PerElement` rule applied to a variable-backed reducer owner), with loop circuits routed to the per-circuit scalar path via `is_broadcast_reduce_edge` -- the read rows are independent of `to`'s dims, so the RELATED-dim (`share[Region]`) and DISJOINT-dim (`share[D9]`) spellings emit identically. Declined: a pure full-extent variable-backed agg keeps the normal reference walker's edges (the true reads for that shape, inert skip). The GH #764 broadcast/permuted result shapes never reach this gate since T4 -- they mint synthetic aggs at enumeration -- so the gate's Iterated-arm alignment check is defense-in-depth. `scalar_feeder_of_variable_backed_agg` (GH #790) is the scalar sibling of `source_is_projection_feeder`: it composes `variable_backed_reduce_agg` with an empty-slice + genuine-`Reduced`-canonical check to recognize a SCALAR FEEDER of a whole-RHS variable-backed reduce (`scale` in `growth[D1] = SUM(matrix[D1,*] * scale)`), so `try_scalar_to_arrayed_link_scores` can route it to the single Bare A2A changed-last feeder score instead of the uncompilable per-target-element partials. Two predicates here answer what LOOKS like one question -- "is this reference inside a reducer?" -- and their answers are inverted on exactly `SIZE` and `RANK`; the inversion is the DEFINITION of the difference, not a disagreement (GH #982, assessed and left as two predicates). `reducer_collapses_to_scalar` is about the reducer's RESULT TYPE (does the subtree fit in a scalar slot?) and is read by the two freeze/capture gates plus the GH #779 bare-reducer-feeder decline: `SIZE` is a count so it collapses, `RANK` is array-valued so it does not. `builtin_routes_through_agg` is about LTM ROUTING (did `enumerate_agg_nodes` mint a node for this call?) and sets `db::ltm_ir::OccurrenceSite::in_reducer`: `SIZE` is `Constant` and is never hoisted, `RANK` gets an array-valued agg. Both read the ONE `reducer_kind_from_name` table, and `builtin_routes_through_agg` is the disjunction of the enumerator's own two minting branches (`reducer_is_hoistable` and `array_valued_rank_arg`) rather than a restatement of them. The `#[cfg(test)]` `REDUCER_DECISION_TABLE` pins all three derived predicates row by row over every arm of the kind table -- including the agreement gate's name-keyed twin of the routing predicate -- so no cell can move silently. `is_synthetic_agg_name` / `synthetic_agg_name` are the `$⁚ltm⁚agg⁚{n}` name helpers. `classify_axis_access` resolves a bare-identifier index through the shared `dimensions::resolve_axis_index_name` (element-first, GH #986), so it and `ltm_augment_post_transform::pin_dimension_name_indices` cannot disagree about which row a colliding name selects. The `#[cfg(test)]` tests live in the sibling `src/ltm_agg_tests.rs` (split out for the per-file line cap). Each node also carries `reducer: BuiltinFn` -- the reducer call the enumerator classified when it decided the hoist, of which `equation_text` is the printed rendering (GH #983). It is what makes the link-score and polarity emitters parse-free: `ltm_augment::classify_reducer_in_builtin` reads the kind/name/body off it and `ltm::CausalGraph::source_to_agg_polarity` analyses it directly, where both used to print `equation_text`, re-parse it, re-lower it against a freshly built scope and re-derive the classification -- per (agg, source) pair, with both fallible steps returning early and silently zeroing the agg's loop score. It is stored in `Expr2::strip_loc_and_bounds` form, which removes two of the three ways this field could make the salsa-cached `AggNodesResult` compare unequal to an identical rebuild: `Loc` (load-bearing -- two AST-identical occurrences differ in byte offsets, so raw storage would make the dedup winner observable and would stop `enumerate_agg_nodes` backdating across an offset-only edit) and `ArrayBounds` (a guard -- inert today, since `reconstruct_model_variables` lowers against an empty model scope and allocates no bound). Neither reader looks at either. It cannot remove the third -- dropping a `nan` literal would change what the equation means -- so that one is closed at the ROOT instead: `Expr2::Const` holds an `ast::Literal`, compared by BIT PATTERN, so a model whose hoisted reducer contains a `nan` literal backdates like any other (GH #987/#981; with a bare `f64` it never could, since `NaN != NaN`). Pinned by `a_nan_literal_in_a_reducer_does_not_defeat_agg_backdating`. The stored builtin is read only for SYNTHETIC aggs (both readers filter to those); the variable-backed arm's copy is unread today, kept so `AggNode` has one shape. `AggNode`/`AggNodesResult` derive `Eq` (reflexivity is a compile-checked property now that the literal is not a bare `f64`) and `Debug` only under `debug-derive`. +- **`src/ltm_augment.rs`** - Equation generators for LTM synthetic variables: `generate_link_score_equation_for_link` (ceteris-paribus link scores; takes `RefShape` and source dimension elements to drive per-shape PREVIOUS wrapping), `generate_loop_score_variables` (emits one `loop_score` per loop as a dimension-shaped `datamodel::Equation`: `Scalar` for scalar loops, `ApplyToAll` for dimensioned loops whose links resolve through Bare A2A names, and per-slot `Equation::Arrayed` for dimensioned loops backed by per-element circuits via `Loop.slot_links` -- GH #653; relative loop scores are computed post-simulation in `ltm_post.rs`), `build_partial_equation_shaped` (the `#[cfg(test)]` TEXT entry point for the ceteris-paribus wrap; arrayed-per-element-equation (`Ast::Arrayed`) targets get one partial per element assembled into an `Equation::Arrayed`). **Production never parses a target equation**: `wrap_changed_first_ast` takes an `Expr0` lowered straight from the target's `Expr2` by `patch::expr2_to_expr0` (which is what `expr2_to_string` prints, so the former print->reparse was a parse of our own output), and every per-occurrence decision -- access shape, the GH #526 other-dep verdict, the literal-element index guard, and the `PerElement` row pinning -- is a lookup into the `db::ltm_ir` occurrence IR by the structural child-index path the wrap tracks, which equals the occurrence's `SiteId` BY CONSTRUCTION now that both walk the same tree. A subscripted source reference the IR did NOT record -- reachable under a `LOOKUP` TABLE argument, which the walker skips as static data ("not a causal edge", which is right about ATTRIBUTION) -- still has to COMPILE, so the pin-only descent discharges it by NAME (`pin_dimension_name_indices`: an index naming one of the TARGET's iterated dimensions becomes the source element this target element reads on that axis, the same structural substitution `pin_bare_source_ref` performs for a bare `Var`). That is a lowering-completeness rule, not a second classifier: it consults no occurrence, infers no shape, and never makes the reference live-selectable -- it asks the SHARED row derivation `per_element_row_for_target` (hence `DimensionsContext::mapped_element_correspondence`) which element an axis reads, so the identity axis and a positionally-MAPPED one (`effect[State, old]` over an `effect[Region, Age]` source, either declaration direction) are one arm and it accepts exactly the mapped pairs `ltm_agg::classify_axis_access` accepts. An index the source's axis DECLARES as an element is resolved BEFORE that dimension-name reading, and that precedence is not the pin's own: it is the shared `dimensions::resolve_axis_index_name`, which `ltm_agg::classify_axis_access` reads too (GH #986 closed the divergence -- the classifier had the opposite order, so a mapped collision described the axis as `Iterated` over a dimension the compiler never iterates there). Element-first is what `compiler::subscript`'s `normalize_subscripts3` does ("First check if it's a named dimension element (takes priority)"), and the simulation is the authority: the two readings collide when a dimension declares an element whose name is also a dimension name (`Bucket = [old, region]` beside a `Region` dimension), and a describer that breaks the tie the other way names rows the simulation never reads (`a_colliding_index_name_reads_the_axis_element_in_the_simulation` is the numeric oracle; the XMILE spec's footnote [9] settles the adjacent VARIABLE-vs-element pair outright and sections 2.1/3.7.1 argue this pair from the namespace rule -- `resolve_axis_index_name`'s rustdoc says which is which). Everything that is NOT a bare identifier is left verbatim, needing no pin: a numeric literal, arithmetic over literals, an `@N` position (which `compiler::context` resolves to a concrete element offset in scalar context -- spelling it out here would be a second implementation of position syntax), and a compound expression selecting the element at RUNTIME. That last one used to be a conditional REFUSAL, and deleting it is GH #984: the wrap now freezes a `LOOKUP` table argument's index reads itself (`freeze_lookup_table_indices`), so a runtime index arrives here already lagged and the rule keeps it. That freeze WIDENS its own descent's dep set with the index idents, and without that it would not fire at all -- `variable::classify_dependencies`' `BuiltinContents::LookupTable` arm records the table's ident and never walks the table expression, so an index variable referenced only there is not a dependency and the wrap's `other_deps` freeze could never reach it. The widened set is scoped to that argument's indices, and the element / dimension-name guards run before the dep check, so it cannot make a selector wrap. (Leaving the dep set itself alone is deliberate: an index dropped from a variable's dependencies is a runlist-ordering question, not an LTM one.) What the SHARED derivation declines -- an unmapped or element-mapped pair (GH #756), a transposition, a dimension the target does not iterate -- is declined LOUDLY (`WrapOutcome::missing_occurrence` -> warned skip) rather than emitted with its dimension-name subscript intact, which would not resolve in a scalar fragment. The verdict space (`Pinned`, `Keep`, and the one loud `Unspellable`) is ENUMERATED cell by cell in `ltm_augment_pin_tests.rs`'s three verdict enumerations rather than sampled. There is ONE access-shape classifier family, on `Expr2`; the Expr0 mirror is test-support only and lives in `ltm_augment_wrap_test_support.rs` (kept because three wrap unit tests -- unparseable text, empty text, and the Fig. 2 Q4 `SUM(w[from]) + from` shape the engine REJECTS as a model -- cannot be db-backed fixtures), with `ltm_classifier_agreement_tests.rs` proving it matches the IR field for field (`SiteId` path, `shape`, `axes`, `in_reducer`) corpus-wide, `link_score_var_name` (synthetic name helper: Bare gets the canonical `{from}\u{2192}{to}` form, FixedIndex prepends `[elem]` to from; the obsolete per-shape `\u{205A}wildcard`/`\u{205A}dynamic` Wildcard/DynamicIndex suffixes were retired -- those shapes now collapse onto the Bare name, since *every statically-describable* inlined reducer (whole-extent or sliced) is hoisted into a `$⁚ltm⁚agg⁚{n}` node and only a `DynamicIndex` reference -- `arr[i+1]`, a range, or the not-hoistable dynamic-index reducer carve-out `SUM(pop[idx,*])` -- a whole-RHS variable-backed reducer's `Wildcard` argument, or a de-hoisted array-valued reducer's `Wildcard` arg (`RANK(pop[*], 1)`, GH #771) reach this function), `quote_ident` (identifier quoting for equations). Array support: `classify_reducer` (walks target Expr2 AST to identify reducing builtins -- Linear for SUM/MEAN, Nonlinear for MIN/MAX/STDDEV/RANK, Constant for SIZE -- a thin reader of `ltm_agg::reducer_kind`; it also hands back the reducer's array-argument AST as `ClassifiedReducer::body`, lowered from `Expr2` rather than printed, so the body-aware row partials never re-parse it), `generate_element_to_scalar_equation` (per-element link score equations for arrayed-to-scalar edges, used by both the variable-backed-reducer path and the `source[d] → $⁚ltm⁚agg⁚{n}` half) which dispatches on `ReducerKind` -- `generate_linear_partial` (SUM/MEAN algebraic shortcut), `generate_nonlinear_partial` (MIN/MAX nested binary calls; STDDEV the unrolled population-variance `sqrt` ceteris-paribus partial -- divisor `N`, matching `vm.rs::Opcode::ArrayStddev`, with the mean string-inlined; RANK the documented delta-ratio stand-in pinned by `test_generate_rank_keeps_delta_ratio` -- an order statistic, non-differentiable and unreachable via a real model RHS), `generate_scalar_to_element_equation` (per-element link score for the `$⁚ltm⁚agg⁚{n} → target[e]` half; takes a `source_ref_override: Option<&str>` so a multi-slot arrayed agg's `Δsource` denominator carries the projected `agg[]` subscript instead of the bare agg name, which wouldn't compile as a scalar), `substitute_reducers_in_expr0` (textually replaces a recognized reducer subexpression in an `Expr0` with its agg name, for the `$⁚ltm⁚agg⁚{n} → target` link score), `resolve_link_score_name_for_loop` (picks the Bare-or-FixedIndex link-score name a loop-score reference should target). Module link score formulas (black-box delta-ratio and composite-ref) are inlined directly into `module_link_score_equation` in `db.rs` (called by the per-shape `link_score_equation_text_shaped`). `subscript_idents_at_element` pins a target's arrayed deps for a per-element scalar partial, and it pins each one over the dimensions THAT DEP declares rather than over the target's element tuple (GH #974): a bare arrayed reference in an apply-to-all body reads its own axes' coordinates matched by dimension NAME, so a subset-dims dep (`w[Age]` under `growth[Region,Age]`) got an over-arity subscript whose fragment failed to compile, and a REORDERED one (`w[Age,Region]`) got a subscript that compiled and silently read the transposed element. The projection is `post_transform::dep_element_pins`/`dep_row_for_target`, reused by `pin_bare_source_ref` for a bare reference to the LIVE SOURCE (which is why a positionally-MAPPED bare source reference resolves through `mapped_element_correspondence` instead of being left bare and frozen into an uncompilable multi-slot `PREVIOUS`). The partial-equation builders (`build_partial_equation_shaped`/`_with_live_ref`, `subscript_idents_at_element`) and every link-score equation generator return `Result<_, PartialEquationError>`: a parse failure (genuine `Err`, or an empty `Ok(None)` equation) has no AST to PREVIOUS-wrap, so emitting the unwrapped input would silently produce a non-ceteris-paribus "partial" identical to the full equation (link score magnitude constant |Δz/Δz| = 1) -- a hidden attribution error that compiles cleanly (GH #311). The db-bearing callers (`link_score_equation_text_shaped` and the `src/db/ltm/link_scores.rs` emitters) convert the error into a `Warning` (`emit_ltm_partial_equation_warning`, naming the variable + offending equation text) and skip the variable -- distinct from `model_ltm_fragment_diagnostics`, which only catches *compile* failures. The failure is effectively unreachable in production (the text is always a `print_eqn` re-print; an empty equation is rejected as an `EmptyEquation` Error upstream), so this is defense-in-depth. The `#[cfg(test)]` tests live in the sibling `src/ltm_augment_tests.rs` (split out for the per-file line cap). Four more siblings are `#[path]`-mounted into `ltm_augment` purely for that cap, so every caller still names their items `crate::ltm_augment::*`: **`ltm_augment_occurrence.rs`** (the wrap's read side of the occurrence IR -- `SlotOccurrences` groups a target's stream by slot ONCE and is the only way to obtain an `OccurrenceLookup`, so the borrow forces callers to hoist it out of their per-element loop), **`ltm_augment_post_transform.rs`** (the concrete-form lowerings: the agg-name substitution, and the `PerElement` row pinning the wrap calls AS IT DESCENDS -- the wrap is the only place that knows both the occurrence and whether it is about to FREEZE the reference, which is what picks the bare row for the live occurrence over the qualified row for every other one), **`ltm_augment_with_lookup.rs`** (the GH #910 implicit-WITH-LOOKUP rules), and **`ltm_augment_wrap_test_support.rs`** (the `#[cfg(test)]` occurrence reconstruction + the Expr0 classifier mirror described above). - **`src/ltm_augment_with_lookup.rs`** - The implicit-WITH-LOOKUP rules for LTM (GH #910), re-exported from `ltm_augment` so every `crate::ltm_augment::*` path is unchanged. A `v = WITH LOOKUP(input, table)` variable is lowered by the compiler to `LOOKUP(v, input)` (`compiler::apply_implicit_with_lookup`), so a link-score partial that RE-EVALUATES such a target's equation is in gf-INPUT units while the guard form's `PREVIOUS(target)` anchor is in gf-OUTPUT units. `is_implicit_with_lookup` carries the coverage doc: every partial is either a **full re-evaluation** (class 1 -- must be wrapped in the gf application) or a **delta-ratio stand-in** (class 2 -- the RANK arm and the nested-arithmetic arm, already in output units, must NEVER be wrapped or a gf output is fed back through the gf). `WithLookupSlotRefs` resolves the target's table reference ONCE per target (`NoGf` / `Shared` / `PerElement`), so a per-element-gf target costs one row-major `SubscriptIterator` walk rather than one per element; an arrayed target's shared table is pinned as `to[1,...]` because a bare arrayed reference resolves each iterated element's own table offset, which the VM reads as NaN past `table_count`. `compose_with_lookup_polarity` (in `src/ltm/polarity.rs`) is the polarity twin, mirroring `apply_implicit_with_lookup`'s placeholder-Positive and zero-point-table rules. The polarity tests live in `src/ltm/with_lookup_tests.rs`, a child module of `ltm::tests`. - **`src/ltm_post.rs`** - Post-simulation relative loop *and link* score computation. `compute_rel_loop_scores(results, loop_partitions)` normalizes each loop's `loop_score` series against the sum of absolute scores within its cycle partition, using SAFEDIV-0 semantics (zero denominator -> zero result). Called after simulation rather than emitted as synthetic equations to avoid O(P^2) equation-text growth on models with dense partitions. `loop_partitions` is an `IndexMap` (re-exported `engine::indexmap`). `compute_rel_loop_scores*` walk its **emission** order rather than re-sorting the loop ids: the partition-sum denominator accumulates `|loop_score|` in that order, and emission order keeps the IEEE-754 (non-associative) sum bit-for-bit identical to the pre-#461 compile-time emitter (GH #468). Emission order is itself deterministic across salsa cache invalidations / processes because `assign_loop_ids` orders loops by a content-derived key (`ltm::graph::loop_id_sort_key`), so it never flaps even though `IndexMap`'s `PartialEq` (salsa cache equality) is order-insensitive. `compute_rel_link_scores(links, step_count)` is the link-level analogue (GH #652): raw link scores divide by the change in the *target*, so they are not comparable across targets and ranking by raw magnitude surfaces numerically-degenerate links (near-constant targets blow up the score). It groups the input `RelLinkInput { to, score }` links by their `to` target and normalizes each link's score by the per-target, per-timestep sum of `|score|` over all that target's *scored* inputs -- a **signed** value in `[-1, 1]` (sign kept like `compute_rel_loop_scores`), with the same `denom_summand` NaN-exclusion / Inf-retention / SAFEDIV-0 semantics. Denominator scope is the scored inputs only: complete in discovery mode (every causal edge scored) but covering just the in-loop subset in exhaustive mode (documented caveat on the fn). libsimlin's `analyze_links_core` calls it over the final (post-synthetic-collapse) link set so the per-target denominator matches the links the caller receives. - **LTM open work**: known LTM bugs and improvements are tracked on GitHub under the `ltm` label; issue #488 is the pinned epic that organises them by area (core algorithm, discovery/post-sim, augmentation, module/array umbrellas). Each open `ltm`-labelled issue carries file:line references and a suggested fix, so a new session can pick a bite-sized piece without re-investigating the subsystem. diff --git a/src/simlin-engine/src/db/ltm/link_scores.rs b/src/simlin-engine/src/db/ltm/link_scores.rs index f7340a747..20e0e0822 100644 --- a/src/simlin-engine/src/db/ltm/link_scores.rs +++ b/src/simlin-engine/src/db/ltm/link_scores.rs @@ -22,12 +22,77 @@ use crate::db::{ variable_dimensions, }; +use crate::ltm_augment::DepElementPin; + use super::compile::{ShapedLinkScore, link_score_equation_text_shaped}; use super::loops::{ ReadSliceRow, ReadSliceRowParts, cartesian_subscripts, read_slice_row_parts, read_slice_rows, }; use super::parse::{ltm_equation_dimensions, retarget_ltm_equation_dims}; +/// The arrayed deps of one target equation that a per-element link-score +/// partial can element-pin, each with its DECLARED dimensions. +/// +/// The declared dimensions are what the pin is projected over (GH #974), so +/// they are resolved once per target equation here and reused for every target +/// element by [`crate::ltm_augment::dep_element_pins`]. A dep is a candidate +/// when it is a non-module arrayed model variable and `keep` admits it (the +/// per-element emitter excludes the live source, whose pinning is the wrap's +/// job); whether it can actually be pinned AT a given element is the +/// projection's answer, not this filter's. +fn pinnable_arrayed_deps( + db: &dyn Db, + source_vars: &HashMap, + project: SourceProject, + deps: &HashSet>, + keep: impl Fn(&Ident) -> bool, +) -> Vec<(Ident, Vec)> { + let mut pinnable: Vec<(Ident, Vec)> = deps + .iter() + .filter(|d| keep(d)) + .filter_map(|d| { + let sv = source_vars.get(d.as_str())?; + if sv.kind(db) == SourceVariableKind::Module { + return None; + } + let dims = variable_dimensions(db, *sv, project); + (!dims.is_empty()).then(|| (d.clone(), dims.clone())) + }) + .collect(); + // `deps` is a `HashSet`, and the pin table it feeds is consumed as a map, + // so ordering is unobservable in the emitted text -- sorted anyway so a + // debug dump of this vector is reproducible run to run. + pinnable.sort_by(|a, b| a.0.as_str().cmp(b.0.as_str())); + pinnable +} + +/// The projection data one target element supplies: target dimension +/// (canonical) -> (that element's coordinate on the dimension, its index within +/// the dimension's element list). +/// +/// `element` is a comma-joined BARE element tuple over `to_dims`, and +/// `dim_element_lists` is `to_dims`' element lists (already computed by every +/// caller for `cartesian_subscripts`). A part that names no element of its +/// dimension leaves that dimension out of the map, which the row derivations +/// read as "not projectable" -- the conservative direction. +fn target_elem_by_dim_for( + to_dims: &[crate::dimensions::Dimension], + dim_element_lists: &[Vec], + element: &str, +) -> HashMap { + let parts: Vec<&str> = element.split(',').collect(); + if parts.len() != to_dims.len() { + return HashMap::new(); + } + let mut by_dim = HashMap::with_capacity(to_dims.len()); + for ((dim, elems), part) in to_dims.iter().zip(dim_element_lists).zip(&parts) { + if let Some(idx) = elems.iter().position(|e| e == part) { + by_dim.insert(dim.name().to_string(), ((*part).to_string(), idx)); + } + } + by_dim +} + /// The occurrence-stream slot each target element maps to when the /// ceteris-paribus wrap walks one slot's expression at a time. /// @@ -1202,33 +1267,20 @@ pub(super) fn try_scalar_to_arrayed_link_scores( }; // Which target deps must be pinned to the element in the per-element - // scalar equation: the arrayed deps that share a dimension with the - // target. (Scalar deps stay bare; the target self-reference is - // pinned implicitly via the subscripted `to[elem]` in the guard - // form built by `generate_scalar_to_element_equation`.) - let deps_to_subscript = |deps: &HashSet>| -> HashSet> { - deps.iter() - .filter(|d| { - source_vars - .get(d.as_str()) - .filter(|sv| sv.kind(db) != SourceVariableKind::Module) - .map(|sv| { - let dd = variable_dimensions(db, *sv, project); - !dd.is_empty() - && dd - .iter() - .any(|x| to_dims.iter().any(|td| td.name() == x.name())) - }) - .unwrap_or(false) - }) - .cloned() - .collect() + // scalar equation: the arrayed deps, each over its OWN declared dimensions. + // (Scalar deps stay bare; the target self-reference is pinned implicitly + // via the subscripted `to[elem]` in the guard form built by + // `generate_scalar_to_element_equation`.) + let pinnable_deps = |deps: &HashSet>| { + pinnable_arrayed_deps(db, source_vars, project, deps, |_| true) }; let dim_element_lists: Vec> = to_dims .iter() .map(crate::ltm_augment::dimension_element_names) .collect(); + let to_dim_names: Vec = to_dims.iter().map(|d| d.name().to_string()).collect(); + let dim_ctx = project_dimensions_context(db, project); let elements = cartesian_subscripts(&dim_element_lists); // GH #910: resolved once for the whole target -- see `WithLookupSlotRefs`. let slot_refs = crate::ltm_augment::WithLookupSlotRefs::new(&to_var, target_ast_dims); @@ -1262,7 +1314,7 @@ pub(super) fn try_scalar_to_arrayed_link_scores( let build_var = |element: &str, elem_eqn: Option<&crate::ast::Expr0>, elem_deps: &HashSet>, - deps_to_sub: &HashSet>| + pinnable: &[(Ident, Vec)]| -> Option { let name = format!( "$\u{205A}ltm\u{205A}link_score\u{205A}{}\u{2192}{}[{}]", @@ -1290,6 +1342,14 @@ pub(super) fn try_scalar_to_arrayed_link_scores( let gf_table_ref = slot_refs.for_element(&crate::common::CanonicalElementName::from_raw(element)); let occ = slot_occurrences.for_slot(slot_map.slot_for(element)); + // Each arrayed dep is pinned over ITS OWN declared dimensions, + // projected off this target element (GH #974). + let deps_to_sub = crate::ltm_augment::dep_element_pins( + pinnable, + &to_dim_names, + &target_elem_by_dim_for(&to_dims, &dim_element_lists, element), + dim_ctx, + ); match crate::ltm_augment::generate_scalar_to_element_equation( from, to, @@ -1302,13 +1362,13 @@ pub(super) fn try_scalar_to_arrayed_link_scores( // substitution is a no-op. &std::collections::HashMap::new(), elem_deps, - deps_to_sub, + &deps_to_sub, // A true scalar source: the bare `quote_ident(from)` // denominator is correct, and there is no source subscript // to pin in the partial body. None, &[], - Some(project_dimensions_context(db, project)), + Some(dim_ctx), gf_table_ref.as_deref(), &occ, ) { @@ -1345,14 +1405,15 @@ pub(super) fn try_scalar_to_arrayed_link_scores( let mut cross_vars = Vec::with_capacity(elements.len()); match ast { // ApplyToAll: one shared body for every element, so its text, its - // dependency set, and the subset to element-pin are all - // element-invariant -- compute them once, outside the loop. + // dependency set, and the deps' declared dimensions are all + // element-invariant -- compute them once, outside the loop. Only the + // pin PROJECTION varies per element, and that is `build_var`'s job. Ast::ApplyToAll(_, expr) => { let elem_eqn = crate::patch::expr2_to_expr0(expr); let elem_deps = crate::variable::identifier_set(ast, target_ast_dims, None); - let deps_to_sub = deps_to_subscript(&elem_deps); + let pinnable = pinnable_deps(&elem_deps); for element in &elements { - match build_var(element, Some(&elem_eqn), &elem_deps, &deps_to_sub) { + match build_var(element, Some(&elem_eqn), &elem_deps, &pinnable) { Some(v) => cross_vars.push(v), None => { unscoreable_edges.insert((from.to_string(), to.to_string())); @@ -1385,8 +1446,8 @@ pub(super) fn try_scalar_to_arrayed_link_scores( // partials. None => (None, HashSet::new()), }; - let deps_to_sub = deps_to_subscript(&elem_deps); - match build_var(element, elem_eqn.as_ref(), &elem_deps, &deps_to_sub) { + let pinnable = pinnable_deps(&elem_deps); + match build_var(element, elem_eqn.as_ref(), &elem_deps, &pinnable) { Some(v) => cross_vars.push(v), None => { unscoreable_edges.insert((from.to_string(), to.to_string())); @@ -2363,11 +2424,11 @@ fn emit_per_element_link_scores( use crate::common::{Canonical, Ident}; /// One target element's equation parts: (body text, full dep set, the - /// arrayed deps to element-pin). + /// arrayed deps to element-pin with their declared dimensions). type ElemEqnParts = ( crate::ast::Expr0, HashSet>, - HashSet>, + Vec<(Ident, Vec)>, ); let Some(from_sv) = source_vars.get(from) else { @@ -2421,34 +2482,19 @@ fn emit_per_element_link_scores( // the whole stream, which was quadratic in the element count. let slot_occurrences = crate::ltm_augment::SlotOccurrences::new(to_occurrences); - // Arrayed deps sharing a target dim get element-pinned in the scalar - // per-element equation (mirroring `try_scalar_to_arrayed_link_scores`); - // the source itself is excluded -- its occurrences are pinned per-row - // by the equation builder's rewrite pass. - let deps_to_subscript = |deps: &HashSet>| -> HashSet> { - deps.iter() - .filter(|d| { - d.as_str() != from - && source_vars - .get(d.as_str()) - .filter(|sv| sv.kind(db) != SourceVariableKind::Module) - .map(|sv| { - let dd = variable_dimensions(db, *sv, project); - !dd.is_empty() - && dd - .iter() - .any(|x| to_dims.iter().any(|td| td.name() == x.name())) - }) - .unwrap_or(false) - }) - .cloned() - .collect() + // Arrayed deps get element-pinned in the scalar per-element equation, each + // over its OWN declared dimensions (mirroring + // `try_scalar_to_arrayed_link_scores`); the source itself is excluded -- + // its occurrences are pinned per-row by the wrap's own row lowering. + let pinnable_deps = |deps: &HashSet>| { + pinnable_arrayed_deps(db, source_vars, project, deps, |d| d.as_str() != from) }; let to_dim_element_lists: Vec> = to_dims .iter() .map(crate::ltm_augment::dimension_element_names) .collect(); + let to_dim_names: Vec = to_dims.iter().map(|d| d.name().to_string()).collect(); let elements = cartesian_subscripts(&to_dim_element_lists); // Per target element: the body text + dep sets (shared for A2A, @@ -2456,8 +2502,8 @@ fn emit_per_element_link_scores( let a2a_parts: Option = if let Ast::ApplyToAll(_, expr) = ast { let eqn = crate::patch::expr2_to_expr0(expr); let deps = crate::variable::identifier_set(ast, target_ast_dims, None); - let to_sub = deps_to_subscript(&deps); - Some((eqn, deps, to_sub)) + let pinnable = pinnable_deps(&deps); + Some((eqn, deps, pinnable)) } else { None }; @@ -2472,17 +2518,10 @@ fn emit_per_element_link_scores( for element in &elements { // The projection data `e` supplies: target dim (canonical) -> // (element name, index within that dim's element list). - let parts: Vec<&str> = element.split(',').collect(); - if parts.len() != to_dims.len() { + if element.split(',').count() != to_dims.len() { continue; } - let mut target_elem_by_dim: HashMap = HashMap::new(); - for ((dim, elems), part) in to_dims.iter().zip(&to_dim_element_lists).zip(&parts) { - let Some(idx) = elems.iter().position(|e| e == part) else { - continue; - }; - target_elem_by_dim.insert(dim.name().to_string(), ((*part).to_string(), idx)); - } + let target_elem_by_dim = target_elem_by_dim_for(&to_dims, &to_dim_element_lists, element); let qualified_element = crate::ltm_augment::qualify_element_csv(element, &to_dims); // GH #910: the partials below re-evaluate this element's RAW equation // while the guard form ratios them against `Δto[e]` -- which is @@ -2504,8 +2543,8 @@ fn emit_per_element_link_scores( target_ast_dims, None, ); - let to_sub = deps_to_subscript(&deps); - Some((eqn, deps, to_sub)) + let pinnable = pinnable_deps(&deps); + Some((eqn, deps, pinnable)) } // No slot, no default: a hole -- this element has no // equation, so the site cannot occur in it; skip. @@ -2514,9 +2553,17 @@ fn emit_per_element_link_scores( } _ => None, }; - let Some((body_eqn, deps, to_sub)) = a2a_parts.as_ref().or(slot_parts.as_ref()) else { + let Some((body_eqn, deps, pinnable)) = a2a_parts.as_ref().or(slot_parts.as_ref()) else { continue; }; + // Each arrayed dep pinned over ITS OWN declared dimensions, projected + // off this target element (GH #974). + let to_sub = crate::ltm_augment::dep_element_pins( + pinnable, + &to_dim_names, + &target_elem_by_dim, + dim_ctx, + ); for (axes, site_target_element) in sites { // A site inside an `Ast::Arrayed` slot contributes only to its @@ -2566,7 +2613,7 @@ fn emit_per_element_link_scores( &qualified_element, body_eqn, deps, - to_sub, + &to_sub, from_dims, &target_elem_by_dim, &target_iterated_dims, @@ -3233,38 +3280,27 @@ pub(super) fn emit_agg_to_target_link_scores( for other_agg in reducer_subst.values() { all_deps.insert(Ident::::new(other_agg)); } - let deps_to_subscript: HashSet> = all_deps - .iter() - .filter(|d| { - source_vars - .get(d.as_str()) - .filter(|sv| sv.kind(db) != SourceVariableKind::Module) - .map(|sv| { - let dd = variable_dimensions(db, *sv, project); - !dd.is_empty() - && dd - .iter() - .any(|x| to_dims.iter().any(|td| td.name() == x.name())) - }) - .unwrap_or(false) - }) - .cloned() - .collect(); + let pinnable_deps = pinnable_arrayed_deps(db, source_vars, project, &all_deps, |_| true); // An arrayed agg (`result_dims` non-empty) is element-pinned in the // per-target-element equation too: `$⁚ltm⁚agg⁚0` → `$⁚ltm⁚agg⁚0[]`. - // But NOT via `deps_to_subscript`: that set pins to `to`'s FULL element - // tuple, which is an agg's correct subscript only in the diagonal case - // (`result_dims` == `to`'s dims) and over-subscripts the agg in the - // broadcast case (`agg[D1]` feeding `to[D1,D2]` -- the fragment then - // fails to compile and the score is stubbed to a constant 0, zeroing - // every loop through the agg; GH #528). Instead EACH arrayed agg -- - // the live one AND every frozen co-agg (GH #751) -- is pinned to the - // target element's PROJECTION onto its own `result_dims` axes (for the - // live agg, the same slot the link-score name and the `Δsource` - // denominator carry) via `generate_scalar_to_element_equation`'s - // per-ident `source_pins`, computed by `source_pins_for_target` below. + // But NOT via `pinnable_deps`: that projection needs a variable's DECLARED + // dimensions, and a synthetic agg is not a model variable and has none -- + // its slot space is the hoisted reducer's `result_dims`, which is the + // target's full element tuple only in the diagonal case and a strict + // prefix in the broadcast case (`agg[D1]` feeding `to[D1,D2]`; GH #528). + // So EACH arrayed agg -- the live one AND every frozen co-agg (GH #751) -- + // is pinned to the target element's PROJECTION onto its own `result_dims` + // axes (for the live agg, the same slot the link-score name and the + // `Δsource` denominator carry) via + // `generate_scalar_to_element_equation`'s per-ident `source_pins`, + // computed by `source_pins_for_target` below. let dim_ctx = project_dimensions_context(db, project); + let to_dim_names: Vec = to_dims.iter().map(|d| d.name().to_string()).collect(); + let dim_element_lists_for_target: Vec> = to_dims + .iter() + .map(crate::ltm_augment::dimension_element_names) + .collect(); #[derive(Clone)] struct AggTargetProjectionAxis { target_pos: usize, @@ -3394,7 +3430,7 @@ pub(super) fn emit_agg_to_target_link_scores( // the AGG'S result-dim space, so mapped target dims pin the helper slot they // actually read (`state·s1`), not the target slot (`region·r1`). let qualified_pin_for_target = - |projection: &AggTargetProjection, element: &str| -> Option { + |projection: &AggTargetProjection, element: &str| -> Option { let slot_parts = slot_parts_for_target(projection, element)?; if slot_parts.is_empty() { return None; @@ -3405,7 +3441,18 @@ pub(super) fn emit_agg_to_target_link_scores( .iter() .map(|axis| axis.result_dim.clone()) .collect(); - Some(crate::ltm_augment::qualify_element_csv(&slot, &result_dims)) + let qualified = crate::ltm_augment::qualify_element_csv(&slot, &result_dims); + Some(DepElementPin { + axes: result_dims + .iter() + .zip(qualified.split(',')) + .map(|(dim, elem)| (dim.name().to_string(), elem.to_string())) + .collect(), + // An agg's slot space IS its `result_dims`, and the projection + // either covers all of them or returned `None` above, so a bare + // agg reference is always spellable here. + complete: true, + }) }; // Every ARRAYED agg referenced by the substituted equation needs a body // pin (GH #751): the LIVE agg (held live; the GH #528 projection) and @@ -3432,14 +3479,24 @@ pub(super) fn emit_agg_to_target_link_scores( }; // The per-ident pin map for one target element: each arrayed agg pinned // to the target element's projection onto its own result axes. - let source_pins_for_target = |element: &str| -> Vec<(Ident, String)> { + let source_pins_for_target = |element: &str| -> Vec<(Ident, DepElementPin)> { arrayed_agg_pin_projections .iter() .filter_map(|(ident, projection)| { - qualified_pin_for_target(projection, element).map(|slot| (ident.clone(), slot)) + qualified_pin_for_target(projection, element).map(|pin| (ident.clone(), pin)) }) .collect() }; + // Each arrayed model-variable dep pinned over ITS OWN declared dimensions, + // projected off the target element (GH #974). + let deps_to_subscript = |element: &str| { + crate::ltm_augment::dep_element_pins( + &pinnable_deps, + &to_dim_names, + &target_elem_by_dim_for(&to_dims, &dim_element_lists_for_target, element), + dim_ctx, + ) + }; // Track A stage 1: reducer -> agg-name substitution is now a POST-transform // lowering INSIDE the generators (`generate_agg_to_scalar_target_equation` / @@ -3535,7 +3592,7 @@ pub(super) fn emit_agg_to_target_link_scores( &to_own_eqn, &reducer_subst, &all_deps, - &deps_to_subscript, + &deps_to_subscript(element), Some(&agg_source_ref_for_target(element)), &source_pins_for_target(element), Some(project_dimensions_context(db, project)), @@ -3614,7 +3671,7 @@ pub(super) fn emit_agg_to_target_link_scores( &crate::patch::expr2_to_expr0(slot_expr), &reducer_subst, &slot_deps, - &deps_to_subscript, + &deps_to_subscript(element), Some(&agg_source_ref_for_target(element)), &source_pins_for_target(element), Some(project_dimensions_context(db, project)), diff --git a/src/simlin-engine/src/db/ltm_char_tests.rs b/src/simlin-engine/src/db/ltm_char_tests.rs index cfaac9672..96d25fd91 100644 --- a/src/simlin-engine/src/db/ltm_char_tests.rs +++ b/src/simlin-engine/src/db/ltm_char_tests.rs @@ -405,6 +405,256 @@ fn per_element_subset_dep_scores_are_live_not_silent_zero() { } } +// --------------------------------------------------------------------------- +// GH #974: pinning a BARE arrayed reference over the DEP's declared dimensions. +// +// The two guards below cover the issue's two sibling arms, which share one +// projection (`post_transform::dep_element_pins`, enumerated by +// `dep_element_pins_projection_enumeration`) but reach it through different +// call paths: an other-dep's bare reference goes through +// `subscript_idents_at_element`, and a bare reference to the LIVE SOURCE goes +// through the wrap's own `pin_bare_source_ref`. +// +// Both models below compile with ZERO diagnostics before the fix -- the defects +// are entirely inside LTM generation. +// --------------------------------------------------------------------------- + +/// The equation text of the one link-score variable whose name contains every +/// fragment in `name_parts`, for a model built with [`char_fixture_db`]. +fn link_score_text( + db: &SimlinDb, + model: SourceModel, + project: SourceProject, + name_parts: &[&str], +) -> String { + let ltm = model_ltm_variables(db, model, project); + let matching: Vec<&LtmSyntheticVar> = ltm + .vars + .iter() + .filter(|v| name_parts.iter().all(|p| v.name.contains(p))) + .collect(); + assert_eq!( + matching.len(), + 1, + "expected exactly one link score matching {name_parts:?}, got {:?}", + matching.iter().map(|v| &v.name).collect::>() + ); + matching[0].equation.source_text().to_string() +} + +fn bare_dep_own_dims_model() -> datamodel::Project { + TestProject::new("bare_dep_own_dims") + .with_sim_time(0.0, 3.0, 1.0) + .named_dimension("Region", &["nyc", "boston"]) + .named_dimension("Age", &["young", "old"]) + .array_with_ranges_direct( + "regw", + vec!["Region".to_string()], + vec![("nyc", "10"), ("boston", "20")], + None, + ) + .array_with_ranges_direct( + "agew", + vec!["Age".to_string()], + vec![("young", "1"), ("old", "2")], + None, + ) + // Three deps referenced BARE in `growth`'s body, differing only in how + // their declared dimensions relate to the target's: identical, the same + // two REORDERED, and a strict SUBSET. + .array_aux_direct( + "same", + vec!["Region".to_string(), "Age".to_string()], + "regw[Region] + agew[Age]", + None, + ) + .array_aux_direct( + "flip", + vec!["Age".to_string(), "Region".to_string()], + "agew[Age] * 100 + regw[Region]", + None, + ) + .array_aux_direct("w", vec!["Age".to_string()], "agew[Age] * 1000", None) + .array_flow( + "growth[Region,Age]", + "pop[Region, young] * (same + flip + w) * 0.0001", + None, + ) + .array_stock("pop[Region,Age]", "10", &["growth"], &[], None) + .build_datamodel() +} + +/// A BARE arrayed dep in a per-element link-score partial is pinned over the +/// dimensions IT declares, matched by name -- not over the target's element +/// tuple (GH #974, arm 1). +/// +/// The test states the executed semantics first and the generated text second, +/// because the first is what makes the second right. The numeric oracle reads +/// the deps' own series out of a real run and asserts +/// `growth[nyc,old] == pop[nyc,young] * (same[nyc,old] + flip[old,nyc] + +/// w[old]) * 1e-4`: a bare reference reads each of its OWN axes' coordinates by +/// dimension name, so the reordered `flip[Age,Region]` reads `[old,nyc]` and the +/// subset `w[Age]` reads `[old]`. Every element carries a distinct value, so a +/// wrong element changes the sum. +/// +/// The pre-fix pin used the target's tuple `[region·nyc, age·old]` for all +/// three, which failed in two DIFFERENT ways -- and the difference is why both +/// rows are here. `w` got an arity-2 subscript over a 1-D variable, so its +/// fragment failed to compile and the score read a constant 0 (loud, via the +/// four `Assembly` warnings the issue reports). `flip` got an arity-2 subscript +/// that RESOLVED -- both indices are valid element references for the axes they +/// landed on -- so it compiled and silently read `flip[young,boston]`. That +/// second row had no diagnostic of any kind. +#[test] +fn bare_arrayed_dep_is_pinned_over_its_own_declared_dims() { + let project = bare_dep_own_dims_model(); + + // 1. What the SIMULATION reads. This is the claim the pin has to match, and + // it is checked against the engine rather than assumed. + let plain_db = SimlinDb::default(); + let plain_project = sync_from_datamodel(&plain_db, &project).project; + let compiled = compile_project_incremental(&plain_db, plain_project, "main") + .expect("the model compiles with no LTM"); + let mut vm = crate::vm::Vm::new(compiled.clone()).expect("VM creation should succeed"); + vm.run_to_end().expect("simulation should run"); + let results = vm.into_results(); + let at = |row: &[f64], name: &str| -> f64 { + let off = compiled + .offsets + .iter() + .find(|(k, _)| k.as_str() == name) + .unwrap_or_else(|| panic!("no slot named {name}")) + .1; + row[*off] + }; + let last = results.iter().next_back().expect("at least one saved step"); + let expected = at(last, "pop[nyc,young]") + * (at(last, "same[nyc,old]") + at(last, "flip[old,nyc]") + at(last, "w[old]")) + * 0.0001; + assert!( + (at(last, "growth[nyc,old]") - expected).abs() < 1e-9, + "a bare arrayed reference must read each of its OWN axes' coordinates by \ + dimension name: growth[nyc,old] = {}, expected {expected}", + at(last, "growth[nyc,old]") + ); + // Non-vacuity: the discriminating elements must differ, or a transposed or + // over-arity read would produce the same number. + assert_ne!(at(last, "flip[old,nyc]"), at(last, "flip[young,boston]")); + assert_ne!(at(last, "same[nyc,old]"), at(last, "same[boston,old]")); + + // 2. What the LTM partial spells, which must be the same reads. + let (db, model, source_project) = char_fixture_db(&project); + assert!( + fragment_compile_failures(&db, model, source_project).is_empty(), + "every per-element link-score fragment must compile: the subset-dims `w` \ + pin was arity-2 over a 1-D variable before the fix" + ); + let text = link_score_text( + &db, + model, + source_project, + &[ + "link_score\u{205A}pop[nyc,young]", + "\u{2192}growth[nyc,old]", + ], + ); + for expected in [ + "previous(same[region\u{B7}nyc, age\u{B7}old])", + "previous(flip[age\u{B7}old, region\u{B7}nyc])", + "previous(w[age\u{B7}old])", + ] { + assert!( + text.contains(expected), + "expected {expected:?} in the partial; got: {text}" + ); + } + assert!( + !text.contains("flip[region\u{B7}nyc, age\u{B7}old]"), + "the reordered dep must not be pinned with the TARGET's tuple -- that \ + spelling compiles and reads flip[young,boston]; got: {text}" + ); +} + +/// A BARE reference to the LIVE SOURCE inside a positionally-MAPPED per-element +/// target is pinned through the correspondence (GH #974, arm 2). +/// +/// `growth[State,Age] = pop[State, young] * pop * 0.01` over a `pop[Region,Age]` +/// source: the emitting `PerElement` site is `pop[State, young]`, and the bare +/// `pop` beside it is a second occurrence the wrap freezes. `pin_bare_source_ref` +/// matched dimension NAMES only, found no `Region` coordinate in a `State`-keyed +/// projection, and left the reference bare -- so the freeze became a bare +/// multi-slot `PREVIOUS(pop)`, which cannot compile in a scalar fragment. All +/// four per-element scores on the edge read a constant 0 behind four `Assembly` +/// warnings. +/// +/// The correspondence is positional, so `state·west` (State's first element) +/// reads `region·nyc` (Region's first) -- the same element the executed A2A +/// lowering reads for that slot. +#[test] +fn mapped_bare_live_source_ref_is_pinned_through_the_correspondence() { + let project = TestProject::new("mapped_bare_live_source") + .with_sim_time(0.0, 3.0, 1.0) + .named_dimension("Region", &["nyc", "boston"]) + .named_dimension("Age", &["young", "old"]) + .named_dimension_with_mapping("State", &["west", "east"], "Region") + .array_stock("pop[Region,Age]", "10", &["popinflow"], &[], None) + .array_flow("popinflow[Region,Age]", "1", None) + .array_flow("growth[State,Age]", "pop[State, young] * pop * 0.01", None) + .array_stock("stock[State,Age]", "0", &["growth"], &[], None) + .build_datamodel(); + + let (db, model, source_project) = char_fixture_db(&project); + assert!( + fragment_compile_failures(&db, model, source_project).is_empty(), + "the bare mapped live-source reference must be pinned, not left as a bare \ + multi-slot PREVIOUS(pop) that fails to compile" + ); + let text = link_score_text( + &db, + model, + source_project, + &[ + "link_score\u{205A}pop[nyc,young]", + "\u{2192}growth[west,young]", + ], + ); + assert!( + text.contains("PREVIOUS(pop[region\u{B7}nyc, age\u{B7}young])"), + "the bare live-source occurrence must be pinned to the mapped row for \ + this target element; got: {text}" + ); + assert!( + !text.contains("PREVIOUS(pop) "), + "the bare multi-slot freeze must be gone; got: {text}" + ); + + // ...and the scores must be materially live, not a compiled constant 0. + let compiled = compile_project_incremental(&db, source_project, "main") + .expect("LTM incremental compilation should succeed"); + let score_offsets: Vec = compiled + .offsets + .iter() + .filter(|(k, _)| { + k.as_str().contains("link_score\u{205A}pop[") && k.as_str().contains("\u{2192}growth[") + }) + .map(|(_, v)| *v) + .collect(); + assert_eq!( + score_offsets.len(), + 4, + "expected four per-element mapped pop->growth link scores" + ); + let mut vm = crate::vm::Vm::new(compiled.clone()).expect("VM creation should succeed"); + vm.run_to_end().expect("simulation should run"); + let results = vm.into_results(); + for off in score_offsets { + assert!( + results.iter().any(|row| row[off] != 0.0), + "mapped per-element pop->growth link score at offset {off} is all-zero" + ); + } +} + // --------------------------------------------------------------------------- // Model A3: per-element target whose body carries an additional pop occurrence // of DIFFERENT shape -- an all-iterated (`Bare`) live-source reference -- @@ -1646,3 +1896,140 @@ fn char_unquotable_source_name() { FragmentExpectation::AllCompile, ); } + +// --------------------------------------------------------------------------- +// GH #984, the REACHABLE half, end to end: a `LOOKUP` inside a scored target's +// equation. No other char fixture has one, so nothing else covers the arm at +// the model level. +// +// The runtime-index half of #984 has NO model fixture on purpose. A `LOOKUP` +// table argument carrying a runtime index does not compile on this engine at +// all: the TARGET's own fragment fails with an `EquationError` `DoesNotExist` +// and `compile_project_incremental` reports `NotSimulatable`, so the silent +// wrong score the issue describes cannot be reached from a compiling model +// today. Three independent constructions were tried and all three fail the +// same way -- an arrayed lookup-only table (`LOOKUP(gtab[idx], src)`), an +// arrayed graphical-function aux, and Vensim's own per-element arrayed-GF +// application (`g[idx](src)` via the MDL reader) -- while every static-index +// twin compiles. The wrap-level behaviour is therefore pinned where it is +// decided, by `ltm_augment::tests::test_partial_equation_lookup_table_index_is_frozen` +// (the general path) and `per_element_pin_freezes_a_runtime_table_arg_index` +// (the per-element path); fabricating a model fixture here would mean shipping +// one that does not exercise the engine. +// --------------------------------------------------------------------------- + +/// The MDL source for the arrayed-GF `LOOKUP` fixtures: `y = g[](src)`, +/// Vensim's per-element arrayed-GF application, which lowers to +/// `LOOKUP(g[], src)`. That is how a `LOOKUP` gets into a target equation +/// without the LTM layer synthesizing it. +fn arrayed_gf_lookup_mdl(index: &str) -> String { + format!( + "\ +{{UTF-8}} +D: A1, A2 ~~| +g[A1]( (0,0),(1,10),(2,20) ) ~~| +g[A2]( (0,0),(1,100),(2,200) ) ~~| +idx = 1 + MODULO(Time, 2) ~~| +inflow = 1 ~~| +src= INTEG(inflow, 1) ~~| +y = g[{index}](src) ~~| +INITIAL TIME = 0 ~~| +FINAL TIME = 3 ~~| +SAVEPER = 1 ~~| +TIME STEP = 1 ~~| +" + ) +} + +/// GH #984 through the PRODUCTION path: a runtime table index is frozen when the +/// dep set is the one production derives, not one a test hands in. +/// +/// This is the test the first version of the fix needed and did not have. The +/// wrap freezes an ident only if it is in `other_deps`, and `other_deps` comes +/// from `variable::identifier_set`, whose `BuiltinContents::LookupTable` arm +/// never walks the table expression -- so `idx`, referenced only as a table +/// index, is not a dependency of `y` and the freeze could not fire. The unit +/// tests supplied `idx` by hand and passed on an input production cannot +/// produce. Here nothing is supplied: `model_ltm_variables` builds the dep set +/// itself, so the assertion below is only satisfiable by +/// `freeze_lookup_table_indices` widening the set from the index's own idents. +/// +/// The target does NOT compile -- a `LOOKUP` table argument with a runtime index +/// is refused upstream, which is the documented reachability limitation above -- +/// so this asserts the generated TEXT rather than a score series, and asserts the +/// upstream refusal too so the reason is recorded rather than hidden. +#[test] +fn lookup_table_runtime_index_is_frozen_through_the_production_path() { + use crate::db::{DiagnosticError, collect_model_diagnostics}; + + let project = crate::open_vensim(&arrayed_gf_lookup_mdl("idx")).expect("the MDL parses"); + + // The upstream limitation, asserted rather than assumed: `y` itself has no + // bytecode, which is why there is no numeric oracle here. + let plain_db = SimlinDb::default(); + let plain = sync_from_datamodel(&plain_db, &project); + assert!( + collect_model_diagnostics(&plain_db, plain.models["main"].source, plain.project) + .iter() + .any(|d| d.variable.as_deref() == Some("y") + && matches!(&d.error, DiagnosticError::Equation(_))), + "the fixture's whole point is that a runtime table index is refused \ + upstream; if `y` now compiles, replace this text assertion with a \ + numeric one" + ); + + let (db, model, source_project) = char_fixture_db(&project); + let text = link_score_text( + &db, + model, + source_project, + &["link_score\u{205A}src\u{2192}y"], + ); + assert!( + text.contains("lookup(g[PREVIOUS(idx)], src)"), + "the table index must be frozen even though production does not classify \ + it as a dependency; got: {text}" + ); + assert!( + !text.contains("lookup(PREVIOUS("), + "the table HEAD must stay bare; got: {text}" + ); +} + +/// A static arrayed-GF table index survives the ceteris-paribus wrap untouched, +/// and the score compiles. +/// +/// Both halves of the `LOOKUP` arm are visible in one partial: the table HEAD +/// `g[a1]` is NOT wrapped in `PREVIOUS` (a graphical-function table has no value +/// slot, so `lookup(PREVIOUS(g[a1]), ...)` would fail to compile and zero the +/// score -- the WRLD3 failure mode), and its STATIC element index is not frozen +/// either (there is no runtime read to hold). The live source stays live in the +/// second argument. +/// +/// This is the reachable half of the arm, and the control for its sibling +/// [`lookup_table_runtime_index_is_frozen_through_the_production_path`]: same +/// model, same production path, index static instead of runtime. +#[test] +fn lookup_table_head_and_static_index_survive_the_wrap() { + let project = crate::open_vensim(&arrayed_gf_lookup_mdl("A1")).expect("the MDL parses"); + let (db, model, source_project) = char_fixture_db(&project); + assert!( + fragment_compile_failures(&db, model, source_project).is_empty(), + "the table-mediated link score must compile" + ); + + let text = link_score_text( + &db, + model, + source_project, + &["link_score\u{205A}src\u{2192}y"], + ); + assert!( + text.contains("lookup(g[a1], src)"), + "the table head must stay bare and its static index untouched; got: {text}" + ); + assert!( + !text.contains("lookup(PREVIOUS("), + "a PREVIOUS of a table has no value slot; got: {text}" + ); +} diff --git a/src/simlin-engine/src/dimensions.rs b/src/simlin-engine/src/dimensions.rs index 4b5e10972..94aeb6680 100644 --- a/src/simlin-engine/src/dimensions.rs +++ b/src/simlin-engine/src/dimensions.rs @@ -101,6 +101,76 @@ pub enum Dimension { Named(CanonicalDimensionName, NamedDimension), } +/// How a bare-identifier subscript index resolves against ONE axis of the +/// variable being subscripted -- see [`resolve_axis_index_name`]. +#[cfg_attr(feature = "debug-derive", derive(Debug))] +#[derive(Clone, PartialEq, Eq)] +pub enum AxisIndexName { + /// The axis's own dimension declares this name as an ELEMENT. Carries the + /// element's canonical name. + Element(String), + /// Not an element of this axis, but a dimension the enclosing equation + /// ITERATES -- the apply-to-all placeholder form, which stands for whatever + /// element the current iteration selects. + IteratedDim, + /// Neither: a variable read, or a name nothing in scope declares. + Unresolved, +} + +/// Resolve a bare-identifier subscript index against the axis it indexes: the +/// engine's SINGLE precedence rule for the element-vs-dimension-name question. +/// +/// The axis's own declared ELEMENTS are tried first; only a name the axis does +/// not declare is read as a dimension the enclosing equation iterates. The two +/// readings collide when a dimension declares an element whose name is also a +/// dimension name -- `Category = [Region, x]` beside a `Region` dimension -- +/// which XMILE permits, and the order is what decides which row a reference +/// like `effect[Region, Region]` reads. +/// +/// **Element-first is the engine's executed behaviour**, and that is the +/// binding reason: `compiler::subscript::normalize_subscripts3`'s +/// `IndexExpr3::Expr` / `Expr3::Var` arm looks the name up in the axis's own +/// `indexed_elements` ("First check if it's a named dimension element (takes +/// priority)") and only then emits an `IndexOp::ActiveDimRef` for a dimension +/// name; its `IndexExpr3::Dimension` arm does the same. Anything that DESCRIBES +/// a reference -- an LTM read slice, an access shape, an element-graph edge -- +/// must resolve it the way the simulation does, or it describes rows the +/// simulation never reads. +/// +/// The XMILE spec (`docs/reference/xmile-v1.0.html`) points the same way but +/// does not settle this exact pair, and the difference is worth being precise +/// about. Footnote [9] settles the ADJACENT pair outright -- "if a variable name +/// is the same as an element name, the element name prevails (i.e., the variable +/// name is hidden within that subscript)" -- but that is variable-vs-element, +/// not dimension-name-vs-element. What bears on this pair is section 2.1's +/// namespace rule, "Dimension names, in turn, define their own Element +/// namespace ... Element names are resolved by context when they appear inside +/// square brackets of a variable", together with section 3.7.1's "Subscript +/// index names MAY be used unambiguously as part of a subscript (i.e., inside +/// the square brackets) ... once the dimensions assigned to the variable have +/// been specified". Inside brackets, at a position whose dimension is known, the +/// spec designates the index-name reading and calls it unambiguous; the +/// apply-to-all dimension-name form is introduced afterwards, as an additional +/// spelling, and cannot retroactively make the base one ambiguous. That is an +/// argument from the namespace rule, not a quotation of a rule for this pair. +/// +/// `target_iterates` answers "does the enclosing equation iterate this +/// dimension?" -- the caller's own notion of scope (the LTM callers pass the +/// target equation's iterated dimensions). +pub fn resolve_axis_index_name( + name: &str, + axis_dim: &Dimension, + target_iterates: impl FnOnce(&str) -> bool, +) -> AxisIndexName { + if let Some(elem) = axis_dim.canonical_element(name) { + return AxisIndexName::Element(elem); + } + if target_iterates(name) { + return AxisIndexName::IteratedDim; + } + AxisIndexName::Unresolved +} + impl Dimension { pub fn len(&self) -> usize { match self { @@ -122,6 +192,28 @@ impl Dimension { } } + /// The canonical name of the element `name` selects on this axis, or `None` + /// when the axis declares no such element. + /// + /// For a NAMED dimension that is the element name itself; for an INDEXED one + /// it is the 1-based position re-formatted (`"01"` -> `"1"`), so the answer + /// always matches the names `ltm_augment::dimension_element_names` produces + /// for the same axis. Membership is the same test `get_offset` performs and + /// the same one `compiler::subscript` resolves an index against. + pub fn canonical_element(&self, name: &str) -> Option { + match self { + Dimension::Named(_, named) => named + .indexed_elements + .get_key_value(&CanonicalElementName::from_raw(name)) + .map(|(elem, _)| elem.as_str().to_string()), + Dimension::Indexed(_, size) => name + .parse::() + .ok() + .filter(|n| *n >= 1 && n <= size) + .map(|n| n.to_string()), + } + } + /// Get the offset of an element by name (for named dimensions) or by index string (for indexed dimensions). /// Returns 0-based offset for use in array indexing. pub fn get_offset(&self, subscript: &CanonicalElementName) -> Option { diff --git a/src/simlin-engine/src/ltm_agg.rs b/src/simlin-engine/src/ltm_agg.rs index dde94a24e..cf1d27ff9 100644 --- a/src/simlin-engine/src/ltm_agg.rs +++ b/src/simlin-engine/src/ltm_agg.rs @@ -1734,45 +1734,50 @@ pub(crate) fn classify_axis_access( IndexExpr2::Expr(Expr2::Var(name, _, _)) => { let name_str = name.as_str(); let src_dim_name = axis_dim.name(); - // An iterated-dimension index: the axis is iterated over the - // target's dimension space (and the agg result varies per - // element of it) iff `name` is one of the target's iterated - // dims AND it lines up with the source's axis dim by name or by - // a positional mapping (GH #534). - if target_iterated_dims.iter().any(|t| t == name_str) { - if name_str == src_dim_name { - Some(AxisRead::Iterated { - dim: name_str.to_string(), - source_dim: src_dim_name.to_string(), - }) - } else { - // The iterated dim names a *different* source axis: a - // positional remap (`State→Region`, GH #534) is accepted - // -- carrying the (target, source) pair so the emitters - // remap each row to its slot -- when the slot remap - // exists. `iterated_axis_slot_elements` consults - // `mapped_element_correspondence`, which accepts BOTH - // declaration directions (GH #757 -- the former - // `has_mapping_to(d, src)` forward-only pre-gate was - // dropped) and declines explicit element maps (execution - // resolves positionally and ignores the map, the GH #756 - // gate). Everything else -- a plain position mismatch, - // an element-mapped pair -- declines, keeping the - // reference on the conservative path. - let elems = crate::ltm_augment::dimension_element_names(axis_dim); - if iterated_axis_slot_elements(name_str, src_dim_name, &elems, dim_ctx) - .is_some() - { + // The element-vs-dimension-name precedence is the SHARED + // `dimensions::resolve_axis_index_name` -- the compiler's own + // element-first order (GH #986), which + // `ltm_augment_post_transform::pin_dimension_name_indices` reads + // too, so the two rules cannot disagree about which row a colliding + // name selects. + match crate::dimensions::resolve_axis_index_name(name_str, axis_dim, |n| { + target_iterated_dims.iter().any(|t| t == n) + }) { + crate::dimensions::AxisIndexName::Element(elem) => Some(AxisRead::Pinned(elem)), + // An iterated-dimension index: the axis is iterated over the + // target's dimension space (and the agg result varies per + // element of it) iff it lines up with the source's axis dim by + // name or by a positional mapping (GH #534). + crate::dimensions::AxisIndexName::IteratedDim => { + if name_str == src_dim_name { Some(AxisRead::Iterated { dim: name_str.to_string(), source_dim: src_dim_name.to_string(), }) } else { - None + // The iterated dim names a *different* source axis: a + // positional remap (`State→Region`, GH #534) is accepted + // -- carrying the (target, source) pair so the emitters + // remap each row to its slot -- when the slot remap + // exists. `iterated_axis_slot_elements` consults + // `mapped_element_correspondence`, which accepts BOTH + // declaration directions (GH #757 -- the former + // `has_mapping_to(d, src)` forward-only pre-gate was + // dropped) and declines explicit element maps (execution + // resolves positionally and ignores the map, the GH #756 + // gate). Everything else -- a plain position mismatch, + // an element-mapped pair -- declines, keeping the + // reference on the conservative path. + let elems = crate::ltm_augment::dimension_element_names(axis_dim); + iterated_axis_slot_elements(name_str, src_dim_name, &elems, dim_ctx).map( + |_| AxisRead::Iterated { + dim: name_str.to_string(), + source_dim: src_dim_name.to_string(), + }, + ) } } - } else { - resolve_literal_axis_index(idx, axis_dim).map(AxisRead::Pinned) + crate::dimensions::AxisIndexName::Unresolved => None, } } IndexExpr2::Expr(Expr2::Const(..)) => { @@ -3039,2828 +3044,5 @@ fn canonical_dim_to_datamodel(canonical: &str, dm_dims: &[crate::datamodel::Dime } #[cfg(test)] -mod tests { - use super::*; - use crate::db::{SimlinDb, sync_from_datamodel}; - use crate::test_common::TestProject; - - /// Test helper: the source-variable names of an agg (sorted + deduped - /// by the [`AggNode::sources`] construction invariant). - fn source_names(a: &AggNode) -> Vec<&str> { - a.sources.iter().map(|s| s.var.as_str()).collect() - } - - /// Build a `TestProject`, sync into salsa, and return the enumerated - /// aggregate nodes for the "main" model. - fn agg_nodes(project: &TestProject) -> AggNodesResult { - let datamodel = project.build_datamodel(); - let db = SimlinDb::default(); - let sync = sync_from_datamodel(&db, &datamodel); - let source_model = sync.models["main"].source; - let source_project = sync.project; - enumerate_agg_nodes(&db, source_model, source_project).clone() - } - - /// Build a `TestProject` and return the GH #791 cartesian-decline verdict - /// for the `from -> to` edge. - fn source_read(project: &TestProject, from: &str, to: &str) -> UnhoistedSourceRead { - let datamodel = project.build_datamodel(); - let db = SimlinDb::default(); - let sync = sync_from_datamodel(&db, &datamodel); - let link = LtmLinkId::new(&db, from.to_string(), to.to_string()); - unhoisted_reducer_source_read(&db, link, sync.models["main"].source, sync.project).clone() - } - - /// GH #791: a multi-source reducer whose source read is a STRICT slice - /// (`pop[nyc,*]`, with no full-extent read of `pop`) is the silent-cartesian - /// family -- `StrictSlice` (the caller loud-skips it), carrying the actual - /// slice so the diagnostic renders `pop[nyc,*]` rather than a canned - /// example. - #[test] - fn unhoisted_source_read_strict_slice_for_pinned_only_read() { - let project = TestProject::new("strict_slice") - .named_dimension("Region", &["nyc", "boston"]) - .named_dimension("D2", &["p", "q"]) - .array_aux("pop[Region,D2]", "1") - .array_aux("w[D2]", "0.5") - .array_aux("share[Region]", "SUM(pop[nyc,*] * w[*])"); - let UnhoistedSourceRead::StrictSlice(slice) = source_read(&project, "pop", "share") else { - panic!("the pinned-only read must classify StrictSlice"); - }; - assert_eq!(render_read_slice_for_diagnostic(&slice), "nyc,*"); - } - - /// GH #793: a hoisted full-extent sibling reducer must not mask an - /// un-hoisted strict-slice sibling on the same `pop -> share` edge. The - /// full-extent read is already represented by synthetic agg halves, so the - /// residual un-hoisted verdict remains `StrictSlice`. - #[test] - fn unhoisted_source_read_ignores_hoisted_sibling_full_read() { - let project = TestProject::new("strict_with_hoisted_sibling") - .named_dimension("Region", &["nyc", "boston"]) - .named_dimension("D2", &["p", "q"]) - .array_aux("pop[Region,D2]", "1") - .array_aux("w[D2]", "0.5") - .array_aux_direct( - "share", - vec!["Region".into()], - "SUM(pop[nyc, *] * w[*]) + SUM(pop[*, *])", - None, - ); - let UnhoistedSourceRead::StrictSlice(slice) = source_read(&project, "pop", "share") else { - panic!("the un-hoisted strict sibling must not be masked by the hoisted full read"); - }; - assert_eq!(render_read_slice_for_diagnostic(&slice), "nyc,*"); - } - - /// GH #791 boundary: the SAME variable read at full extent (`pop[*]`) AND - /// pinned (`pop[north]`) -- the GH #744 self-reference family -- leaves NO - /// row unread, so it is `FullExtent` (the caller keeps the conservative - /// delta-ratio cartesian, unchanged). - #[test] - fn unhoisted_source_read_full_extent_when_full_read_present() { - let project = TestProject::new("self_ref") - .named_dimension("region", &["north", "south"]) - .array_aux("pop[region]", "1") - .scalar_aux("tp", "SUM(pop[*] * pop[north])"); - assert!(matches!( - source_read(&project, "pop", "tp"), - UnhoistedSourceRead::FullExtent - )); - } - - /// GH #791 boundary: a pure full-extent multi-source read (`matrix[D1,*]`, - /// `[Iterated, Reduced]`) is `FullExtent` -- the #779 bare-feeder fixture's - /// `matrix -> growth` edge keeps its correct cartesian diagonal. - #[test] - fn unhoisted_source_read_full_extent_for_iterated_reduced() { - let project = TestProject::new("iter_reduced") - .named_dimension("D1", &["a", "b"]) - .named_dimension("D2", &["c", "d"]) - .array_aux("matrix[D1,D2]", "1") - .array_aux("frac", "0.5") - .array_aux("growth[D1]", "SUM(matrix[D1,*] * frac)"); - assert!(matches!( - source_read(&project, "matrix", "growth"), - UnhoistedSourceRead::FullExtent - )); - } - - /// GH #791 boundary: a dynamic-index reducer (`SUM(pop[idx,*])`, `idx` - /// non-literal) is NOT statically describable -- `NotDescribable`, so the - /// caller keeps the DOCUMENTED conservative cartesian cross-product. - #[test] - fn unhoisted_source_read_not_describable_for_dynamic_index() { - let project = TestProject::new("dyn_index") - .named_dimension("Region", &["nyc", "boston"]) - .named_dimension("D2", &["p", "q"]) - .array_aux("pop[Region,D2]", "1") - .scalar_aux("idx", "2") - .array_aux("share[Region]", "SUM(pop[idx,*])"); - assert!(matches!( - source_read(&project, "pop", "share"), - UnhoistedSourceRead::NotDescribable - )); - } - - /// GH #792: a PER-ELEMENT-EQUATION (`Ast::Arrayed`) owner whose every slot - /// holds a strict-slice multi-source reducer (each `share` slot is - /// `SUM(pop[,*] * w[*])`) classifies the `pop -> share` edge - /// `PerElementReducerRead` -- a decline. The first describable slice in - /// sorted-slot order (`boston`) rides along for the diagnostic. - #[test] - fn unhoisted_source_read_declines_per_element_strict_slots() { - let project = TestProject::new("per_element_strict") - .named_dimension("Region", &["nyc", "boston"]) - .named_dimension("D2", &["p", "q"]) - .array_aux("pop[Region,D2]", "1") - .array_aux("w[D2]", "0.5") - .array_with_ranges_direct( - "share", - vec!["Region".into()], - vec![ - ("nyc", "SUM(pop[nyc,*] * w[*])"), - ("boston", "SUM(pop[boston,*] * w[*])"), - ], - None, - ); - let UnhoistedSourceRead::PerElementReducerRead(Some(slice)) = - source_read(&project, "pop", "share") - else { - panic!("per-element strict-slice slots must classify PerElementReducerRead(Some)"); - }; - // Sorted-key walk visits `boston` before `nyc`. - assert_eq!(render_read_slice_for_diagnostic(&slice), "boston,*"); - } - - /// GH #792 any-reducer-read rule: ONLY ONE slot reads `pop` (inside a - /// reducer); the other slot does not read `pop` at all. Any slot's reducer - /// read declines the WHOLE edge (the Bare stand-in conflates the slots). - #[test] - fn unhoisted_source_read_declines_when_only_some_slots_read() { - let project = TestProject::new("per_element_some") - .named_dimension("Region", &["nyc", "boston"]) - .named_dimension("D2", &["p", "q"]) - .array_aux("pop[Region,D2]", "1") - .array_aux("w[D2]", "0.5") - .array_with_ranges_direct( - "share", - vec!["Region".into()], - vec![("nyc", "SUM(pop[nyc,*] * w[*])"), ("boston", "0")], - None, - ); - let UnhoistedSourceRead::PerElementReducerRead(Some(slice)) = - source_read(&project, "pop", "share") - else { - panic!("a single reducer-reading slot must decline the whole edge"); - }; - assert_eq!(render_read_slice_for_diagnostic(&slice), "nyc,*"); - } - - /// GH #792 finding 2: a per-element owner whose every slot reads `pop` at - /// FULL EXTENT inside an I1-declined multi-source reducer - /// (`SUM(pop[*,*] * w[*])`) ALSO declines -- the full-extent verdict only - /// validates the cartesian projection, which needs a single dt-expression - /// a per-element owner does not have; the Bare stand-in is just as wrong - /// for full reads (verified ~-0.0 empirically). - #[test] - fn unhoisted_source_read_declines_per_element_full_extent_slots() { - let project = TestProject::new("per_element_full") - .named_dimension("Region", &["nyc", "boston"]) - .named_dimension("D2", &["p", "q"]) - .array_aux("pop[Region,D2]", "1") - .array_aux("w[D2]", "0.5") - .array_with_ranges_direct( - "share", - vec!["Region".into()], - vec![ - ("nyc", "SUM(pop[*,*] * w[*])"), - ("boston", "SUM(pop[*,*] * w[*])"), - ], - None, - ); - let UnhoistedSourceRead::PerElementReducerRead(Some(slice)) = - source_read(&project, "pop", "share") - else { - panic!("per-element full-extent multi-source slots must still decline"); - }; - assert_eq!(render_read_slice_for_diagnostic(&slice), "*,*"); - } - - /// GH #792 finding 1: the DIM-NAME spelling (`SUM(pop[Region,*] * w[*])` - /// per slot). In a per-element slot no iterated dimension is in scope - /// (mirroring `enumerate_agg_nodes`' Arrayed arm), so the dim-named index - /// is not statically describable -- but the read IS a reducer read, so the - /// edge still declines, with no representative slice for the diagnostic. - /// (Execution pins `Region` to the slot's element -- a strict read -- so - /// the previous `Iterated => full extent` classification was wrong; the - /// executed-value pin lives in the integration twin.) - #[test] - fn unhoisted_source_read_declines_per_element_dim_named_slots() { - let project = TestProject::new("per_element_dim_named") - .named_dimension("Region", &["nyc", "boston"]) - .named_dimension("D2", &["p", "q"]) - .array_aux("pop[Region,D2]", "1") - .array_aux("w[D2]", "0.5") - .array_with_ranges_direct( - "share", - vec!["Region".into()], - vec![ - ("nyc", "SUM(pop[Region,*] * w[*])"), - ("boston", "SUM(pop[Region,*] * w[*])"), - ], - None, - ); - assert!(matches!( - source_read(&project, "pop", "share"), - UnhoistedSourceRead::PerElementReducerRead(None) - )); - } - - /// GH #792 explicit sub-case decision: a DYNAMIC-INDEX reducer read inside - /// a per-element slot (`SUM(pop[idx,*])`) also declines. Pre-fix it was - /// silently stand-in'd exactly like the strict spelling (the scalar/A2A - /// dynamic-index family keeps its documented conservative cartesian, but a - /// per-element owner has no cartesian arm to keep), so declining is both - /// sound and consistent; no existing test pinned the old silent behavior. - #[test] - fn unhoisted_source_read_declines_per_element_dynamic_index_slots() { - let project = TestProject::new("per_element_dyn") - .named_dimension("Region", &["nyc", "boston"]) - .named_dimension("D2", &["p", "q"]) - .array_aux("pop[Region,D2]", "1") - .scalar_aux("idx", "2") - .array_with_ranges_direct( - "share", - vec!["Region".into()], - vec![("nyc", "SUM(pop[idx,*])"), ("boston", "SUM(pop[idx,*])")], - None, - ); - assert!(matches!( - source_read(&project, "pop", "share"), - UnhoistedSourceRead::PerElementReducerRead(None) - )); - } - - /// GH #792 non-decline boundary: a per-element owner that references `pop` - /// only OUTSIDE any reducer (the disjoint-dim FixedIndex family's shape) - /// classifies `NotDescribable` -- no reducer read, so the edge keeps its - /// existing emission path (`try_disjoint_dim_arrayed_link_scores` et al). - #[test] - fn unhoisted_source_read_not_describable_for_per_element_non_reducer_refs() { - let project = TestProject::new("per_element_bare") - .named_dimension("Region", &["nyc", "boston"]) - .named_dimension("D2", &["p", "q"]) - .array_aux("pop[Region,D2]", "1") - .array_with_ranges_direct( - "share", - vec!["Region".into()], - vec![ - ("nyc", "pop[nyc,p] * 0.5"), - ("boston", "pop[boston,q] * 0.5"), - ], - None, - ); - assert!(matches!( - source_read(&project, "pop", "share"), - UnhoistedSourceRead::NotDescribable - )); - } - - /// AC4.3: a variable whose entire dt-equation is exactly one reducer call - /// (scalar) mints no synthetic agg -- the variable itself is the agg. - #[test] - fn whole_rhs_scalar_reducer_is_its_own_agg() { - let project = TestProject::new("whole_rhs") - .named_dimension("Region", &["NYC", "Boston", "LA"]) - .array_aux("population[Region]", "100") - .scalar_aux("total_population", "SUM(population[*])"); - - let result = agg_nodes(&project); - - // No `$⁚ltm⁚agg⁚{n}` minted. - assert!( - result.aggs.iter().all(|a| !a.is_synthetic), - "whole-RHS scalar reducer must not mint a synthetic agg; got: {:?}", - result.aggs - ); - // The reducer maps to a variable-backed agg named `total_population`, - // owned by `total_population`'s equation. (Variable-backed aggs are - // resolved via `aggs_in_var`, not `agg_for_key` -- the latter is - // synthetic-only, since two different scalars can each be `SUM(pop[*])`.) - let agg = result - .aggs_in_var("total_population") - .find(|a| a.name == "total_population") - .expect("expected a variable-backed agg owned by `total_population`"); - assert!(!agg.is_synthetic); - assert_eq!(source_names(agg), vec!["population"]); - assert!(agg.result_dims.is_empty()); - // `agg_for_key` resolves only synthetic aggs, so it must not find this one. - assert!(result.agg_for_key("sum(population[*])").is_none()); - } - - /// AC4.3 (arrayed variant): `agg[D1] = SUM(matrix[D1,*])` is whole-RHS, so - /// the variable is the agg; `result_dims` carries `D1` and `read_slice` - /// records the `Iterated(D1)` / `Reduced` axis split (the `D1` axis is - /// iterated over the A2A dimension space, the second axis is reduced). - #[test] - fn whole_rhs_arrayed_partial_reduce_is_its_own_agg() { - let project = TestProject::new("whole_rhs_partial") - .named_dimension("D1", &["a", "b"]) - .named_dimension("D2", &["x", "y"]) - .array_aux_direct("matrix", vec!["D1".into(), "D2".into()], "1", None) - .array_aux_direct("agg", vec!["D1".into()], "SUM(matrix[D1, *])", None); - - let result = agg_nodes(&project); - - assert!( - result.aggs.iter().all(|a| !a.is_synthetic), - "whole-RHS arrayed reducer must not mint a synthetic agg; got: {:?}", - result.aggs - ); - let agg = result - .aggs_in_var("agg") - .next() - .expect("expected an agg owned by `agg`"); - assert_eq!(agg.name, "agg"); - assert!(!agg.is_synthetic); - assert_eq!(source_names(agg), vec!["matrix"]); - assert_eq!(agg.result_dims, vec!["D1".to_string()]); - assert_eq!( - agg.canonical_read_slice(), - vec![ - AxisRead::Iterated { - dim: "d1".to_string(), - source_dim: "d1".to_string() - }, - AxisRead::Reduced { subset: None } - ] - ); - } - - /// AC4.3 (arrayed full-reduce broadcast): `share[Region] = SUM(pop[*])` is - /// a whole-RHS reducer, so the variable is the agg -- but `SUM(pop[*])` is a - /// *full* reduce (scalar result) merely broadcast to `[Region]`, so the - /// agg's `result_dims` is `[]`, not `[Region]`. (Contrast with - /// `agg[D1] = SUM(matrix[D1, *])`, a partial reduce that genuinely varies - /// per `D1`.) - #[test] - fn whole_rhs_arrayed_full_reduce_broadcast_has_scalar_result_dims() { - let project = TestProject::new("whole_rhs_broadcast") - .named_dimension("Region", &["NYC", "Boston"]) - .array_aux("pop[Region]", "100") - .array_aux("share[Region]", "SUM(pop[*])"); - - let result = agg_nodes(&project); - - assert!( - result.aggs.iter().all(|a| !a.is_synthetic), - "whole-RHS reducer must not mint a synthetic agg; got: {:?}", - result.aggs - ); - let agg = result - .aggs_in_var("share") - .next() - .expect("expected an agg owned by `share`"); - assert_eq!(agg.name, "share"); - assert!(!agg.is_synthetic); - assert_eq!(source_names(agg), vec!["pop"]); - assert!( - agg.result_dims.is_empty(), - "a full reduce broadcast to an arrayed variable has scalar result dims, got: {:?}", - agg.result_dims - ); - } - - /// AC4.1 (the basic mint): `share[r] = pop[r] / SUM(pop[*])` mints one - /// synthetic agg `$⁚ltm⁚agg⁚0` for the sub-expression `SUM(pop[*])`. - #[test] - fn subexpression_reducer_mints_one_synthetic_agg() { - let project = TestProject::new("share_mint") - .named_dimension("Region", &["NYC", "Boston", "LA"]) - .array_aux("pop[Region]", "100") - .array_aux("share[Region]", "pop / SUM(pop[*])"); - - let result = agg_nodes(&project); - - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!( - synthetic.len(), - 1, - "expected exactly one synthetic agg; got: {:?}", - result.aggs - ); - assert_eq!(synthetic[0].name, "$\u{205A}ltm\u{205A}agg\u{205A}0"); - assert_eq!(synthetic[0].equation_text, "sum(pop[*])"); - assert_eq!(source_names(synthetic[0]), vec!["pop"]); - assert!(synthetic[0].result_dims.is_empty()); - assert!( - result - .aggs_in_var("share") - .any(|a| a.name == "$\u{205A}ltm\u{205A}agg\u{205A}0") - ); - } - - /// P2 regression: an inline reducer (`share[r] = pop[r] / SUM(pop[*])`, - /// which must mint a *synthetic* agg) sharing canonical text with a - /// *whole-RHS* reducer of the same shape (`denom = SUM(pop[*])`, which - /// is *variable-backed*) must NOT reuse the variable-backed agg -- - /// regardless of declaration order. Dedup-by-key applies to synthetic - /// aggs only; variable-backed aggs are never deduped (a whole-RHS - /// reducer variable is its own distinct agg node). Before the fix, with - /// `denom` visited first (canonical-sorted: `denom` < `share`), the - /// inline use found `by_key["sum(pop[*])"]` already populated by `denom` - /// and reused it, so `share` got no synthetic agg and its reducer fell - /// back to the conservative direct path. - #[test] - fn inline_reducer_does_not_reuse_variable_backed_agg() { - let project = TestProject::new("inline_vs_var_backed") - .named_dimension("Region", &["NYC", "Boston"]) - .array_aux("pop[Region]", "100") - // `denom` (canonical-sorted first) is a whole-RHS reducer -> - // variable-backed agg named `denom`. - .scalar_aux("denom", "SUM(pop[*])") - // `share` (visited after `denom`) uses the same reducer text as - // a sub-expression -> must mint its own synthetic agg. - .array_aux("share[Region]", "pop / SUM(pop[*])"); - - let result = agg_nodes(&project); - - // The variable-backed agg `denom` exists and is not synthetic. - // (`agg_for_key` now resolves only synthetic aggs, so look up the - // variable-backed one through `by_var` instead.) - let denom_agg = result - .aggs_in_var("denom") - .find(|a| a.name == "denom") - .expect("expected a variable-backed agg owned by `denom`"); - assert!( - !denom_agg.is_synthetic, - "`denom`'s agg must be variable-backed" - ); - assert_eq!(denom_agg.equation_text, "sum(pop[*])"); - - // `share` must own a *synthetic* agg with the same reducer text. - let share_agg = result - .aggs_in_var("share") - .find(|a| a.is_synthetic) - .expect("expected a synthetic agg owned by `share`"); - assert_eq!(share_agg.name, "$\u{205A}ltm\u{205A}agg\u{205A}0"); - assert_eq!(share_agg.equation_text, "sum(pop[*])"); - assert_eq!(source_names(share_agg), vec!["pop"]); - // `agg_for_key` resolves the reducer text to the *synthetic* agg. - assert_eq!( - result.agg_for_key("sum(pop[*])").map(|a| a.name.as_str()), - Some("$\u{205A}ltm\u{205A}agg\u{205A}0") - ); - - // There must be exactly one synthetic agg and exactly one - // variable-backed agg -- two distinct nodes despite identical text. - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - let var_backed_aggs: Vec<&AggNode> = - result.aggs.iter().filter(|a| !a.is_synthetic).collect(); - assert_eq!( - synthetic.len(), - 1, - "expected one synthetic agg, got: {:?}", - result.aggs - ); - assert_eq!( - var_backed_aggs.len(), - 1, - "expected one variable-backed agg, got: {:?}", - result.aggs - ); - } - - /// P2 regression (reverse declaration order): the same model as - /// `inline_reducer_does_not_reuse_variable_backed_agg` but built so that - /// the inline-use variable would be visited first if order mattered. - /// `enumerate_agg_nodes` visits variables in canonical-sorted order, so - /// `denom` < `share` always; this test instead uses different names - /// (`a_share` < `z_denom`) to confirm the synthetic agg is minted when - /// the inline use is encountered *before* the whole-RHS reducer. - #[test] - fn inline_reducer_mints_synthetic_when_visited_before_variable_backed() { - let project = TestProject::new("inline_first") - .named_dimension("Region", &["NYC", "Boston"]) - .array_aux("pop[Region]", "100") - // `a_share` (canonical-sorted first) uses the reducer inline. - .array_aux("a_share[Region]", "pop / SUM(pop[*])") - // `z_denom` (visited after) is the whole-RHS reducer. - .scalar_aux("z_denom", "SUM(pop[*])"); - - let result = agg_nodes(&project); - - let share_agg = result - .aggs_in_var("a_share") - .find(|a| a.is_synthetic) - .expect("expected a synthetic agg owned by `a_share`"); - assert_eq!(share_agg.name, "$\u{205A}ltm\u{205A}agg\u{205A}0"); - assert_eq!(share_agg.equation_text, "sum(pop[*])"); - - let denom_agg = result - .aggs_in_var("z_denom") - .find(|a| a.name == "z_denom") - .expect("expected a variable-backed agg owned by `z_denom`"); - assert!(!denom_agg.is_synthetic); - - assert_eq!(result.aggs.iter().filter(|a| a.is_synthetic).count(), 1); - assert_eq!(result.aggs.iter().filter(|a| !a.is_synthetic).count(), 1); - } - - /// Two whole-RHS reducers with *identical* canonical text are two - /// distinct variable-backed agg nodes (one per variable) -- never - /// deduped, because each variable genuinely is its own aggregate. - #[test] - fn two_whole_rhs_reducers_same_text_are_distinct_aggs() { - let project = TestProject::new("two_var_backed") - .named_dimension("Region", &["NYC", "Boston"]) - .array_aux("pop[Region]", "100") - .scalar_aux("total_a", "SUM(pop[*])") - .scalar_aux("total_b", "SUM(pop[*])"); - - let result = agg_nodes(&project); - - let var_backed: Vec<&AggNode> = result.aggs.iter().filter(|a| !a.is_synthetic).collect(); - assert_eq!( - var_backed.len(), - 2, - "two whole-RHS reducers must be two distinct variable-backed aggs; got: {:?}", - result.aggs - ); - let names: std::collections::HashSet<&str> = - var_backed.iter().map(|a| a.name.as_str()).collect(); - assert!(names.contains("total_a"), "missing total_a: {names:?}"); - assert!(names.contains("total_b"), "missing total_b: {names:?}"); - // No synthetic aggs (neither reducer is a sub-expression). - assert_eq!(result.aggs.iter().filter(|a| a.is_synthetic).count(), 0); - } - - /// Two *inline* uses of the same reducer text still dedupe to one - /// synthetic agg (the synthetic dedup-by-key path is preserved). - #[test] - fn two_inline_uses_same_text_dedupe_to_one_synthetic() { - let project = TestProject::new("two_inline") - .named_dimension("Region", &["NYC", "Boston"]) - .array_aux("pop[Region]", "100") - .array_aux("share_a[Region]", "pop / SUM(pop[*])") - .array_aux("share_b[Region]", "pop * 2 / SUM(pop[*])"); - - let result = agg_nodes(&project); - - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!( - synthetic.len(), - 1, - "two inline uses of the same reducer must dedupe to one synthetic agg; got: {:?}", - result.aggs - ); - assert_eq!(synthetic[0].name, "$\u{205A}ltm\u{205A}agg\u{205A}0"); - // Both variables reference the same deduped synthetic agg index. - let a_idx = result.by_var.get("share_a").cloned().unwrap_or_default(); - let b_idx = result.by_var.get("share_b").cloned().unwrap_or_default(); - assert_eq!(a_idx, b_idx); - } - - /// AC4.4 (nested reducers): `x = SUM(a[*]) / SUM(b[*])` mints two distinct - /// synthetic agg nodes (`$⁚ltm⁚agg⁚0` for `SUM(a[*])`, `$⁚ltm⁚agg⁚1` for - /// `SUM(b[*])`). The `/` is not a reducer; neither `SUM` is inside the - /// other, so both are maximal. - #[test] - fn nested_reducers_mint_two_aggs() { - let project = TestProject::new("nested") - .named_dimension("Region", &["NYC", "Boston"]) - .array_aux("a[Region]", "10") - .array_aux("b[Region]", "20") - .scalar_aux("x", "SUM(a[*]) / SUM(b[*])"); - - let result = agg_nodes(&project); - - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!( - synthetic.len(), - 2, - "expected two synthetic aggs; got: {:?}", - result.aggs - ); - // First-encounter (left-to-right DFS) order: SUM(a[*]) then SUM(b[*]). - assert_eq!(synthetic[0].name, "$\u{205A}ltm\u{205A}agg\u{205A}0"); - assert_eq!(synthetic[0].equation_text, "sum(a[*])"); - assert_eq!(source_names(synthetic[0]), vec!["a"]); - assert_eq!(synthetic[1].name, "$\u{205A}ltm\u{205A}agg\u{205A}1"); - assert_eq!(synthetic[1].equation_text, "sum(b[*])"); - assert_eq!(source_names(synthetic[1]), vec!["b"]); - } - - /// AC4.4 (dedup): the same reducer subexpression appearing in two - /// variables' equations (with whitespace/casing differences in the - /// source text) maps to one synthetic agg node referenced by both. - #[test] - fn ast_identical_reducers_dedupe() { - let project = TestProject::new("dedup") - .named_dimension("Region", &["NYC", "Boston"]) - .array_aux("pop[Region]", "100") - // Two different equations both contain SUM(pop[*]); the first is - // spelled with extra spacing and uppercase. - .array_aux("share_a[Region]", "pop / SUM( POP [ * ] )") - .array_aux("share_b[Region]", "pop * 2 / sum(pop[*])"); - - let result = agg_nodes(&project); - - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!( - synthetic.len(), - 1, - "AST-identical reducers must dedupe to one agg; got: {:?}", - result.aggs - ); - assert_eq!(synthetic[0].equation_text, "sum(pop[*])"); - // Both variables reference the same agg index. - let a_idx: Vec = result.by_var.get("share_a").cloned().unwrap_or_default(); - let b_idx: Vec = result.by_var.get("share_b").cloned().unwrap_or_default(); - assert_eq!(a_idx.len(), 1); - assert_eq!(b_idx.len(), 1); - assert_eq!( - a_idx, b_idx, - "both variables must point at the same deduped agg index" - ); - } - - /// Per-element `Ast::Arrayed` target with a different reducer per element: - /// `x[a] = SUM(p[*]); x[b] = MEAN(p[*])` mints two synthetic agg nodes, - /// one per element's reducer. - #[test] - fn per_element_arrayed_target_mints_one_agg_per_element_reducer() { - let project = TestProject::new("per_elem") - .named_dimension("D", &["a", "b"]) - .array_aux("p[D]", "1") - .array_with_ranges_direct( - "x", - vec!["D".into()], - vec![("a", "SUM(p[*])"), ("b", "MEAN(p[*])")], - None, - ); - - let result = agg_nodes(&project); - - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!( - synthetic.len(), - 2, - "per-element reducers must mint one agg per element; got: {:?}", - result.aggs - ); - let texts: std::collections::HashSet<&str> = - synthetic.iter().map(|a| a.equation_text.as_str()).collect(); - assert!(texts.contains("sum(p[*])"), "missing sum(p[*]): {texts:?}"); - assert!( - texts.contains("mean(p[*])"), - "missing mean(p[*]): {texts:?}" - ); - // Both are owned by `x`. - let x_idx = result.by_var.get("x").cloned().unwrap_or_default(); - assert_eq!(x_idx.len(), 2); - } - - /// Determinism: the same model built twice (or with variables declared in - /// a different order) yields identical agg names assigned to the same - /// subexpressions. - #[test] - fn enumeration_is_deterministic_under_variable_reordering() { - // Two synthetic aggs: SUM(a[*]) and SUM(b[*]). Whichever variable - // happens to be visited first is irrelevant -- we always visit in - // canonical-name sorted order, and within an equation left-to-right. - let build = |order_a_first: bool| { - let mut p = TestProject::new("determinism") - .named_dimension("Region", &["NYC", "Boston"]) - .array_aux("a[Region]", "10") - .array_aux("b[Region]", "20"); - // `q` references SUM(a[*]) and SUM(b[*]); `r` references the same - // pair. We add them in different orders to confirm the result is - // identical. - if order_a_first { - p = p - .scalar_aux("q", "SUM(a[*]) + SUM(b[*])") - .scalar_aux("r", "SUM(a[*]) * SUM(b[*])"); - } else { - p = p - .scalar_aux("r", "SUM(a[*]) * SUM(b[*])") - .scalar_aux("q", "SUM(a[*]) + SUM(b[*])"); - } - agg_nodes(&p) - }; - - let r1 = build(true); - let r2 = build(false); - assert_eq!( - r1.aggs, r2.aggs, - "enumeration must be deterministic regardless of declaration order" - ); - assert_eq!(r1.synthetic_by_key, r2.synthetic_by_key); - // Specifically: SUM(a[*]) -> agg 0, SUM(b[*]) -> agg 1 (a < b, and - // within q's equation SUM(a[*]) precedes SUM(b[*])). - assert_eq!( - r1.agg_for_key("sum(a[*])").map(|a| a.name.clone()), - Some("$\u{205A}ltm\u{205A}agg\u{205A}0".to_string()) - ); - assert_eq!( - r1.agg_for_key("sum(b[*])").map(|a| a.name.clone()), - Some("$\u{205A}ltm\u{205A}agg\u{205A}1".to_string()) - ); - } - - /// A model with no reducers produces an empty result. - #[test] - fn model_without_reducers_has_no_aggs() { - let project = TestProject::new("no_reducers") - .stock("population", "100", &["births"], &["deaths"], None) - .flow("births", "population * 0.1", None) - .flow("deaths", "population * 0.05", None) - .scalar_const("rate", 0.1); - - let result = agg_nodes(&project); - assert!( - result.aggs.is_empty(), - "model without reducers must have no aggs; got: {:?}", - result.aggs - ); - assert!(result.synthetic_by_key.is_empty()); - assert!(result.by_var.is_empty()); - } - - /// A reducer over a *scalar* source is not hoisted (the parser would - /// normally reject it anyway, but be defensive). - #[test] - fn reducer_over_scalar_source_is_not_hoisted() { - // `SUM(s)` where `s` is scalar -- pathological, but must not mint an - // agg. (We also keep a real arrayed reducer to confirm the - // enumerator still finds the legitimate one.) - let project = TestProject::new("scalar_reducer") - .named_dimension("Region", &["NYC", "Boston"]) - .scalar_aux("s", "5") - .array_aux("pop[Region]", "100") - .scalar_aux("y", "SUM(s) + SUM(pop[*])"); - - let result = agg_nodes(&project); - // Only the arrayed reducer is recognized. - assert!( - result.agg_for_key("sum(pop[*])").is_some(), - "the arrayed reducer must be recognized; got: {:?}", - result.aggs - ); - assert!( - result.agg_for_key("sum(s)").is_none(), - "a reducer over a scalar source must not be hoisted; got: {:?}", - result.aggs - ); - } - - /// SIZE is not hoisted -- its link score is always 0, matching - /// `try_cross_dimensional_link_scores`'s `Some(vec![])` for SIZE. - #[test] - fn size_reducer_is_not_hoisted() { - let project = TestProject::new("size_reducer") - .named_dimension("Region", &["NYC", "Boston"]) - .array_aux("pop[Region]", "100") - .scalar_aux("n", "SIZE(pop[*])"); - - let result = agg_nodes(&project); - assert!( - result.aggs.is_empty(), - "SIZE must not be hoisted as an agg; got: {:?}", - result.aggs - ); - } - - /// AC4.1: a reducer over an explicit *slice* used as a sub-expression - /// (`x[r] = ... + SUM(pop[NYC, *])`) IS hoisted into a synthetic agg -- - /// the `read_slice` descriptor records which rows it reads - /// (`[Pinned(nyc), Reduced]` over `pop`'s `[Region, Age]` axes), so the - /// element-graph reroute and the per-element reducer link scores route - /// only those rows. `result_dims` is `[]` here: there is no `Iterated` - /// axis (the `Region` on the target `x` is broadcast; the read is a - /// single row). The `pop[NYC, Adult]` `Direct` reference is separate -- - /// not part of the agg. - #[test] - fn slice_reducer_subexpression_is_hoisted() { - let project = TestProject::new("slice_subexpr") - .named_dimension("Region", &["NYC", "Boston"]) - .named_dimension("Age", &["Adult", "Child"]) - .array_aux_direct("pop", vec!["Region".into(), "Age".into()], "10", None) - .array_aux_direct( - "x", - vec!["Region".into()], - "pop[NYC, Adult] + SUM(pop[NYC, *])", - None, - ); - - let result = agg_nodes(&project); - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!( - synthetic.len(), - 1, - "a slice-reducer subexpression must mint one synthetic agg; got: {:?}", - result.aggs - ); - assert_eq!(synthetic[0].name, "$\u{205A}ltm\u{205A}agg\u{205A}0"); - // `expr2_to_string` puts a space after the comma in a multi-index - // subscript -- assert the canonical text it actually produces. - assert_eq!(synthetic[0].equation_text, "sum(pop[nyc, *])"); - assert_eq!(source_names(synthetic[0]), vec!["pop"]); - assert_eq!( - synthetic[0].canonical_read_slice(), - vec![ - AxisRead::Pinned("nyc".to_string()), - AxisRead::Reduced { subset: None } - ] - ); - assert!( - synthetic[0].result_dims.is_empty(), - "no Iterated axis -- result dims must be empty; got: {:?}", - synthetic[0].result_dims - ); - assert!( - result - .aggs_in_var("x") - .any(|a| a.name == "$\u{205A}ltm\u{205A}agg\u{205A}0") - ); - } - - /// AC4.2: a *partial*-reduce slice over an iterated dimension used as a - /// sub-expression (`x[D1] = ... + SUM(matrix[D1, *])`, `matrix[D1, D2]`, - /// `x` A2A over `D1`) mints an arrayed synthetic agg over `D1`: - /// `read_slice = [Iterated(d1), Reduced]`, `result_dims = [D1]`. The - /// element graph routes `matrix[d1, d2] → agg[d1]`. - #[test] - fn sliced_reducer_over_iterated_dim_mints_arrayed_agg() { - let project = TestProject::new("iterated_slice_subexpr") - .named_dimension("D1", &["a", "b"]) - .named_dimension("D2", &["x", "y"]) - .array_aux_direct("matrix", vec!["D1".into(), "D2".into()], "1", None) - .array_aux_direct( - "x", - vec!["D1".into()], - "matrix[a, x] + SUM(matrix[D1, *])", - None, - ); - - let result = agg_nodes(&project); - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!( - synthetic.len(), - 1, - "an iterated-dim slice-reducer subexpression must mint one synthetic agg; got: {:?}", - result.aggs - ); - assert_eq!( - synthetic[0].canonical_read_slice(), - vec![ - AxisRead::Iterated { - dim: "d1".to_string(), - source_dim: "d1".to_string() - }, - AxisRead::Reduced { subset: None } - ] - ); - assert_eq!(synthetic[0].result_dims, vec!["D1".to_string()]); - assert_eq!(source_names(synthetic[0]), vec!["matrix"]); - // `expr2_to_string` canonicalizes the iterated dim name lowercase. - assert_eq!(synthetic[0].equation_text, "sum(matrix[d1, *])"); - } - - /// #514: a *mixed* read slice -- `Iterated` + `Pinned` + `Reduced` axes - /// on one source. `matrix3d[D1, Region, Age]`, `x` A2A over `D1`, - /// `x[D1] = ... + SUM(matrix3d[D1, NYC, *])`: the first axis is iterated - /// over the target's `D1`, the second is pinned to the literal `NYC`, - /// the third (wildcard) is reduced ⇒ `read_slice = [Iterated(d1), - /// Pinned(nyc), Reduced]`, `result_dims = [D1]` (only the iterated axis - /// shapes the agg). Mints one arrayed synthetic agg over `D1`. - #[test] - fn mixed_pinned_iterated_reduced_slice_mints_arrayed_agg() { - let project = TestProject::new("mixed_slice_subexpr") - .named_dimension("D1", &["a", "b"]) - .named_dimension("Region", &["NYC", "Boston"]) - .named_dimension("Age", &["Adult", "Child"]) - .array_aux_direct( - "matrix3d", - vec!["D1".into(), "Region".into(), "Age".into()], - "1", - None, - ) - .array_aux_direct( - "x", - vec!["D1".into()], - "matrix3d[a, NYC, Adult] + SUM(matrix3d[D1, NYC, *])", - None, - ); - - let result = agg_nodes(&project); - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!( - synthetic.len(), - 1, - "a mixed pinned/iterated/reduced slice must mint one synthetic agg; got: {:?}", - result.aggs - ); - assert_eq!( - synthetic[0].canonical_read_slice(), - vec![ - AxisRead::Iterated { - dim: "d1".to_string(), - source_dim: "d1".to_string() - }, - AxisRead::Pinned("nyc".to_string()), - AxisRead::Reduced { subset: None } - ] - ); - assert_eq!(synthetic[0].result_dims, vec!["D1".to_string()]); - assert_eq!(source_names(synthetic[0]), vec!["matrix3d"]); - assert_eq!(synthetic[0].equation_text, "sum(matrix3d[d1, nyc, *])"); - } - - /// #514: a multi-source reducer whose arrayed args agree on their read - /// slice -- `total = 1 + SUM(a[*] + b[*])`, `a`, `b` both over `D`. The - /// reducer's argument expression references two arrayed sources; each - /// reads its whole extent (`[Reduced]`), the slices agree, so one - /// synthetic agg is minted carrying that combined slice and *both* source - /// variables. - #[test] - fn multi_source_reducer_agreeing_slices_mints_one_agg() { - let project = TestProject::new("multi_source_reducer") - .named_dimension("D", &["p", "q"]) - .array_aux_direct("a", vec!["D".into()], "1", None) - .array_aux_direct("b", vec!["D".into()], "2", None) - .scalar_aux("total", "1 + SUM(a[*] + b[*])"); - - let result = agg_nodes(&project); - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!( - synthetic.len(), - 1, - "a multi-source reducer with agreeing slices must mint one synthetic agg; got: {:?}", - result.aggs - ); - assert_eq!( - synthetic[0].canonical_read_slice(), - vec![AxisRead::Reduced { subset: None }] - ); - assert!(synthetic[0].result_dims.is_empty()); - // `sources` lists every arrayed model variable in the argument - // (sorted by name), each carrying the IDENTICAL canonical slice -- - // invariant I1's identical-co-source form (T2 of the - // shape-expressiveness design: acceptance is identical-only, so - // per-source slices cannot yet differ). - assert_eq!(source_names(synthetic[0]), vec!["a", "b"]); - for s in &synthetic[0].sources { - assert_eq!( - s.read_slice, - vec![AxisRead::Reduced { subset: None }], - "every arrayed co-source must carry the canonical slice; got {:?} for {}", - s.read_slice, - s.var - ); - } - } - - /// #514 (negative guard): a multi-source reducer whose arrayed args read - /// *incompatible* slices is NOT hoisted -- `total = 1 + SUM(a[*] + b[*])` - /// where `a` is over `D1` and `b` is over `D2` (disjoint dims, so - /// `[Reduced]` for `a`'s one axis vs `[Reduced]` for `b`'s -- the slices - /// have the same shape but the *sources* differ in dimensionality; more - /// to the point, a 1-axis `a[*]` and a 2-axis `b[*, *]` disagree on - /// length). Use clearly-different ranks to force the disagreement: `a` - /// over `D1`, `b` over `D1 x D2`. `combined_read_slice` returns `None`, - /// so no agg is minted for this reducer. - #[test] - fn multi_source_reducer_disagreeing_slices_is_not_hoisted() { - let project = TestProject::new("multi_source_disagree") - .named_dimension("D1", &["p", "q"]) - .named_dimension("D2", &["x", "y"]) - .array_aux_direct("a", vec!["D1".into()], "1", None) - .array_aux_direct("b", vec!["D1".into(), "D2".into()], "2", None) - .scalar_aux("total", "1 + SUM(a[*] + b[*, *])"); - - let result = agg_nodes(&project); - assert!( - result - .aggs - .iter() - .all(|ag| !ag.reads_var("a") && !ag.reads_var("b")), - "a multi-source reducer whose args read incompatible slices must not be hoisted; \ - got: {:?}", - result.aggs - ); - assert!(result.synthetic_by_key.is_empty()); - } - - /// GH #534: a sliced reducer whose iterated index lines up with the - /// source's row axis via a *positional dimension mapping* - /// (`matrix[Region, D2]`, `State` over `{s1, s2}` with a `State→Region` - /// mapping, target A2A over `State` with `... + SUM(matrix[State, *])`) - /// IS hoisted: the `Iterated` axis carries the (target, source) dim - /// pair, `result_dims` is the TARGET's iterated dim (`State` -- the - /// dimension the agg variable is arrayed over), and the emitters remap - /// each source row to its positionally-corresponding slot. - /// (`classify_iterated_dim_shape`'s own mapped branch -- a - /// whole-equation-iterated subscript, not a sliced reducer argument -- - /// is a separate path and stays `Bare`; see - /// `db::ltm_ir_tests::ir_mapped_iterated_dim_subscript_is_bare`.) - #[test] - fn mapped_iterated_dim_sliced_reducer_is_hoisted_with_pair() { - let project = TestProject::new("mapped_iterated_slice") - .named_dimension("Region", &["r1", "r2"]) - .named_dimension("D2", &["x", "y"]) - .named_dimension_with_mapping("State", &["s1", "s2"], "Region") - .array_aux_direct("matrix", vec!["Region".into(), "D2".into()], "1", None) - .array_aux_direct( - "out", - vec!["State".into()], - "matrix[r1, x] + SUM(matrix[State, *])", - None, - ); - - let result = agg_nodes(&project); - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!( - synthetic.len(), - 1, - "a positionally-mapped sliced reducer must mint one synthetic agg; got: {:?}", - result.aggs - ); - assert_eq!( - synthetic[0].canonical_read_slice(), - vec![ - AxisRead::Iterated { - dim: "state".to_string(), - source_dim: "region".to_string() - }, - AxisRead::Reduced { subset: None } - ] - ); - assert_eq!( - synthetic[0].result_dims, - vec!["State".to_string()], - "the agg's result axis is the TARGET equation's iterated dim" - ); - assert_eq!(source_names(synthetic[0]), vec!["matrix"]); - assert_eq!(synthetic[0].equation_text, "sum(matrix[state, *])"); - } - - /// GH #534 (conservative gate, element-mapped): a sliced reducer over an - /// EXPLICIT element-mapped pair stays un-hoisted -- the executed A2A - /// lowering resolves mapped references positionally and ignores the - /// element map (GH #756), so `mapped_element_correspondence` declines - /// and the reference keeps its conservative shape. - #[test] - fn element_mapped_sliced_reducer_is_not_hoisted() { - let project = TestProject::new("element_mapped_slice") - .named_dimension("Region", &["r1", "r2"]) - .named_dimension("D2", &["x", "y"]) - .named_dimension_with_element_mapping( - "State", - &["s1", "s2"], - "Region", - &[("s1", "r2"), ("s2", "r1")], - ) - .array_aux_direct("matrix", vec!["Region".into(), "D2".into()], "1", None) - .array_aux_direct( - "out", - vec!["State".into()], - "1 + SUM(matrix[State, *])", - None, - ); - - let result = agg_nodes(&project); - assert!( - result.aggs.iter().all(|a| !a.reads_var("matrix")), - "an element-mapped sliced reducer must not be hoisted; got: {:?}", - result.aggs - ); - assert!(result.synthetic_by_key.is_empty()); - } - - /// GH #757 (flipped from the GH #534-era conservative pin): a sliced - /// reducer whose POSITIONAL mapping is declared only in the REVERSE - /// direction (on the source's `Region` toward `State`) is now hoisted -- - /// `classify_axis_access`'s mapped arm gates on - /// `iterated_axis_slot_elements` / `mapped_element_correspondence`, - /// which accepts both declaration directions (the compiler's - /// `translate_via_mapping` resolves both, so declining one direction - /// was pure over-conservatism). The slice and `result_dims` are - /// identical to the forward-declared twin. - #[test] - fn reverse_declared_mapped_sliced_reducer_is_hoisted() { - let project = TestProject::new("reverse_mapped_slice") - .named_dimension_with_mapping("Region", &["r1", "r2"], "State") - .named_dimension("D2", &["x", "y"]) - .named_dimension("State", &["s1", "s2"]) - .array_aux_direct("matrix", vec!["Region".into(), "D2".into()], "1", None) - .array_aux_direct( - "out", - vec!["State".into()], - "1 + SUM(matrix[State, *])", - None, - ); - - let result = agg_nodes(&project); - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!( - synthetic.len(), - 1, - "the reverse-declared positionally-mapped sliced reducer must be hoisted; got: {:?}", - result.aggs - ); - assert_eq!( - synthetic[0].canonical_read_slice(), - vec![ - AxisRead::Iterated { - dim: "state".to_string(), - source_dim: "region".to_string() - }, - AxisRead::Reduced { subset: None } - ] - ); - assert_eq!(synthetic[0].result_dims, vec!["State".to_string()]); - } - - /// GH #534: the whole-RHS twin -- `out[State] = SUM(matrix[State,*])` - /// over a positionally-mapped pair mints a SYNTHETIC agg, an exception - /// to the variable-is-the-agg rule for whole-RHS reducers: the - /// variable-backed link-score path (`try_cross_dimensional_link_scores`) - /// matches result axes against source axes by name, so a remapped pair - /// falls off it onto the `Wildcard` per-shape partial, whose - /// PREVIOUS-wrapping mangles the iterated index into a non-compiling - /// `matrix[PREVIOUS(state),*]` (silently stubbed to 0). The synthetic - /// agg gives the whole-RHS case the same remapped two-half scoring as - /// an inline mapped reducer. - #[test] - fn whole_rhs_mapped_partial_reduce_mints_synthetic_agg() { - let project = TestProject::new("whole_rhs_mapped") - .named_dimension("Region", &["r1", "r2"]) - .named_dimension("D2", &["x", "y"]) - .named_dimension_with_mapping("State", &["s1", "s2"], "Region") - .array_aux_direct("matrix", vec!["Region".into(), "D2".into()], "1", None) - .array_aux_direct("out", vec!["State".into()], "SUM(matrix[State, *])", None); - - let result = agg_nodes(&project); - assert!( - result.aggs.iter().all(|a| a.is_synthetic), - "a whole-RHS MAPPED reducer must mint a synthetic agg (not variable-backed); got: {:?}", - result.aggs - ); - let agg = result - .aggs_in_var("out") - .next() - .expect("expected a synthetic agg owned by `out`"); - assert_eq!(agg.name, "$\u{205A}ltm\u{205A}agg\u{205A}0"); - assert_eq!( - agg.canonical_read_slice(), - vec![ - AxisRead::Iterated { - dim: "state".to_string(), - source_dim: "region".to_string() - }, - AxisRead::Reduced { subset: None } - ] - ); - assert_eq!(agg.result_dims, vec!["State".to_string()]); - } - - /// GH #764 (T4): the whole-RHS BROADCAST twin -- `out[D1,D3] = - /// SUM(matrix[D1,*])`, `result_dims` (`[D1]`) a strict subset of the - /// owner's dims (`[D1,D3]`) -- mints a SYNTHETIC agg, generalizing the - /// GH #534 carve-out: the variable-backed per-`(row, slot)` machinery - /// requires each slot to name a complete `to` element, which a - /// broadcast slot does not. The synthetic agg is arrayed over - /// `result_dims` and rides the two-half emitters + the GH #528 - /// agg-to-target projection. - #[test] - fn whole_rhs_broadcast_partial_reduce_mints_synthetic_agg() { - let project = TestProject::new("whole_rhs_broadcast_764") - .named_dimension("D1", &["a", "b"]) - .named_dimension("D2", &["x", "y"]) - .named_dimension("D3", &["p", "q"]) - .array_aux_direct("matrix", vec!["D1".into(), "D2".into()], "1", None) - .array_aux_direct( - "out", - vec!["D1".into(), "D3".into()], - "SUM(matrix[D1, *])", - None, - ); - - let result = agg_nodes(&project); - assert!( - result.aggs.iter().all(|a| a.is_synthetic), - "a whole-RHS BROADCAST reducer must mint a synthetic agg (not variable-backed); \ - got: {:?}", - result.aggs - ); - let agg = result - .aggs_in_var("out") - .next() - .expect("expected a synthetic agg owned by `out`"); - assert_eq!(agg.name, "$\u{205A}ltm\u{205A}agg\u{205A}0"); - assert_eq!( - agg.canonical_read_slice(), - vec![ - AxisRead::Iterated { - dim: "d1".to_string(), - source_dim: "d1".to_string() - }, - AxisRead::Reduced { subset: None } - ] - ); - assert_eq!(agg.result_dims, vec!["D1".to_string()]); - assert_eq!(source_names(agg), vec!["matrix"]); - } - - /// GH #764 (T4): the whole-RHS PERMUTED twin -- `out[D2,D1] = - /// SUM(cube[D1,D2,*])`, `result_dims` (`[D1,D2]`, slice order) in a - /// different order than the owner's dims (`[D2,D1]`) -- mints a - /// SYNTHETIC agg too: variable-backed slot coordinates are in - /// `Iterated`-axis order, which would mis-subscript `to`. Slots of the - /// synthetic agg are keyed by `result_dims` order, and the GH #528 - /// projection reorders per target element. - #[test] - fn whole_rhs_permuted_partial_reduce_mints_synthetic_agg() { - let project = TestProject::new("whole_rhs_permuted_764") - .named_dimension("D1", &["a", "b"]) - .named_dimension("D2", &["x", "y"]) - .named_dimension("D3", &["p", "q"]) - .array_aux_direct( - "cube", - vec!["D1".into(), "D2".into(), "D3".into()], - "1", - None, - ) - .array_aux_direct( - "out", - vec!["D2".into(), "D1".into()], - "SUM(cube[D1, D2, *])", - None, - ); - - let result = agg_nodes(&project); - assert!( - result.aggs.iter().all(|a| a.is_synthetic), - "a whole-RHS PERMUTED reducer must mint a synthetic agg (not variable-backed); \ - got: {:?}", - result.aggs - ); - let agg = result - .aggs_in_var("out") - .next() - .expect("expected a synthetic agg owned by `out`"); - assert_eq!( - agg.result_dims, - vec!["D1".to_string(), "D2".to_string()], - "result_dims stay in Iterated-axis (slice) order, not the owner's declared order" - ); - assert_eq!( - agg.canonical_read_slice(), - vec![ - AxisRead::Iterated { - dim: "d1".to_string(), - source_dim: "d1".to_string() - }, - AxisRead::Iterated { - dim: "d2".to_string(), - source_dim: "d2".to_string() - }, - AxisRead::Reduced { subset: None } - ] - ); - } - - /// GH #764 ∩ GH #765 (T4): a non-aligned whole-RHS reduce that ALSO - /// carries a `Pinned` axis (`out[D1,D2] = SUM(cube[D1,nyc,*])` over - /// `cube[D1,Region,D2]`) mints a synthetic agg whose slice keeps the - /// `Pinned` axis -- so the synthetic-half emitters (which are - /// Pinned-correct via `read_slice_rows`) score only the read rows. - /// Pre-T4 this shape rode the OLD full-cartesian link-score - /// derivation, scoring unread (`boston`) rows. - #[test] - fn whole_rhs_broadcast_pinned_mix_mints_synthetic_agg() { - let project = TestProject::new("whole_rhs_broadcast_pinned_764") - .named_dimension("D1", &["a", "b"]) - .named_dimension("Region", &["nyc", "boston"]) - .named_dimension("D2", &["x", "y"]) - .array_aux_direct( - "cube", - vec!["D1".into(), "Region".into(), "D2".into()], - "1", - None, - ) - .array_aux_direct( - "out", - vec!["D1".into(), "D2".into()], - "SUM(cube[D1, nyc, *])", - None, - ); - - let result = agg_nodes(&project); - assert!( - result.aggs.iter().all(|a| a.is_synthetic), - "a Pinned-bearing non-aligned whole-RHS reducer must mint a synthetic agg; got: {:?}", - result.aggs - ); - let agg = result - .aggs_in_var("out") - .next() - .expect("expected a synthetic agg owned by `out`"); - assert_eq!(agg.result_dims, vec!["D1".to_string()]); - assert_eq!( - agg.canonical_read_slice(), - vec![ - AxisRead::Iterated { - dim: "d1".to_string(), - source_dim: "d1".to_string() - }, - AxisRead::Pinned("nyc".to_string()), - AxisRead::Reduced { subset: None } - ] - ); - } - - /// GH #534: `iterated_axis_slot_elements` -- identity for the literal - /// case, the positional preimage for a mapped pair, `None` for an - /// element-mapped or unmapped pair. - #[test] - fn iterated_axis_slot_elements_cases() { - use crate::datamodel::{Dimension as DmDimension, DimensionMapping}; - use crate::dimensions::DimensionsContext; - - let named = |name: &str, elems: &[&str], mappings: Vec| { - let mut d = DmDimension::named( - name.to_string(), - elems.iter().map(|e| e.to_string()).collect(), - ); - d.mappings = mappings; - d - }; - let positional = DimensionMapping { - target: "Region".to_string(), - element_map: vec![], - }; - let element_mapped = DimensionMapping { - target: "Region".to_string(), - element_map: vec![ - ("s1".to_string(), "r2".to_string()), - ("s2".to_string(), "r1".to_string()), - ], - }; - - let region_elems = vec!["r1".to_string(), "r2".to_string()]; - - // Literal: identity (no dim_ctx lookups consulted). - let ctx = DimensionsContext::from(&[ - named("Region", &["r1", "r2"], vec![]), - named("State", &["s1", "s2"], vec![positional.clone()]), - ]); - assert_eq!( - iterated_axis_slot_elements("region", "region", ®ion_elems, &ctx), - Some(region_elems.clone()) - ); - - // Positional mapping: source row r1 feeds slot s1, r2 feeds s2 - // (index-identity under the positional correspondence). - assert_eq!( - iterated_axis_slot_elements("state", "region", ®ion_elems, &ctx), - Some(vec!["s1".to_string(), "s2".to_string()]) - ); - - // Explicit element map: declined (GH #756 positional-only gate). - let ctx_elem = DimensionsContext::from(&[ - named("Region", &["r1", "r2"], vec![]), - named("State", &["s1", "s2"], vec![element_mapped]), - ]); - assert_eq!( - iterated_axis_slot_elements("state", "region", ®ion_elems, &ctx_elem), - None - ); - - // Unmapped pair: declined. - let ctx_unmapped = DimensionsContext::from(&[ - named("Region", &["r1", "r2"], vec![]), - named("State", &["s1", "s2"], vec![]), - ]); - assert_eq!( - iterated_axis_slot_elements("state", "region", ®ion_elems, &ctx_unmapped), - None - ); - } - - /// AC4.4 (the carve-out): a reducer over a *dynamic* index - /// (`x[Region] = SUM(pop[idx, *])`, `idx` a scalar aux -- a non-literal - /// index) is NOT statically describable: `compute_read_slice` returns - /// `None` for the `idx` axis, so the reducer is not hoisted and its - /// reference stays on the conservative path. Pin this narrow case. - #[test] - fn dynamic_index_reducer_subexpression_is_not_hoisted() { - let project = TestProject::new("dynamic_index_reducer") - .named_dimension("Region", &["NYC", "Boston"]) - .named_dimension("Age", &["Adult", "Child"]) - .array_aux_direct("pop", vec!["Region".into(), "Age".into()], "10", None) - .scalar_aux("idx", "1") - .array_aux_direct("x", vec!["Region".into()], "SUM(pop[idx, *])", None); - - let result = agg_nodes(&project); - assert!( - result.aggs.iter().all(|a| !a.reads_var("pop")), - "a dynamic-index reducer must not be hoisted; got: {:?}", - result.aggs - ); - assert!(result.synthetic_by_key.is_empty()); - } - - /// AC4.2 (positive guard): a whole-RHS slice/partial reduce - /// (`agg[D1] = SUM(matrix[D1, *])`) IS recognized -- but as a - /// variable-backed agg, not a synthetic one (covered by - /// `whole_rhs_arrayed_partial_reduce_is_its_own_agg`); and an all- - /// wildcard reducer subexpression (`SUM(matrix[*, *])`, no literal pin) - /// is still hoisted as a synthetic agg with an all-`Reduced` slice. - #[test] - fn full_wildcard_reducer_subexpression_is_still_hoisted() { - // `SUM(matrix[*, *])` (all-wildcard, no literal pin) is a full - // reduce and IS hoistable as a synthetic agg. - let project = TestProject::new("full_wildcard_subexpr") - .named_dimension("D1", &["a", "b"]) - .named_dimension("D2", &["x", "y"]) - .array_aux_direct("matrix", vec!["D1".into(), "D2".into()], "1", None) - .scalar_aux("y", "5 + SUM(matrix[*, *])"); - - let result = agg_nodes(&project); - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!( - synthetic.len(), - 1, - "an all-wildcard reducer subexpression must mint one synthetic agg; got: {:?}", - result.aggs - ); - assert_eq!(source_names(synthetic[0]), vec!["matrix"]); - assert_eq!( - synthetic[0].canonical_read_slice(), - vec![ - AxisRead::Reduced { subset: None }, - AxisRead::Reduced { subset: None } - ] - ); - assert!(synthetic[0].result_dims.is_empty()); - } - - /// AC1.2 + GH #982: the consolidated reducer table classifies every array - /// reducer the LTM machinery cares about, and all THREE derived predicates - /// agree with the table row for row. - /// - /// The rows come from [`REDUCER_DECISION_TABLE`], whose 11 entries are - /// derived from `reducer_kind_from_name`'s own arms (see its doc): every - /// arm, every arity guard in both directions, and the catch-all. Each row - /// asserts the name-keyed decider, the builtin-keyed decider, the arity - /// the builtin form reports, and the three consumers -- so the `SIZE` / - /// `RANK` inversion between `reducer_collapses_to_scalar` ("does this - /// collapse to a scalar?") and `builtin_routes_through_agg` ("did an agg - /// get minted for it?") is pinned in both directions on both rows, and an - /// edit to either predicate reds here rather than drifting silently - /// (GH #982). - #[test] - fn reducer_kind_classifies_every_array_reducer() { - for row in REDUCER_DECISION_TABLE { - let name = row.name; - let arity = row.arity; - let builtin = row.builtin(); - assert_eq!( - builtin_reducer_arity(&builtin), - arity, - "{name}/{arity}: the builtin form must report the row's arity" - ); - assert_eq!( - reducer_kind_from_name(name, arity), - row.kind, - "{name}/{arity}: reducer_kind_from_name" - ); - assert_eq!( - reducer_kind(&builtin), - row.kind, - "{name}/{arity}: reducer_kind" - ); - assert_eq!( - reducer_collapses_to_scalar(name, arity), - row.collapses_to_scalar, - "{name}/{arity}: reducer_collapses_to_scalar" - ); - assert_eq!( - reducer_is_hoistable(&builtin), - row.is_hoistable, - "{name}/{arity}: reducer_is_hoistable" - ); - assert_eq!( - builtin_routes_through_agg(&builtin), - row.routes_through_agg, - "{name}/{arity}: builtin_routes_through_agg" - ); - } - - // The table keys `stddev`/`rank`/`size`/`sum` at one arity each, but - // `reducer_kind_from_name` ignores arity for them -- only - // `mean`/`min`/`max` carry an `arity == 1` guard. Spot-check one, so a - // stray arity guard added to an arity-insensitive arm is caught. - assert_eq!( - reducer_kind_from_name("stddev", 7), - Some(ReducerKind::Nonlinear) - ); - assert_eq!( - reducer_kind_from_name("rank", 2), - Some(ReducerKind::Nonlinear) - ); - } - - /// GH #776: a RANK subexpression mints an ARRAY-valued synthetic agg, - /// not a scalar reducer agg. Its result dims are the ranked argument's - /// non-pinned axes, so a bare one-dimensional source ranks over Region. - #[test] - fn rank_subexpression_mints_array_valued_synthetic_agg() { - let project = TestProject::new("rank_array_agg") - .named_dimension("Region", &["north", "south"]) - .array_aux("pop[Region]", "100") - .array_aux("scale[Region]", "pop[Region] * 0.01") - .array_aux("grow[Region]", "scale[Region] * RANK(pop, 1)"); - - let result = agg_nodes(&project); - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!( - synthetic.len(), - 1, - "RANK must mint one synthetic aggregate node; got: {:?}", - result.aggs - ); - assert!(synthetic[0].array_valued_rank); - assert_eq!(synthetic[0].result_dims, vec!["Region"]); - assert_eq!(source_names(synthetic[0]), vec!["pop"]); - } - - /// GH #796 review: multi-source RANK result dims must not depend on - /// `HashMap` iteration order. When several sources share the canonical - /// rank slice but carry differently named mapped axes, choose the first - /// source in canonical source-name order -- the same order `AggSource` - /// emission uses. - #[test] - fn rank_multi_source_result_dims_use_sorted_source_order() { - let project = TestProject::new("rank_multi_source_dim_order") - .named_dimension("Region", &["r1", "r2"]) - .named_dimension_with_mapping("State", &["s1", "s2"], "Region") - // Declare `b` first to make the expected order source-name-based, - // not model declaration order. - .array_aux("b[State]", "1") - .array_aux("a[Region]", "2") - .array_aux("r[Region]", "RANK(a[*] + b[*], 1)"); - - let result = agg_nodes(&project); - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!( - synthetic.len(), - 1, - "multi-source RANK must mint one synthetic aggregate node; got: {:?}", - result.aggs - ); - assert!(synthetic[0].array_valued_rank); - assert_eq!(source_names(synthetic[0]), vec!["a", "b"]); - assert_eq!(synthetic[0].result_dims, vec!["Region"]); - } - - /// GH #796 review: array-valued RANK over a proper StarRange returns the - /// ranked subdimension view. Its synthetic helper must be dimensioned over - /// `Core`, not the parent `Region`, or helper slots and source halves drift. - #[test] - fn rank_star_range_result_dims_preserve_subdimension() { - let project = TestProject::new("rank_star_range_subdimension") - .named_dimension("Region", &["a", "b", "c"]) - .named_dimension("Core", &["a", "b"]) - .array_aux("arr[Region]", "10") - .array_aux("ranking[Core]", "RANK(arr[*:Core], 1)"); - - let result = agg_nodes(&project); - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!( - synthetic.len(), - 1, - "subdimension RANK must mint one synthetic aggregate node; got: {:?}", - result.aggs - ); - assert!(synthetic[0].array_valued_rank); - assert_eq!(synthetic[0].result_dims, vec!["Core"]); - } - - /// GH #796 review: the source-to-RANK slot helper is shared by element - /// graph edges and link-score names. It must fan active/iterated axes out - /// across every RANK output slot and use result-dimension element names, - /// not source-dimension names, for mapped sibling sources. - #[test] - fn rank_output_slots_use_result_dimension_elements() { - let active_dim_read = vec![AxisRead::Iterated { - dim: "region".to_string(), - source_dim: "region".to_string(), - }]; - assert_eq!( - rank_output_slot_parts_for_row( - &active_dim_read, - &[vec!["north".to_string(), "south".to_string()]], - &["north".to_string()], - ), - Some(vec![vec!["north".to_string()], vec!["south".to_string()]]) - ); - - let mapped_source_read = vec![AxisRead::Reduced { subset: None }]; - assert_eq!( - rank_output_slot_parts_for_row( - &mapped_source_read, - &[vec!["r1".to_string(), "r2".to_string()]], - &[], - ), - Some(vec![vec!["r1".to_string()], vec!["r2".to_string()]]) - ); - - let context_plus_ranked_axis = vec![ - AxisRead::Iterated { - dim: "region".to_string(), - source_dim: "region".to_string(), - }, - AxisRead::Reduced { subset: None }, - ]; - assert_eq!( - rank_output_slot_parts_for_row( - &context_plus_ranked_axis, - &[ - vec!["north".to_string(), "south".to_string()], - vec!["x".to_string(), "y".to_string()], - ], - &["north".to_string()], - ), - Some(vec![ - vec!["north".to_string(), "x".to_string()], - vec!["north".to_string(), "y".to_string()] - ]) - ); - } - - /// GH #776 whole-RHS form: `r[Region] = RANK(pop, 1)` uses the same - /// synthetic array-valued helper as the inline spelling, rather than - /// becoming a variable-backed scalar reducer agg. - #[test] - fn rank_whole_rhs_mints_synthetic_not_variable_backed() { - let project = TestProject::new("rank_whole_rhs") - .named_dimension("Region", &["north", "south"]) - .array_aux("pop[Region]", "100") - .array_aux("r[Region]", "RANK(pop, 1)"); - - let result = agg_nodes(&project); - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!( - synthetic.len(), - 1, - "whole-RHS RANK must mint one synthetic aggregate node; got: {:?}", - result.aggs - ); - assert!(synthetic[0].array_valued_rank); - assert!(result.aggs.iter().all(|a| a.is_synthetic)); - } - - /// GH #766: a StarRange naming a dimension that is NEITHER the axis's - /// own dimension NOR a proper subdimension of it (at best a mid-edit - /// inconsistency) DECLINES the hoist -- it must not silently widen to - /// the full extent. The reducer stays on the conservative path. - #[test] - fn star_range_non_subdimension_declines_hoist() { - let project = TestProject::new("star_range_decline") - .named_dimension("Region", &["a", "b", "c"]) - .named_dimension("Other", &["p", "q"]) - .array_aux("arr[Region]", "10") - .scalar_aux("x", "1 + MEAN(arr[*:Other])"); - - let result = agg_nodes(&project); - assert!( - result.aggs.is_empty(), - "a non-subdimension StarRange must decline the hoist; got: {:?}", - result.aggs - ); - } - - /// GH #766: a StarRange over the axis's OWN dimension (`SUM(arr[*:Region])` - /// where `arr` is declared over `Region`) is the full extent -- - /// `Reduced{subset: None}`, byte-identical to a plain `*`. - #[test] - fn star_range_own_dimension_is_full_extent() { - let project = TestProject::new("star_range_own_dim") - .named_dimension("Region", &["a", "b", "c"]) - .array_aux("arr[Region]", "10") - .scalar_aux("x", "1 + SUM(arr[*:Region])"); - - let result = agg_nodes(&project); - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!(synthetic.len(), 1, "got: {:?}", result.aggs); - assert_eq!( - synthetic[0].canonical_read_slice(), - vec![AxisRead::Reduced { subset: None }] - ); - } - - /// GH #766: a StarRange over a PROPER subdimension carries the - /// subdimension's elements as the `Reduced` subset (canonical names, in - /// subdimension-declared order, resolved via `SubdimensionRelation`). - #[test] - fn star_range_proper_subdimension_carries_subset() { - let project = TestProject::new("star_range_subset") - .named_dimension("Region", &["a", "b", "c"]) - .named_dimension("Core", &["a", "b"]) - .array_aux("arr[Region]", "10") - .scalar_aux("x", "1 + MEAN(arr[*:Core])"); - - let result = agg_nodes(&project); - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!(synthetic.len(), 1, "got: {:?}", result.aggs); - assert_eq!( - synthetic[0].canonical_read_slice(), - vec![AxisRead::Reduced { - subset: Some(vec!["a".to_string(), "b".to_string()]) - }] - ); - assert!(synthetic[0].result_dims.is_empty()); - } - - /// GH #766 (composition): a subset StarRange composes with an iterated - /// axis -- `out[D1] = 1 + SUM(matrix[D1, *:SubD2])` hoists a synthetic - /// agg whose slice is `[Iterated(d1), Reduced{subset}]` and whose - /// `result_dims` carry `D1`. - #[test] - fn star_range_subset_composes_with_iterated_axis() { - let project = TestProject::new("star_range_iterated_subset") - .named_dimension("D1", &["a", "b"]) - .named_dimension("D2", &["x", "y", "z"]) - .named_dimension("SubD2", &["x", "y"]) - .array_aux_direct("matrix", vec!["D1".into(), "D2".into()], "1", None) - .array_aux_direct( - "out", - vec!["D1".into()], - "1 + SUM(matrix[D1, *:SubD2])", - None, - ); - - let result = agg_nodes(&project); - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!(synthetic.len(), 1, "got: {:?}", result.aggs); - assert_eq!( - synthetic[0].canonical_read_slice(), - vec![ - AxisRead::Iterated { - dim: "d1".to_string(), - source_dim: "d1".to_string() - }, - AxisRead::Reduced { - subset: Some(vec!["x".to_string(), "y".to_string()]) - } - ] - ); - assert_eq!(synthetic[0].result_dims, vec!["D1".to_string()]); - } - - /// Test helper: resolve the named dimensions of a synced project into - /// `Dimension` objects (for the gate's `to_dims` argument). - fn resolve_dims( - db: &SimlinDb, - project: crate::db::SourceProject, - names: &[&str], - ) -> Vec { - let dim_ctx = crate::db::project_dimensions_context(db, project); - names - .iter() - .map(|n| { - dim_ctx - .get(&crate::common::CanonicalDimensionName::from_raw(n)) - .unwrap_or_else(|| panic!("dimension {n} resolves")) - .clone() - }) - .collect() - } - - /// GH #766 x T3: a VARIABLE-BACKED partial reduce whose slice carries a - /// SUBSET (`out[D1] = SUM(matrix[D1,*:SubD2])` as the whole RHS) is - /// ACCEPTED by the reduce gate: `try_cross_dimensional_link_scores` - /// derives co-reduced rows from the same `read_slice_rows`, so the - /// subset edges pair with subset divisors. (Pre-T3 the slice was - /// excluded onto the loud conservative regime because the score - /// derivation enumerated the full cartesian.) - #[test] - fn variable_backed_subset_slice_is_accepted_by_reduce_gate() { - let project = TestProject::new("vb_subset_accepted") - .named_dimension("D1", &["a", "b"]) - .named_dimension("D2", &["x", "y", "z"]) - .named_dimension("SubD2", &["x", "y"]) - .array_aux_direct("matrix", vec!["D1".into(), "D2".into()], "1", None) - .array_aux_direct("out", vec!["D1".into()], "SUM(matrix[D1, *:SubD2])", None); - - let datamodel = project.build_datamodel(); - let db = SimlinDb::default(); - let sync = sync_from_datamodel(&db, &datamodel); - let result = enumerate_agg_nodes(&db, sync.models["main"].source, sync.project); - - // The variable-backed agg exists and carries the subset... - let agg = result - .aggs_in_var("out") - .find(|a| a.name == "out") - .expect("expected a variable-backed agg owned by `out`"); - assert!(matches!( - &agg.canonical_read_slice()[1], - AxisRead::Reduced { subset: Some(s) } if s == &["x".to_string(), "y".to_string()] - )); - - // ...and the gate admits it (T3 of the shape-expressiveness design). - let to_dims = resolve_dims(&db, sync.project, &["d1"]); - let accepted = variable_backed_reduce_agg(result, "matrix", "out", &to_dims) - .expect("the subset-bearing aligned slice must be admitted by the reduce gate"); - assert_eq!(accepted.name, "out"); - } - - /// GH #765 x T3: a VARIABLE-BACKED Pinned-mixed aligned slice - /// (`outf[D1] = MEAN(cube[D1,x,*])`) is ACCEPTED by the reduce gate -- - /// the T1-era Pinned exclusion is deleted atomically with the - /// `read_slice_rows` derivation swap. - #[test] - fn reduce_gate_accepts_pinned_mixed_aligned_slice() { - let project = TestProject::new("gate_pinned_mixed") - .named_dimension("D1", &["a", "b"]) - .named_dimension("D2", &["x", "y"]) - .named_dimension("D3", &["p", "q"]) - .array_aux_direct( - "cube", - vec!["D1".into(), "D2".into(), "D3".into()], - "1", - None, - ) - .array_aux_direct("outf", vec!["D1".into()], "MEAN(cube[D1, x, *])", None); - - let datamodel = project.build_datamodel(); - let db = SimlinDb::default(); - let sync = sync_from_datamodel(&db, &datamodel); - let result = enumerate_agg_nodes(&db, sync.models["main"].source, sync.project); - - let to_dims = resolve_dims(&db, sync.project, &["d1"]); - let accepted = variable_backed_reduce_agg(result, "cube", "outf", &to_dims) - .expect("the Pinned-mixed aligned slice must be admitted by the reduce gate"); - assert_eq!(accepted.name, "outf"); - } - - /// Section 6 (scalar owner): a scalar-result Pinned slice - /// (`total = SUM(pop[nyc,*])`, `to_dims` empty) is admitted -- the slot - /// is the bare `total` node, so `emit_agg_routed_edges` emits exactly - /// the read rows into `to`, matching the per-read-row scores. - #[test] - fn reduce_gate_accepts_scalar_owner_pinned_slice() { - let project = TestProject::new("gate_scalar_owner_pinned") - .named_dimension("Region", &["nyc", "boston"]) - .named_dimension("D2", &["p", "q"]) - .array_aux_direct("pop", vec!["Region".into(), "D2".into()], "1", None) - .scalar_aux("total", "SUM(pop[nyc, *])"); - - let datamodel = project.build_datamodel(); - let db = SimlinDb::default(); - let sync = sync_from_datamodel(&db, &datamodel); - let result = enumerate_agg_nodes(&db, sync.models["main"].source, sync.project); - - let accepted = variable_backed_reduce_agg(result, "pop", "total", &[]) - .expect("the scalar-owner Pinned slice must be admitted by the reduce gate"); - assert_eq!(accepted.name, "total"); - } - - /// Section 6 (inert skip): a PURE full-extent scalar reduce - /// (`total = SUM(pop[*])`) stays OUT of the gate -- the reference - /// walker's reduction edges are already the true reads, so routing it - /// through the gate would change nothing and is skipped to keep the - /// diff inert (byte-identity). - #[test] - fn reduce_gate_declines_pure_full_extent_slice() { - let project = TestProject::new("gate_full_extent") - .named_dimension("Region", &["nyc", "boston"]) - .array_aux("pop[Region]", "1") - .scalar_aux("total", "SUM(pop[*])"); - - let datamodel = project.build_datamodel(); - let db = SimlinDb::default(); - let sync = sync_from_datamodel(&db, &datamodel); - let result = enumerate_agg_nodes(&db, sync.models["main"].source, sync.project); - - assert!( - variable_backed_reduce_agg(result, "pop", "total", &[]).is_none(), - "a pure full-extent slice keeps the reference walker's edges (inert skip)" - ); - } - - /// GH #777: an ARRAYED-owner scalar-result Pinned slice - /// (`share[Region] = SUM(pop[nyc,*])` -- no `Iterated` axis, arrayed - /// `to`) is ADMITTED: the single scalar reducer value broadcasts over - /// the owner's dims, and the per-(read-row, full-target-element) - /// machinery (`emit_agg_routed_edges`' broadcast fan-out + - /// `try_cross_dimensional_link_scores`' broadcast-reduce branch) names - /// every slot. - #[test] - fn reduce_gate_admits_arrayed_owner_scalar_result_slice() { - let project = TestProject::new("gate_broadcast_pinned") - .named_dimension("Region", &["nyc", "boston"]) - .named_dimension("D2", &["p", "q"]) - .array_aux_direct("pop", vec!["Region".into(), "D2".into()], "1", None) - .array_aux_direct("share", vec!["Region".into()], "SUM(pop[nyc, *])", None); - - let datamodel = project.build_datamodel(); - let db = SimlinDb::default(); - let sync = sync_from_datamodel(&db, &datamodel); - let result = enumerate_agg_nodes(&db, sync.models["main"].source, sync.project); - - let to_dims = resolve_dims(&db, sync.project, &["region"]); - let accepted = variable_backed_reduce_agg(result, "pop", "share", &to_dims) - .expect("the arrayed-owner broadcast slice must be admitted (GH #777)"); - assert_eq!(accepted.name, "share"); - assert!( - accepted.result_dims.is_empty(), - "the broadcast reducer's result is scalar (no Iterated axis); got: {:?}", - accepted.result_dims - ); - } - - /// GH #764 boundary (T4): a partial reduce BROADCAST over extra target - /// dims (`out[D1,D3] = SUM(matrix[D1,*])` -- `result_dims` a strict - /// subset of `to`'s dims) never reaches the variable-backed gate at all - /// anymore: T4's minting condition routes it to a SYNTHETIC agg, so - /// `variable_backed_reduce_agg` finds no variable-backed candidate (its - /// Iterated-arm alignment check stays as defense). - #[test] - fn reduce_gate_declines_broadcast_result_dims() { - let project = TestProject::new("gate_broadcast_result") - .named_dimension("D1", &["a", "b"]) - .named_dimension("D2", &["x", "y"]) - .named_dimension("D3", &["p", "q"]) - .array_aux_direct("matrix", vec!["D1".into(), "D2".into()], "1", None) - .array_aux_direct( - "out", - vec!["D1".into(), "D3".into()], - "SUM(matrix[D1, *])", - None, - ); - - let datamodel = project.build_datamodel(); - let db = SimlinDb::default(); - let sync = sync_from_datamodel(&db, &datamodel); - let result = enumerate_agg_nodes(&db, sync.models["main"].source, sync.project); - - let to_dims = resolve_dims(&db, sync.project, &["d1", "d3"]); - assert!( - variable_backed_reduce_agg(result, "matrix", "out", &to_dims).is_none(), - "a broadcast whole-RHS reduce must have no variable-backed agg (GH #764)" - ); - assert!( - result.aggs.iter().all(|a| a.is_synthetic), - "T4 mints a synthetic agg for the broadcast shape; got: {:?}", - result.aggs - ); - } - - /// GH #766 / invariant I3 (uniqueness of the full-extent form): a - /// StarRange naming a SAME-CARDINALITY "subdimension" -- including a - /// permuted alias of the axis's element set (`Alias = [c, a, b]` over - /// `Region = [a, b, c]`: containment + equal size means the same element - /// SET) -- normalizes to `Reduced{subset: None}`, never a `Some` subset - /// covering the whole axis. Reduction order is irrelevant (the reduced - /// rows are a set), and keeping the full-extent representation unique - /// means downstream byte-identity does not depend on which spelling the - /// modeler used. - #[test] - fn star_range_same_cardinality_alias_normalizes_to_full_extent() { - let project = TestProject::new("star_range_alias") - .named_dimension("Region", &["a", "b", "c"]) - .named_dimension("Alias", &["c", "a", "b"]) - .array_aux("arr[Region]", "10") - .scalar_aux("x", "1 + SUM(arr[*:Alias])"); - - let result = agg_nodes(&project); - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!(synthetic.len(), 1, "got: {:?}", result.aggs); - assert_eq!( - synthetic[0].canonical_read_slice(), - vec![AxisRead::Reduced { subset: None }], - "a whole-axis alias must normalize to the unique full-extent form" - ); - } - - /// GH #766 (indexed dimensions): a StarRange over an INDEXED - /// subdimension (`SubIndex(3)` with declared `parent = Index(5)`, which - /// maps to the parent's first 3 elements) resolves the subset through - /// the same `SubdimensionRelation` path as named dimensions -- the - /// subset elements are the canonical indexed names `"1".."3"`, matching - /// `dimension_element_names`'s output. - #[test] - fn star_range_indexed_subdimension_carries_subset() { - let project = TestProject::new("star_range_indexed_subdim") - .indexed_dimension("Index", 5) - .indexed_subdimension("SubIndex", 3, "Index") - .array_aux("arr[Index]", "10") - .scalar_aux("x", "1 + MEAN(arr[*:SubIndex])"); - - let result = agg_nodes(&project); - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!(synthetic.len(), 1, "got: {:?}", result.aggs); - assert_eq!( - synthetic[0].canonical_read_slice(), - vec![AxisRead::Reduced { - subset: Some(vec!["1".to_string(), "2".to_string(), "3".to_string()]) - }] - ); - } - - // --- T2 (shape-expressiveness design): per-source `AggNode` invariant - // pins -- the per-source REPRESENTATION invariants (I2, I3b, sorted - // ordering) and the declines `accept_source_slices` enforces. I1's - // *feeder clause* pins (deferred from T2 to avoid the GH #739 vacuity - // trap) landed with T5's RED fixtures, in the section above. - - /// T2 / I3b ordering: `sources` is sorted by canonical variable name - /// regardless of AST occurrence order -- `SUM(b[*] + a[*])` (with `b` - /// first in the argument) still yields `[a, b]`, so salsa cache - /// equality and downstream emission order never depend on how the - /// modeler spelled the argument. - #[test] - fn multi_source_sources_are_sorted_by_var_name() { - let project = TestProject::new("sorted_sources") - .named_dimension("D", &["p", "q"]) - .array_aux_direct("a", vec!["D".into()], "1", None) - .array_aux_direct("b", vec!["D".into()], "2", None) - .scalar_aux("total", "1 + SUM(b[*] + a[*])"); - - let result = agg_nodes(&project); - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!(synthetic.len(), 1, "got: {:?}", result.aggs); - assert_eq!( - source_names(synthetic[0]), - vec!["a", "b"], - "sources must be sorted by canonical name, not AST occurrence order" - ); - } - - /// T2 / I3b dedup: the same variable referenced twice with the SAME - /// slice (`SUM(a[*] + a[*])`) collapses to ONE `AggSource` -- the - /// by-name downstream consumers (`aggs_in_var` routing, the - /// half-emitters) key `sources` on the variable name, so a duplicate - /// entry would make them ambiguous. - #[test] - fn duplicate_var_same_slice_collapses_to_one_source() { - let project = TestProject::new("dup_var_same_slice") - .named_dimension("D", &["p", "q"]) - .array_aux_direct("a", vec!["D".into()], "1", None) - .scalar_aux("total", "1 + SUM(a[*] + a[*])"); - - let result = agg_nodes(&project); - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!(synthetic.len(), 1, "got: {:?}", result.aggs); - assert_eq!( - source_names(synthetic[0]), - vec!["a"], - "a variable read twice with the same slice is one AggSource" - ); - assert_eq!( - synthetic[0].sources[0].read_slice, - vec![AxisRead::Reduced { subset: None }] - ); - } - - /// T2 / I3b decline: the same variable referenced with two DIFFERENT - /// slices (`SUM(a[*] + a[p])` -- `[Reduced]` vs `[Pinned(p)]`) declines - /// the hoist -- since T5, `accept_source_slices`' per-variable - /// one-slice check (the I3b clause); the pin keeps I3b from regressing - /// under the widened per-source acceptance. - #[test] - fn duplicate_var_with_conflicting_slices_declines_hoist() { - let project = TestProject::new("dup_var_conflicting") - .named_dimension("D", &["p", "q"]) - .array_aux_direct("a", vec!["D".into()], "1", None) - .scalar_aux("total", "1 + SUM(a[*] + a[p])"); - - let result = agg_nodes(&project); - assert!( - result.aggs.iter().all(|ag| !ag.reads_var("a")), - "one variable with two different slices must decline the hoist; got: {:?}", - result.aggs - ); - assert!(result.synthetic_by_key.is_empty()); - } - - /// T2 / I1 decline (GREEN characterization): two co-sources with - /// DIFFERING `Reduced` subsets (`SUM(a[*:Sub1] + b[*:Sub2])`) decline - /// the hoist -- their co-reduced rows per slot would disagree, so no - /// canonical slice exists. Enforced by `accept_source_slices`' - /// co-source-identity clause (subset is part of `AxisRead` equality). - #[test] - fn differing_reduced_subsets_decline_hoist() { - let project = TestProject::new("differing_subsets") - .named_dimension("Region", &["a", "b", "c"]) - .named_dimension("Sub1", &["a", "b"]) - .named_dimension("Sub2", &["b", "c"]) - .array_aux("p[Region]", "1") - .array_aux("q[Region]", "2") - .scalar_aux("total", "1 + SUM(p[*:Sub1] + q[*:Sub2])"); - - let result = agg_nodes(&project); - assert!( - result - .aggs - .iter() - .all(|ag| !ag.reads_var("p") && !ag.reads_var("q")), - "co-sources with differing Reduced subsets must decline the hoist; got: {:?}", - result.aggs - ); - assert!(result.synthetic_by_key.is_empty()); - } - - /// T2 / I1 positive twin: two co-sources with the SAME `Reduced` subset - /// (`SUM(p[*:Sub] + q[*:Sub])`) hoist one agg whose every source - /// carries the identical subset-bearing canonical slice. - #[test] - fn agreeing_reduced_subsets_hoist_with_shared_subset() { - let project = TestProject::new("agreeing_subsets") - .named_dimension("Region", &["a", "b", "c"]) - .named_dimension("Sub", &["a", "b"]) - .array_aux("p[Region]", "1") - .array_aux("q[Region]", "2") - .scalar_aux("total", "1 + SUM(p[*:Sub] + q[*:Sub])"); - - let result = agg_nodes(&project); - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!(synthetic.len(), 1, "got: {:?}", result.aggs); - assert_eq!(source_names(synthetic[0]), vec!["p", "q"]); - let expected = vec![AxisRead::Reduced { - subset: Some(vec!["a".to_string(), "b".to_string()]), - }]; - for s in &synthetic[0].sources { - assert_eq!(s.read_slice, expected, "source {} slice", s.var); - } - } - - /// T2 / I2 + the scalar-feeder representation: a scalar feeder of a - /// hoisted reducer (`scale` in `SUM(pop[*] * scale)`, GH #737) IS a - /// source -- the routing filter and the element graph's scalar-feeder - /// arm key on membership -- and carries an EMPTY read slice (one - /// `AxisRead` per axis, and a scalar has none), while the arrayed - /// co-source's slice has one entry per its declared axis. - #[test] - fn scalar_feeder_source_carries_empty_slice() { - let project = TestProject::new("scalar_feeder") - .named_dimension("Region", &["NYC", "Boston"]) - .array_aux("pop[Region]", "100") - .scalar_aux("scale", "0.5") - .scalar_aux("total", "1 + SUM(pop[*] * scale)"); - - let result = agg_nodes(&project); - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!(synthetic.len(), 1, "got: {:?}", result.aggs); - assert_eq!(source_names(synthetic[0]), vec!["pop", "scale"]); - // I2: one AxisRead per the source's OWN declared axes. - assert_eq!( - synthetic[0].source_read_slice("pop"), - vec![AxisRead::Reduced { subset: None }], - "the arrayed co-source's slice has one entry per its axis" - ); - assert!( - synthetic[0].source_read_slice("scale").is_empty(), - "a scalar feeder has no axes, so its slice is empty" - ); - // The canonical slice skips the feeder's empty slice. - assert_eq!( - synthetic[0].canonical_read_slice(), - vec![AxisRead::Reduced { subset: None }] - ); - // And the defensive non-source lookup is the empty slice too. - assert!(synthetic[0].source_read_slice("absent").is_empty()); - assert!(synthetic[0].reads_var("scale")); - assert!(!synthetic[0].reads_var("absent")); - } - - // -- T5 / GH #767: the I1 FEEDER clause ------------------------------- - // - // These pins were deliberately deferred from T2 (pinning them before the - // acceptance widened would have been vacuous -- the GH #739 trap). They - // land with T5's RED fixtures: an iterated-dim projection feeder is - // accepted as an `AggSource` with ITS OWN slice, the canonical slice is - // the co-source (Reduced-bearing) slice regardless of source sort order, - // and everything outside the projection rule still declines. - - /// T5 / I1 feeder clause (GH #767): the iterated-dim-feeder reducer - /// `1 + SUM(matrix[D1,*] * frac[D1])` (inline => synthetic) IS hoisted: - /// `matrix` is the co-source carrying the canonical - /// `[Iterated, Reduced]` slice, `frac` is a projection feeder carrying - /// its OWN `[Iterated]` slice. `frac` sorts BEFORE `matrix`, so this - /// also pins the `canonical_read_slice` contract fix: the canonical - /// slice is the first slice WITH a `Reduced` axis, never an - /// alphabetically-first feeder slice. - #[test] - fn iterated_dim_feeder_projection_hoists_with_per_source_slices() { - let project = TestProject::new("feeder_projection") - .named_dimension("D1", &["r1", "r2"]) - .named_dimension("D2", &["c1", "c2"]) - .array_aux_direct("matrix", vec!["D1".into(), "D2".into()], "5", None) - .array_aux("frac[D1]", "0.5") - .array_aux("growth[D1]", "1 + SUM(matrix[D1, *] * frac[D1])"); - - let result = agg_nodes(&project); - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!( - synthetic.len(), - 1, - "the projection-feeder reducer must be hoisted (GH #767); got: {:?}", - result.aggs - ); - let agg = synthetic[0]; - assert_eq!(source_names(agg), vec!["frac", "matrix"]); - assert_eq!( - agg.source_read_slice("frac"), - vec![AxisRead::Iterated { - dim: "d1".to_string(), - source_dim: "d1".to_string() - }], - "the feeder carries its OWN projection slice" - ); - assert_eq!( - agg.source_read_slice("matrix"), - vec![ - AxisRead::Iterated { - dim: "d1".to_string(), - source_dim: "d1".to_string() - }, - AxisRead::Reduced { subset: None } - ], - "the co-source carries the canonical slice" - ); - // The contract fix: even though `frac` sorts first, the canonical - // slice is the Reduced-bearing co-source slice. - assert_eq!(agg.canonical_read_slice(), agg.source_read_slice("matrix")); - assert_eq!(agg.result_dims, vec!["D1".to_string()]); - } - - /// T5 / I1: the WHOLE-RHS form of the feeder shape - /// (`growth[D1] = SUM(matrix[D1,*] * frac[D1])`, the GH #743/#767 - /// fixture) is VARIABLE-BACKED -- the canonical (co-source) slice is - /// aligned with the owner's dims, so the variable IS the agg and no - /// synthetic is minted. - #[test] - fn iterated_dim_feeder_whole_rhs_is_variable_backed() { - let project = TestProject::new("feeder_whole_rhs") - .named_dimension("D1", &["r1", "r2"]) - .named_dimension("D2", &["c1", "c2"]) - .array_aux_direct("matrix", vec!["D1".into(), "D2".into()], "5", None) - .array_aux("frac[D1]", "0.5") - .array_aux("growth[D1]", "SUM(matrix[D1, *] * frac[D1])"); - - let result = agg_nodes(&project); - assert!(result.synthetic_by_key.is_empty(), "got: {:?}", result.aggs); - let vb: Vec<&AggNode> = result.aggs.iter().filter(|a| !a.is_synthetic).collect(); - assert_eq!(vb.len(), 1, "got: {:?}", result.aggs); - assert_eq!(vb[0].name, "growth"); - assert_eq!(source_names(vb[0]), vec!["frac", "matrix"]); - assert_eq!(vb[0].result_dims, vec!["D1".to_string()]); - assert!(vb[0].source_is_projection_feeder("frac")); - assert!(!vb[0].source_is_projection_feeder("matrix")); - } - - /// T5 / I1: a feeder combined with a SCALAR feeder still hoists -- - /// `SUM(matrix[D1,*] * frac[D1] * scale)` has three sources, the scalar - /// one with an empty slice (it is NOT a projection feeder: the - /// changed-last machinery for scalar feeders is `generate_scalar_feeder_ - /// to_agg_equation`, not the per-row form). - #[test] - fn projection_feeder_and_scalar_feeder_combo_hoists() { - let project = TestProject::new("feeder_combo") - .named_dimension("D1", &["r1", "r2"]) - .named_dimension("D2", &["c1", "c2"]) - .array_aux_direct("matrix", vec!["D1".into(), "D2".into()], "5", None) - .array_aux("frac[D1]", "0.5") - .scalar_aux("scale", "2") - .array_aux("growth[D1]", "1 + SUM(matrix[D1, *] * frac[D1] * scale)"); - - let result = agg_nodes(&project); - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!(synthetic.len(), 1, "got: {:?}", result.aggs); - let agg = synthetic[0]; - assert_eq!(source_names(agg), vec!["frac", "matrix", "scale"]); - assert!(agg.source_read_slice("scale").is_empty()); - assert!(agg.source_is_projection_feeder("frac")); - assert!(!agg.source_is_projection_feeder("scale")); - } - - /// T5 / I1 (review MINOR-5): a PINNED-bearing CANONICAL slice is within - /// the feeder clause's scope -- the clause keys only on the canonical - /// slice's Iterated target dims, so `SUM(cube[D1, c1, *] * frac[D1])` - /// hoists with canonical `[Iterated, Pinned(c1), Reduced]` and the - /// feeder's own `[Iterated]` projection. (It is the FEEDER's slice that - /// must be Iterated-only, not the canonical one.) - #[test] - fn pinned_bearing_canonical_with_feeder_hoists() { - let project = TestProject::new("pinned_canonical_feeder") - .named_dimension("D1", &["r1", "r2"]) - .named_dimension("D2", &["c1", "c2"]) - .named_dimension("D3", &["k1", "k2"]) - .array_aux_direct( - "cube", - vec!["D1".into(), "D2".into(), "D3".into()], - "5", - None, - ) - .array_aux("frac[D1]", "0.5") - .array_aux("growth[D1]", "1 + SUM(cube[D1, c1, *] * frac[D1])"); - - let result = agg_nodes(&project); - let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!(synthetic.len(), 1, "got: {:?}", result.aggs); - let agg = synthetic[0]; - assert_eq!( - agg.source_read_slice("cube"), - vec![ - AxisRead::Iterated { - dim: "d1".to_string(), - source_dim: "d1".to_string() - }, - AxisRead::Pinned("c1".to_string()), - AxisRead::Reduced { subset: None } - ] - ); - assert!(agg.source_is_projection_feeder("frac")); - assert_eq!(agg.result_dims, vec!["D1".to_string()]); - } - - /// T5 / I1 decline: a no-`Reduced` source with a PINNED axis is NOT a - /// projection feeder (the design's clause: a feeder slice consists ONLY - /// of `Iterated` axes) -- the hoist declines. - #[test] - fn feeder_with_pinned_axis_declines_hoist() { - let project = TestProject::new("feeder_pinned") - .named_dimension("D1", &["r1", "r2"]) - .named_dimension("D2", &["c1", "c2"]) - .array_aux_direct("matrix", vec!["D1".into(), "D2".into()], "5", None) - .array_aux_direct("w", vec!["D1".into(), "D2".into()], "0.5", None) - .array_aux("growth[D1]", "1 + SUM(matrix[D1, *] * w[D1, c1])"); - - let result = agg_nodes(&project); - assert!( - result.aggs.iter().all(|a| !a.reads_var("matrix")), - "a Pinned-axis no-Reduced source must decline the hoist; got: {:?}", - result.aggs - ); - } - - /// T5 / I1 decline: a feeder whose Iterated dims are a PROPER SUBSET of - /// the canonical slice's Iterated dims declines -- its rows are not 1:1 - /// with the agg result slots (one feeder row would feed every slot it - /// projects from, a broadcast the per-`(row, slot)` machinery cannot - /// name). Documented residual: the design's I1 wording ("drawn from the - /// canonical Iterated target-dim set") is implemented as ordered - /// EQUALITY for exactly this reason. - #[test] - fn feeder_with_subset_iterated_dims_declines_hoist() { - let project = TestProject::new("feeder_subset") - .named_dimension("D1", &["r1", "r2"]) - .named_dimension("D2", &["c1", "c2"]) - .named_dimension("D3", &["x", "y"]) - .array_aux_direct( - "cube", - vec!["D1".into(), "D2".into(), "D3".into()], - "5", - None, - ) - .array_aux("w[D1]", "0.5") - .array_aux_direct( - "growth", - vec!["D1".into(), "D2".into()], - "1 + SUM(cube[D1, D2, *] * w[D1])", - None, - ); - - let result = agg_nodes(&project); - assert!( - result.aggs.iter().all(|a| !a.reads_var("cube")), - "a proper-subset feeder must decline the hoist; got: {:?}", - result.aggs - ); - } - - /// T5 / I1 decline: a feeder whose Iterated dims are a PERMUTATION of - /// the canonical order declines -- `read_slice_rows` derives slot - /// coordinates in the source's axis order, so a permuted feeder's slots - /// would mis-name the agg's `result_dims`-ordered slots. - #[test] - fn feeder_with_permuted_iterated_dims_declines_hoist() { - let project = TestProject::new("feeder_permuted") - .named_dimension("D1", &["r1", "r2"]) - .named_dimension("D2", &["c1", "c2"]) - .named_dimension("D3", &["x", "y"]) - .array_aux_direct( - "cube", - vec!["D1".into(), "D2".into(), "D3".into()], - "5", - None, - ) - .array_aux_direct("w", vec!["D2".into(), "D1".into()], "0.5", None) - .array_aux_direct( - "growth", - vec!["D1".into(), "D2".into()], - "1 + SUM(cube[D1, D2, *] * w[D2, D1])", - None, - ); - - let result = agg_nodes(&project); - assert!( - result.aggs.iter().all(|a| !a.reads_var("cube")), - "a permuted feeder must decline the hoist; got: {:?}", - result.aggs - ); - } - - /// T5 / I1 decline: a MAPPED Iterated axis (GH #534) anywhere in the - /// combination declines the feeder clause -- pinning the slot element - /// into the equation text reads the TARGET-dim element, which is not - /// the source row a mapped reference reads, so the changed-last feeder - /// equation would mis-pin. The mapped sliced reducer WITHOUT a feeder - /// stays hoisted (the GH #534 path is unchanged). - #[test] - fn mapped_iterated_axis_with_feeder_declines_hoist() { - let project = TestProject::new("feeder_mapped") - .named_dimension("Region", &["r1", "r2"]) - .named_dimension("D2", &["x", "y"]) - .named_dimension_with_mapping("State", &["s1", "s2"], "Region") - .array_aux_direct("matrix", vec!["Region".into(), "D2".into()], "1", None) - .array_aux_direct("frac", vec!["State".into()], "0.5", None) - .array_aux_direct( - "out", - vec!["State".into()], - "1 + SUM(matrix[State, *] * frac[State])", - None, - ); - - let result = agg_nodes(&project); - assert!( - result.aggs.iter().all(|a| !a.reads_var("matrix")), - "a mapped iterated axis with a feeder must decline the hoist; got: {:?}", - result.aggs - ); - } - - /// T5 / I3b decline: the same variable appearing as both a co-source - /// and a feeder-shaped reference (`SUM(matrix[D1,*] * matrix[D1,c1])`) - /// declines -- one variable, two different slices. - #[test] - fn duplicate_var_as_co_source_and_feeder_declines_hoist() { - let project = TestProject::new("dup_co_source_feeder") - .named_dimension("D1", &["r1", "r2"]) - .named_dimension("D2", &["c1", "c2"]) - .array_aux_direct("matrix", vec!["D1".into(), "D2".into()], "5", None) - .array_aux("growth[D1]", "1 + SUM(matrix[D1, *] * matrix[D1, c1])"); - - let result = agg_nodes(&project); - assert!( - result.aggs.iter().all(|a| !a.reads_var("matrix")), - "one variable with co-source AND feeder slices must decline; got: {:?}", - result.aggs - ); - } - - /// T5 / I1 decline: two CO-SOURCES (both Reduced-bearing) with - /// differing slices still decline, exactly as before the feeder clause - /// -- the clause widens acceptance only for no-`Reduced` projections. - #[test] - fn co_sources_with_differing_slices_still_decline() { - let project = TestProject::new("co_source_differ") - .named_dimension("D1", &["r1", "r2"]) - .named_dimension("D2", &["c1", "c2"]) - .array_aux_direct("a", vec!["D1".into(), "D2".into()], "1", None) - .array_aux_direct("b", vec!["D2".into(), "D1".into()], "2", None) - .array_aux("growth[D1]", "1 + SUM(a[D1, *] + b[*, D1])"); - - let result = agg_nodes(&project); - assert!( - result - .aggs - .iter() - .all(|ag| !ag.reads_var("a") && !ag.reads_var("b")), - "co-sources with differing slices must still decline; got: {:?}", - result.aggs - ); - } - - /// PR #784 review (P3), purely defensive: every arrayed reducer source - /// has a `per_var` slice by construction (`collect_var_refs` and - /// `collect_arrayed_source_slices` walk the identical reference - /// surface), but if that invariant ever broke, [`agg_sources`] must - /// DECLINE the hoist (`None` -- the reference stays on the conservative - /// Direct path) rather than silently substituting the CANONICAL slice: - /// for a projection feeder (whose slice differs from canonical by - /// design, GH #767) that substitution would mislabel the feeder as a - /// co-source and corrupt the per-`(row, slot)` link scores downstream. - #[test] - fn agg_sources_declines_when_arrayed_source_lacks_per_var_slice() { - let project = TestProject::new("agg_sources_invariant") - .named_dimension("D1", &["r1", "r2"]) - .array_aux("pop[D1]", "1") - .scalar_aux("scale", "2") - .scalar_aux("total", "1 + SUM(pop[*] * scale)"); - let datamodel = project.build_datamodel(); - let db = SimlinDb::default(); - let sync = sync_from_datamodel(&db, &datamodel); - let model = sync.models["main"].source; - let variables = crate::db::reconstruct_model_variables(&db, model, sync.project); - let dm_dims = crate::db::project_datamodel_dims(&db, sync.project); - let dim_ctx = crate::db::project_dimensions_context(&db, sync.project); - let ctx = AggWalkCtx { - variables: &variables, - target_iterated_dims: &[], - dm_dims: dm_dims.as_slice(), - dim_ctx, - }; - let canonical = vec![AxisRead::Reduced { subset: None }]; - - // The invariant broken by hand: `pop` (arrayed) absent from `per_var`. - let broken = CombinedReadSlices { - canonical: canonical.clone(), - per_var: HashMap::new(), - }; - assert_eq!( - agg_sources(vec!["pop".to_string()], &broken, &ctx), - None, - "a missing per-var slice for an arrayed source must decline the \ - hoist, never substitute the canonical slice" - ); - - // The intact invariant: each source carries its own slice; a scalar - // source still gets the empty slice. - let intact = CombinedReadSlices { - canonical: canonical.clone(), - per_var: HashMap::from([("pop".to_string(), canonical.clone())]), - }; - let sources = agg_sources(vec!["scale".to_string(), "pop".to_string()], &intact, &ctx) - .expect("an intact per-var map must build the sources"); - assert_eq!( - sources, - vec![ - AggSource { - var: "pop".to_string(), - read_slice: canonical, - }, - AggSource { - var: "scale".to_string(), - read_slice: vec![], - }, - ] - ); - } - - /// GH #983: every recognized reducer's classification is CARRIED on the - /// SYNTHETIC node it decided, so no emitter has to recover it by - /// re-parsing [`AggNode::equation_text`]. - /// - /// Scope, stated because "every recognized reducer" is only one of the two - /// axes here: this covers all seven reducers on the SYNTHETIC producer arm, - /// which is the arm both readers actually reach. `register_agg`'s - /// variable-backed arm stores a `reducer` too and no test covers it, because - /// no reader consumes it -- see [`AggNode::reducer`]'s doc. - /// - /// The rows are the reducer set itself, not a sample. `reducer_kind` is - /// the only admission test, and [`crate::ltm_augment`]'s - /// `classify_builtin_if_references_source` -- the function that reads the - /// kind, name and body back off `AggNode::reducer` -- destructures exactly - /// SEVEN `BuiltinFn` variants (`Sum`, `Mean`, `Min`, `Max`, `Stddev`, - /// `Rank`, `Size`) and calls `unreachable!()` on the rest, so those seven - /// are the whole space and this table has seven rows. Each row states the - /// three facts the emitters read: whether a node is minted at all, and (if - /// so) the `ReducerKind` and uppercase name the carried builtin classifies - /// to. - /// - /// `SIZE` is the one row that mints nothing (`ReducerKind::Constant` is - /// never hoisted -- its link score is always 0), and `RANK` is the one - /// row that mints an ARRAY-valued node; both are properties of the - /// enumerator this table would notice changing. - #[test] - fn every_reducer_carries_its_classification_on_the_agg_node() { - use crate::ltm_augment::{ReducerKind, classify_reducer_in_builtin}; - - // (equation for `out`, out's dims, expected (kind, name, array_valued_rank)) - struct Row { - equation: &'static str, - out_dims: &'static [&'static str], - expected: Option<(ReducerKind, &'static str, bool)>, - } - let rows = [ - Row { - equation: "1 + SUM(pop[*])", - out_dims: &[], - expected: Some((ReducerKind::Linear, "SUM", false)), - }, - Row { - equation: "1 + MEAN(pop[*])", - out_dims: &[], - expected: Some((ReducerKind::Linear, "MEAN", false)), - }, - Row { - equation: "1 + MIN(pop[*])", - out_dims: &[], - expected: Some((ReducerKind::Nonlinear, "MIN", false)), - }, - Row { - equation: "1 + MAX(pop[*])", - out_dims: &[], - expected: Some((ReducerKind::Nonlinear, "MAX", false)), - }, - Row { - equation: "1 + STDDEV(pop[*])", - out_dims: &[], - expected: Some((ReducerKind::Nonlinear, "STDDEV", false)), - }, - Row { - // Array-valued: the node is arrayed over the ranked axis, so - // its consumer must be arrayed too. - equation: "1 + RANK(pop[*], 1)", - out_dims: &["Region"], - expected: Some((ReducerKind::Nonlinear, "RANK", true)), - }, - Row { - // `ReducerKind::Constant`: recognized, never hoisted. - equation: "1 + SIZE(pop[*])", - out_dims: &[], - expected: None, - }, - ]; - - for Row { - equation, - out_dims, - expected, - } in rows - { - let mut project = TestProject::new("carried") - .named_dimension("Region", &["nyc", "boston"]) - .array_aux("pop[Region]", "1"); - project = if out_dims.is_empty() { - project.scalar_aux("out", equation) - } else { - project.array_aux_direct( - "out", - out_dims.iter().map(|d| (*d).to_string()).collect(), - equation, - None, - ) - }; - let aggs = agg_nodes(&project); - let synthetic: Vec<&AggNode> = aggs.aggs.iter().filter(|a| a.is_synthetic).collect(); - - let Some((kind, name, array_valued_rank)) = expected else { - assert!( - synthetic.is_empty(), - "{equation}: a Constant reducer must mint no aggregate node" - ); - continue; - }; - assert_eq!( - synthetic.len(), - 1, - "{equation}: expected exactly one synthetic aggregate node" - ); - let agg = synthetic[0]; - assert_eq!( - agg.array_valued_rank, array_valued_rank, - "{equation}: array_valued_rank" - ); - - let classified = classify_reducer_in_builtin(&agg.reducer, "pop", true) - .unwrap_or_else(|| panic!("{equation}: the carried builtin must classify")); - assert_eq!(classified.kind, kind, "{equation}: kind"); - assert_eq!(classified.name, name, "{equation}: name"); - assert!( - classified.is_bare, - "{equation}: an aggregate node's equation IS the reducer call" - ); - // The body is the reducer's array argument, taken from the AST the - // enumerator walked rather than re-parsed from `equation_text`. - assert_eq!( - crate::ast::print_eqn(&classified.body), - "pop[*]", - "{equation}: body" - ); - } - } - - /// GH #983: the carried reducer is stored in - /// [`crate::ast::Expr2::strip_loc_and_bounds`] form, so a `Loc`-only edit - /// leaves the salsa-cached `AggNodesResult` equal. - /// - /// **This is one of the two ways the backdating claim can fail, and it - /// measures only this one.** The other is float non-reflexivity on a NaN - /// literal, which normalization cannot touch and which is closed at the - /// root by [`crate::ast::Literal`]'s bit-pattern equality; - /// [`a_nan_literal_in_a_reducer_does_not_defeat_agg_backdating`] pins that - /// arm. - /// - /// The two spellings differ ONLY in a leading term that mints no - /// aggregate node of its own -- a `SIZE` call, which is recognized as a - /// reducer but never hoisted -- so `SUM(pop[*])` moves to a different byte - /// offset and nothing else about the enumeration changes. The whole - /// `AggNodesResult` must therefore compare EQUAL, which is exactly - /// salsa's backdating criterion and so is the invalidation claim: an edit - /// that changes neither the reducer nor its sources must not invalidate - /// this query's consumers, as it did not before the node carried an AST - /// at all. - /// - /// It is the `Loc` half of the normalization that this measures. The - /// `ArrayBounds` half is inert TODAY and cannot be measured, because - /// `db::analysis::reconstruct_model_variables` -- the source of the ASTs - /// this query walks -- lowers against an EMPTY model scope, so - /// `Expr2Context::get_dimensions` resolves nothing and no bound is ever - /// allocated. The leading `SIZE(other[*])` is chosen over a bare constant - /// so that this test starts measuring the bounds half the moment that - /// stops being true: its arrayed argument would take the first temp id - /// and push `pop[*]` to the second. - /// - /// The second assertion is a weaker independent check on the same - /// property (re-normalizing the stored builtin is a no-op); it catches - /// normalization being dropped entirely but, being idempotence, cannot - /// by itself catch the normalization being made too weak. That is what - /// the equality assertion above is for. - #[test] - fn the_carried_reducer_is_normalized_so_offset_only_edits_backdate() { - let build = |leading: &str| { - TestProject::new("normalized") - .named_dimension("Region", &["nyc", "boston"]) - .array_aux("pop[Region]", "1") - .array_aux("other[Region]", "2") - .scalar_aux("out", &format!("{leading} + SUM(pop[*])")) - }; - let before = agg_nodes(&build("1")); - let after = agg_nodes(&build("SIZE(other[*])")); - - let sum_agg = |r: &AggNodesResult| { - r.aggs - .iter() - .find(|a| a.equation_text == "sum(pop[*])") - .cloned() - .expect("the SUM subexpression must be hoisted") - }; - assert_eq!( - before, after, - "an edit that only moves the reducer's byte offsets must leave the \ - enumerated aggregate nodes equal, so salsa backdates" - ); - let agg = sum_agg(&before); - assert_eq!( - agg.reducer - .clone() - .map(crate::ast::Expr2::strip_loc_and_bounds) - .strip_own_locs(), - agg.reducer, - "the stored reducer must already be normalized" - ); - } - - // The third channel by which `AggNode::reducer` (GH #983) could make the - // salsa-cached, `PartialEq`-compared `AggNodesResult` unequal to an - // identical rebuild: the `f64` on `Expr2::Const`, which - // `strip_loc_and_bounds` cannot normalize away (there is no rewrite that - // removes a NaN literal without changing what the equation means). - // - // It is closed at the ROOT rather than here: `ast::Literal` compares float - // literals by BIT PATTERN, so `nan == nan` and a bit-identical AST is - // equal to itself (GH #987/#981). Salsa's backdating criterion is exactly - // that equality, so without the root fix every revision bump -- from any - // unrelated edit anywhere in the project -- re-executed - // `model_element_causal_edges`, `model_ltm_reference_sites` and - // `model_ltm_variables` for any model with a `nan` in a hoisted reducer. - // - // This test was written inverted (`assert_ne!`) as the characterization of - // that defect; flipping it is how the root fix demonstrates it reached a - // real consumer rather than only its own unit test. - #[test] - fn a_nan_literal_in_a_reducer_does_not_defeat_agg_backdating() { - let build = || { - TestProject::new("nan_backdating") - .named_dimension("Region", &["nyc", "boston"]) - .array_aux("pop[Region]", "1") - .scalar_aux("out", "1 + SUM(pop[*] * nan)") - }; - - // Guard against a vacuous pass: the reducer really is hoisted, so the - // NaN really does ride on a stored `AggNode::reducer`. - let enumerated = agg_nodes(&build()); - let synthetic: Vec<&AggNode> = enumerated.aggs.iter().filter(|a| a.is_synthetic).collect(); - assert_eq!( - synthetic.len(), - 1, - "the NaN-bearing reducer must still be hoisted for this to measure anything" - ); - - assert_eq!( - agg_nodes(&build()), - agg_nodes(&build()), - "two enumerations of an identical NaN-bearing model must compare \ - equal, so `enumerate_agg_nodes` backdates for this model like any \ - other" - ); - } -} +#[path = "ltm_agg_tests.rs"] +mod tests; diff --git a/src/simlin-engine/src/ltm_agg_tests.rs b/src/simlin-engine/src/ltm_agg_tests.rs new file mode 100644 index 000000000..52bba71d6 --- /dev/null +++ b/src/simlin-engine/src/ltm_agg_tests.rs @@ -0,0 +1,3205 @@ +// Copyright 2026 The Simlin Authors. All rights reserved. +// Use of this source code is governed by the Apache License, +// Version 2.0, that can be found in the LICENSE file. + +//! Tests for `ltm_agg`: aggregate-node enumeration, the per-axis access +//! classifier, and the reducer decision table. +//! +//! Split out of `ltm_agg.rs` only to keep that file under the project +//! line-count lint, and mounted with `#[path]` as its `tests` child module, so +//! `use super::*` resolves the parent's private items exactly as an inline +//! `mod tests` did. + +use super::*; +use crate::db::{SimlinDb, sync_from_datamodel}; +use crate::test_common::TestProject; + +/// Test helper: the source-variable names of an agg (sorted + deduped +/// by the [`AggNode::sources`] construction invariant). +fn source_names(a: &AggNode) -> Vec<&str> { + a.sources.iter().map(|s| s.var.as_str()).collect() +} + +/// Build a `TestProject`, sync into salsa, and return the enumerated +/// aggregate nodes for the "main" model. +fn agg_nodes(project: &TestProject) -> AggNodesResult { + let datamodel = project.build_datamodel(); + let db = SimlinDb::default(); + let sync = sync_from_datamodel(&db, &datamodel); + let source_model = sync.models["main"].source; + let source_project = sync.project; + enumerate_agg_nodes(&db, source_model, source_project).clone() +} + +/// Build a `TestProject` and return the GH #791 cartesian-decline verdict +/// for the `from -> to` edge. +fn source_read(project: &TestProject, from: &str, to: &str) -> UnhoistedSourceRead { + let datamodel = project.build_datamodel(); + let db = SimlinDb::default(); + let sync = sync_from_datamodel(&db, &datamodel); + let link = LtmLinkId::new(&db, from.to_string(), to.to_string()); + unhoisted_reducer_source_read(&db, link, sync.models["main"].source, sync.project).clone() +} + +/// GH #791: a multi-source reducer whose source read is a STRICT slice +/// (`pop[nyc,*]`, with no full-extent read of `pop`) is the silent-cartesian +/// family -- `StrictSlice` (the caller loud-skips it), carrying the actual +/// slice so the diagnostic renders `pop[nyc,*]` rather than a canned +/// example. +#[test] +fn unhoisted_source_read_strict_slice_for_pinned_only_read() { + let project = TestProject::new("strict_slice") + .named_dimension("Region", &["nyc", "boston"]) + .named_dimension("D2", &["p", "q"]) + .array_aux("pop[Region,D2]", "1") + .array_aux("w[D2]", "0.5") + .array_aux("share[Region]", "SUM(pop[nyc,*] * w[*])"); + let UnhoistedSourceRead::StrictSlice(slice) = source_read(&project, "pop", "share") else { + panic!("the pinned-only read must classify StrictSlice"); + }; + assert_eq!(render_read_slice_for_diagnostic(&slice), "nyc,*"); +} + +/// GH #793: a hoisted full-extent sibling reducer must not mask an +/// un-hoisted strict-slice sibling on the same `pop -> share` edge. The +/// full-extent read is already represented by synthetic agg halves, so the +/// residual un-hoisted verdict remains `StrictSlice`. +#[test] +fn unhoisted_source_read_ignores_hoisted_sibling_full_read() { + let project = TestProject::new("strict_with_hoisted_sibling") + .named_dimension("Region", &["nyc", "boston"]) + .named_dimension("D2", &["p", "q"]) + .array_aux("pop[Region,D2]", "1") + .array_aux("w[D2]", "0.5") + .array_aux_direct( + "share", + vec!["Region".into()], + "SUM(pop[nyc, *] * w[*]) + SUM(pop[*, *])", + None, + ); + let UnhoistedSourceRead::StrictSlice(slice) = source_read(&project, "pop", "share") else { + panic!("the un-hoisted strict sibling must not be masked by the hoisted full read"); + }; + assert_eq!(render_read_slice_for_diagnostic(&slice), "nyc,*"); +} + +/// GH #791 boundary: the SAME variable read at full extent (`pop[*]`) AND +/// pinned (`pop[north]`) -- the GH #744 self-reference family -- leaves NO +/// row unread, so it is `FullExtent` (the caller keeps the conservative +/// delta-ratio cartesian, unchanged). +#[test] +fn unhoisted_source_read_full_extent_when_full_read_present() { + let project = TestProject::new("self_ref") + .named_dimension("region", &["north", "south"]) + .array_aux("pop[region]", "1") + .scalar_aux("tp", "SUM(pop[*] * pop[north])"); + assert!(matches!( + source_read(&project, "pop", "tp"), + UnhoistedSourceRead::FullExtent + )); +} + +/// GH #791 boundary: a pure full-extent multi-source read (`matrix[D1,*]`, +/// `[Iterated, Reduced]`) is `FullExtent` -- the #779 bare-feeder fixture's +/// `matrix -> growth` edge keeps its correct cartesian diagonal. +#[test] +fn unhoisted_source_read_full_extent_for_iterated_reduced() { + let project = TestProject::new("iter_reduced") + .named_dimension("D1", &["a", "b"]) + .named_dimension("D2", &["c", "d"]) + .array_aux("matrix[D1,D2]", "1") + .array_aux("frac", "0.5") + .array_aux("growth[D1]", "SUM(matrix[D1,*] * frac)"); + assert!(matches!( + source_read(&project, "matrix", "growth"), + UnhoistedSourceRead::FullExtent + )); +} + +/// GH #791 boundary: a dynamic-index reducer (`SUM(pop[idx,*])`, `idx` +/// non-literal) is NOT statically describable -- `NotDescribable`, so the +/// caller keeps the DOCUMENTED conservative cartesian cross-product. +#[test] +fn unhoisted_source_read_not_describable_for_dynamic_index() { + let project = TestProject::new("dyn_index") + .named_dimension("Region", &["nyc", "boston"]) + .named_dimension("D2", &["p", "q"]) + .array_aux("pop[Region,D2]", "1") + .scalar_aux("idx", "2") + .array_aux("share[Region]", "SUM(pop[idx,*])"); + assert!(matches!( + source_read(&project, "pop", "share"), + UnhoistedSourceRead::NotDescribable + )); +} + +/// GH #792: a PER-ELEMENT-EQUATION (`Ast::Arrayed`) owner whose every slot +/// holds a strict-slice multi-source reducer (each `share` slot is +/// `SUM(pop[,*] * w[*])`) classifies the `pop -> share` edge +/// `PerElementReducerRead` -- a decline. The first describable slice in +/// sorted-slot order (`boston`) rides along for the diagnostic. +#[test] +fn unhoisted_source_read_declines_per_element_strict_slots() { + let project = TestProject::new("per_element_strict") + .named_dimension("Region", &["nyc", "boston"]) + .named_dimension("D2", &["p", "q"]) + .array_aux("pop[Region,D2]", "1") + .array_aux("w[D2]", "0.5") + .array_with_ranges_direct( + "share", + vec!["Region".into()], + vec![ + ("nyc", "SUM(pop[nyc,*] * w[*])"), + ("boston", "SUM(pop[boston,*] * w[*])"), + ], + None, + ); + let UnhoistedSourceRead::PerElementReducerRead(Some(slice)) = + source_read(&project, "pop", "share") + else { + panic!("per-element strict-slice slots must classify PerElementReducerRead(Some)"); + }; + // Sorted-key walk visits `boston` before `nyc`. + assert_eq!(render_read_slice_for_diagnostic(&slice), "boston,*"); +} + +/// GH #792 any-reducer-read rule: ONLY ONE slot reads `pop` (inside a +/// reducer); the other slot does not read `pop` at all. Any slot's reducer +/// read declines the WHOLE edge (the Bare stand-in conflates the slots). +#[test] +fn unhoisted_source_read_declines_when_only_some_slots_read() { + let project = TestProject::new("per_element_some") + .named_dimension("Region", &["nyc", "boston"]) + .named_dimension("D2", &["p", "q"]) + .array_aux("pop[Region,D2]", "1") + .array_aux("w[D2]", "0.5") + .array_with_ranges_direct( + "share", + vec!["Region".into()], + vec![("nyc", "SUM(pop[nyc,*] * w[*])"), ("boston", "0")], + None, + ); + let UnhoistedSourceRead::PerElementReducerRead(Some(slice)) = + source_read(&project, "pop", "share") + else { + panic!("a single reducer-reading slot must decline the whole edge"); + }; + assert_eq!(render_read_slice_for_diagnostic(&slice), "nyc,*"); +} + +/// GH #792 finding 2: a per-element owner whose every slot reads `pop` at +/// FULL EXTENT inside an I1-declined multi-source reducer +/// (`SUM(pop[*,*] * w[*])`) ALSO declines -- the full-extent verdict only +/// validates the cartesian projection, which needs a single dt-expression +/// a per-element owner does not have; the Bare stand-in is just as wrong +/// for full reads (verified ~-0.0 empirically). +#[test] +fn unhoisted_source_read_declines_per_element_full_extent_slots() { + let project = TestProject::new("per_element_full") + .named_dimension("Region", &["nyc", "boston"]) + .named_dimension("D2", &["p", "q"]) + .array_aux("pop[Region,D2]", "1") + .array_aux("w[D2]", "0.5") + .array_with_ranges_direct( + "share", + vec!["Region".into()], + vec![ + ("nyc", "SUM(pop[*,*] * w[*])"), + ("boston", "SUM(pop[*,*] * w[*])"), + ], + None, + ); + let UnhoistedSourceRead::PerElementReducerRead(Some(slice)) = + source_read(&project, "pop", "share") + else { + panic!("per-element full-extent multi-source slots must still decline"); + }; + assert_eq!(render_read_slice_for_diagnostic(&slice), "*,*"); +} + +/// GH #792 finding 1: the DIM-NAME spelling (`SUM(pop[Region,*] * w[*])` +/// per slot). In a per-element slot no iterated dimension is in scope +/// (mirroring `enumerate_agg_nodes`' Arrayed arm), so the dim-named index +/// is not statically describable -- but the read IS a reducer read, so the +/// edge still declines, with no representative slice for the diagnostic. +/// (Execution pins `Region` to the slot's element -- a strict read -- so +/// the previous `Iterated => full extent` classification was wrong; the +/// executed-value pin lives in the integration twin.) +#[test] +fn unhoisted_source_read_declines_per_element_dim_named_slots() { + let project = TestProject::new("per_element_dim_named") + .named_dimension("Region", &["nyc", "boston"]) + .named_dimension("D2", &["p", "q"]) + .array_aux("pop[Region,D2]", "1") + .array_aux("w[D2]", "0.5") + .array_with_ranges_direct( + "share", + vec!["Region".into()], + vec![ + ("nyc", "SUM(pop[Region,*] * w[*])"), + ("boston", "SUM(pop[Region,*] * w[*])"), + ], + None, + ); + assert!(matches!( + source_read(&project, "pop", "share"), + UnhoistedSourceRead::PerElementReducerRead(None) + )); +} + +/// GH #792 explicit sub-case decision: a DYNAMIC-INDEX reducer read inside +/// a per-element slot (`SUM(pop[idx,*])`) also declines. Pre-fix it was +/// silently stand-in'd exactly like the strict spelling (the scalar/A2A +/// dynamic-index family keeps its documented conservative cartesian, but a +/// per-element owner has no cartesian arm to keep), so declining is both +/// sound and consistent; no existing test pinned the old silent behavior. +#[test] +fn unhoisted_source_read_declines_per_element_dynamic_index_slots() { + let project = TestProject::new("per_element_dyn") + .named_dimension("Region", &["nyc", "boston"]) + .named_dimension("D2", &["p", "q"]) + .array_aux("pop[Region,D2]", "1") + .scalar_aux("idx", "2") + .array_with_ranges_direct( + "share", + vec!["Region".into()], + vec![("nyc", "SUM(pop[idx,*])"), ("boston", "SUM(pop[idx,*])")], + None, + ); + assert!(matches!( + source_read(&project, "pop", "share"), + UnhoistedSourceRead::PerElementReducerRead(None) + )); +} + +/// GH #792 non-decline boundary: a per-element owner that references `pop` +/// only OUTSIDE any reducer (the disjoint-dim FixedIndex family's shape) +/// classifies `NotDescribable` -- no reducer read, so the edge keeps its +/// existing emission path (`try_disjoint_dim_arrayed_link_scores` et al). +#[test] +fn unhoisted_source_read_not_describable_for_per_element_non_reducer_refs() { + let project = TestProject::new("per_element_bare") + .named_dimension("Region", &["nyc", "boston"]) + .named_dimension("D2", &["p", "q"]) + .array_aux("pop[Region,D2]", "1") + .array_with_ranges_direct( + "share", + vec!["Region".into()], + vec![ + ("nyc", "pop[nyc,p] * 0.5"), + ("boston", "pop[boston,q] * 0.5"), + ], + None, + ); + assert!(matches!( + source_read(&project, "pop", "share"), + UnhoistedSourceRead::NotDescribable + )); +} + +/// AC4.3: a variable whose entire dt-equation is exactly one reducer call +/// (scalar) mints no synthetic agg -- the variable itself is the agg. +#[test] +fn whole_rhs_scalar_reducer_is_its_own_agg() { + let project = TestProject::new("whole_rhs") + .named_dimension("Region", &["NYC", "Boston", "LA"]) + .array_aux("population[Region]", "100") + .scalar_aux("total_population", "SUM(population[*])"); + + let result = agg_nodes(&project); + + // No `$⁚ltm⁚agg⁚{n}` minted. + assert!( + result.aggs.iter().all(|a| !a.is_synthetic), + "whole-RHS scalar reducer must not mint a synthetic agg; got: {:?}", + result.aggs + ); + // The reducer maps to a variable-backed agg named `total_population`, + // owned by `total_population`'s equation. (Variable-backed aggs are + // resolved via `aggs_in_var`, not `agg_for_key` -- the latter is + // synthetic-only, since two different scalars can each be `SUM(pop[*])`.) + let agg = result + .aggs_in_var("total_population") + .find(|a| a.name == "total_population") + .expect("expected a variable-backed agg owned by `total_population`"); + assert!(!agg.is_synthetic); + assert_eq!(source_names(agg), vec!["population"]); + assert!(agg.result_dims.is_empty()); + // `agg_for_key` resolves only synthetic aggs, so it must not find this one. + assert!(result.agg_for_key("sum(population[*])").is_none()); +} + +/// AC4.3 (arrayed variant): `agg[D1] = SUM(matrix[D1,*])` is whole-RHS, so +/// the variable is the agg; `result_dims` carries `D1` and `read_slice` +/// records the `Iterated(D1)` / `Reduced` axis split (the `D1` axis is +/// iterated over the A2A dimension space, the second axis is reduced). +#[test] +fn whole_rhs_arrayed_partial_reduce_is_its_own_agg() { + let project = TestProject::new("whole_rhs_partial") + .named_dimension("D1", &["a", "b"]) + .named_dimension("D2", &["x", "y"]) + .array_aux_direct("matrix", vec!["D1".into(), "D2".into()], "1", None) + .array_aux_direct("agg", vec!["D1".into()], "SUM(matrix[D1, *])", None); + + let result = agg_nodes(&project); + + assert!( + result.aggs.iter().all(|a| !a.is_synthetic), + "whole-RHS arrayed reducer must not mint a synthetic agg; got: {:?}", + result.aggs + ); + let agg = result + .aggs_in_var("agg") + .next() + .expect("expected an agg owned by `agg`"); + assert_eq!(agg.name, "agg"); + assert!(!agg.is_synthetic); + assert_eq!(source_names(agg), vec!["matrix"]); + assert_eq!(agg.result_dims, vec!["D1".to_string()]); + assert_eq!( + agg.canonical_read_slice(), + vec![ + AxisRead::Iterated { + dim: "d1".to_string(), + source_dim: "d1".to_string() + }, + AxisRead::Reduced { subset: None } + ] + ); +} + +/// AC4.3 (arrayed full-reduce broadcast): `share[Region] = SUM(pop[*])` is +/// a whole-RHS reducer, so the variable is the agg -- but `SUM(pop[*])` is a +/// *full* reduce (scalar result) merely broadcast to `[Region]`, so the +/// agg's `result_dims` is `[]`, not `[Region]`. (Contrast with +/// `agg[D1] = SUM(matrix[D1, *])`, a partial reduce that genuinely varies +/// per `D1`.) +#[test] +fn whole_rhs_arrayed_full_reduce_broadcast_has_scalar_result_dims() { + let project = TestProject::new("whole_rhs_broadcast") + .named_dimension("Region", &["NYC", "Boston"]) + .array_aux("pop[Region]", "100") + .array_aux("share[Region]", "SUM(pop[*])"); + + let result = agg_nodes(&project); + + assert!( + result.aggs.iter().all(|a| !a.is_synthetic), + "whole-RHS reducer must not mint a synthetic agg; got: {:?}", + result.aggs + ); + let agg = result + .aggs_in_var("share") + .next() + .expect("expected an agg owned by `share`"); + assert_eq!(agg.name, "share"); + assert!(!agg.is_synthetic); + assert_eq!(source_names(agg), vec!["pop"]); + assert!( + agg.result_dims.is_empty(), + "a full reduce broadcast to an arrayed variable has scalar result dims, got: {:?}", + agg.result_dims + ); +} + +/// AC4.1 (the basic mint): `share[r] = pop[r] / SUM(pop[*])` mints one +/// synthetic agg `$⁚ltm⁚agg⁚0` for the sub-expression `SUM(pop[*])`. +#[test] +fn subexpression_reducer_mints_one_synthetic_agg() { + let project = TestProject::new("share_mint") + .named_dimension("Region", &["NYC", "Boston", "LA"]) + .array_aux("pop[Region]", "100") + .array_aux("share[Region]", "pop / SUM(pop[*])"); + + let result = agg_nodes(&project); + + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!( + synthetic.len(), + 1, + "expected exactly one synthetic agg; got: {:?}", + result.aggs + ); + assert_eq!(synthetic[0].name, "$\u{205A}ltm\u{205A}agg\u{205A}0"); + assert_eq!(synthetic[0].equation_text, "sum(pop[*])"); + assert_eq!(source_names(synthetic[0]), vec!["pop"]); + assert!(synthetic[0].result_dims.is_empty()); + assert!( + result + .aggs_in_var("share") + .any(|a| a.name == "$\u{205A}ltm\u{205A}agg\u{205A}0") + ); +} + +/// P2 regression: an inline reducer (`share[r] = pop[r] / SUM(pop[*])`, +/// which must mint a *synthetic* agg) sharing canonical text with a +/// *whole-RHS* reducer of the same shape (`denom = SUM(pop[*])`, which +/// is *variable-backed*) must NOT reuse the variable-backed agg -- +/// regardless of declaration order. Dedup-by-key applies to synthetic +/// aggs only; variable-backed aggs are never deduped (a whole-RHS +/// reducer variable is its own distinct agg node). Before the fix, with +/// `denom` visited first (canonical-sorted: `denom` < `share`), the +/// inline use found `by_key["sum(pop[*])"]` already populated by `denom` +/// and reused it, so `share` got no synthetic agg and its reducer fell +/// back to the conservative direct path. +#[test] +fn inline_reducer_does_not_reuse_variable_backed_agg() { + let project = TestProject::new("inline_vs_var_backed") + .named_dimension("Region", &["NYC", "Boston"]) + .array_aux("pop[Region]", "100") + // `denom` (canonical-sorted first) is a whole-RHS reducer -> + // variable-backed agg named `denom`. + .scalar_aux("denom", "SUM(pop[*])") + // `share` (visited after `denom`) uses the same reducer text as + // a sub-expression -> must mint its own synthetic agg. + .array_aux("share[Region]", "pop / SUM(pop[*])"); + + let result = agg_nodes(&project); + + // The variable-backed agg `denom` exists and is not synthetic. + // (`agg_for_key` now resolves only synthetic aggs, so look up the + // variable-backed one through `by_var` instead.) + let denom_agg = result + .aggs_in_var("denom") + .find(|a| a.name == "denom") + .expect("expected a variable-backed agg owned by `denom`"); + assert!( + !denom_agg.is_synthetic, + "`denom`'s agg must be variable-backed" + ); + assert_eq!(denom_agg.equation_text, "sum(pop[*])"); + + // `share` must own a *synthetic* agg with the same reducer text. + let share_agg = result + .aggs_in_var("share") + .find(|a| a.is_synthetic) + .expect("expected a synthetic agg owned by `share`"); + assert_eq!(share_agg.name, "$\u{205A}ltm\u{205A}agg\u{205A}0"); + assert_eq!(share_agg.equation_text, "sum(pop[*])"); + assert_eq!(source_names(share_agg), vec!["pop"]); + // `agg_for_key` resolves the reducer text to the *synthetic* agg. + assert_eq!( + result.agg_for_key("sum(pop[*])").map(|a| a.name.as_str()), + Some("$\u{205A}ltm\u{205A}agg\u{205A}0") + ); + + // There must be exactly one synthetic agg and exactly one + // variable-backed agg -- two distinct nodes despite identical text. + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + let var_backed_aggs: Vec<&AggNode> = result.aggs.iter().filter(|a| !a.is_synthetic).collect(); + assert_eq!( + synthetic.len(), + 1, + "expected one synthetic agg, got: {:?}", + result.aggs + ); + assert_eq!( + var_backed_aggs.len(), + 1, + "expected one variable-backed agg, got: {:?}", + result.aggs + ); +} + +/// P2 regression (reverse declaration order): the same model as +/// `inline_reducer_does_not_reuse_variable_backed_agg` but built so that +/// the inline-use variable would be visited first if order mattered. +/// `enumerate_agg_nodes` visits variables in canonical-sorted order, so +/// `denom` < `share` always; this test instead uses different names +/// (`a_share` < `z_denom`) to confirm the synthetic agg is minted when +/// the inline use is encountered *before* the whole-RHS reducer. +#[test] +fn inline_reducer_mints_synthetic_when_visited_before_variable_backed() { + let project = TestProject::new("inline_first") + .named_dimension("Region", &["NYC", "Boston"]) + .array_aux("pop[Region]", "100") + // `a_share` (canonical-sorted first) uses the reducer inline. + .array_aux("a_share[Region]", "pop / SUM(pop[*])") + // `z_denom` (visited after) is the whole-RHS reducer. + .scalar_aux("z_denom", "SUM(pop[*])"); + + let result = agg_nodes(&project); + + let share_agg = result + .aggs_in_var("a_share") + .find(|a| a.is_synthetic) + .expect("expected a synthetic agg owned by `a_share`"); + assert_eq!(share_agg.name, "$\u{205A}ltm\u{205A}agg\u{205A}0"); + assert_eq!(share_agg.equation_text, "sum(pop[*])"); + + let denom_agg = result + .aggs_in_var("z_denom") + .find(|a| a.name == "z_denom") + .expect("expected a variable-backed agg owned by `z_denom`"); + assert!(!denom_agg.is_synthetic); + + assert_eq!(result.aggs.iter().filter(|a| a.is_synthetic).count(), 1); + assert_eq!(result.aggs.iter().filter(|a| !a.is_synthetic).count(), 1); +} + +/// Two whole-RHS reducers with *identical* canonical text are two +/// distinct variable-backed agg nodes (one per variable) -- never +/// deduped, because each variable genuinely is its own aggregate. +#[test] +fn two_whole_rhs_reducers_same_text_are_distinct_aggs() { + let project = TestProject::new("two_var_backed") + .named_dimension("Region", &["NYC", "Boston"]) + .array_aux("pop[Region]", "100") + .scalar_aux("total_a", "SUM(pop[*])") + .scalar_aux("total_b", "SUM(pop[*])"); + + let result = agg_nodes(&project); + + let var_backed: Vec<&AggNode> = result.aggs.iter().filter(|a| !a.is_synthetic).collect(); + assert_eq!( + var_backed.len(), + 2, + "two whole-RHS reducers must be two distinct variable-backed aggs; got: {:?}", + result.aggs + ); + let names: std::collections::HashSet<&str> = + var_backed.iter().map(|a| a.name.as_str()).collect(); + assert!(names.contains("total_a"), "missing total_a: {names:?}"); + assert!(names.contains("total_b"), "missing total_b: {names:?}"); + // No synthetic aggs (neither reducer is a sub-expression). + assert_eq!(result.aggs.iter().filter(|a| a.is_synthetic).count(), 0); +} + +/// Two *inline* uses of the same reducer text still dedupe to one +/// synthetic agg (the synthetic dedup-by-key path is preserved). +#[test] +fn two_inline_uses_same_text_dedupe_to_one_synthetic() { + let project = TestProject::new("two_inline") + .named_dimension("Region", &["NYC", "Boston"]) + .array_aux("pop[Region]", "100") + .array_aux("share_a[Region]", "pop / SUM(pop[*])") + .array_aux("share_b[Region]", "pop * 2 / SUM(pop[*])"); + + let result = agg_nodes(&project); + + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!( + synthetic.len(), + 1, + "two inline uses of the same reducer must dedupe to one synthetic agg; got: {:?}", + result.aggs + ); + assert_eq!(synthetic[0].name, "$\u{205A}ltm\u{205A}agg\u{205A}0"); + // Both variables reference the same deduped synthetic agg index. + let a_idx = result.by_var.get("share_a").cloned().unwrap_or_default(); + let b_idx = result.by_var.get("share_b").cloned().unwrap_or_default(); + assert_eq!(a_idx, b_idx); +} + +/// AC4.4 (nested reducers): `x = SUM(a[*]) / SUM(b[*])` mints two distinct +/// synthetic agg nodes (`$⁚ltm⁚agg⁚0` for `SUM(a[*])`, `$⁚ltm⁚agg⁚1` for +/// `SUM(b[*])`). The `/` is not a reducer; neither `SUM` is inside the +/// other, so both are maximal. +#[test] +fn nested_reducers_mint_two_aggs() { + let project = TestProject::new("nested") + .named_dimension("Region", &["NYC", "Boston"]) + .array_aux("a[Region]", "10") + .array_aux("b[Region]", "20") + .scalar_aux("x", "SUM(a[*]) / SUM(b[*])"); + + let result = agg_nodes(&project); + + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!( + synthetic.len(), + 2, + "expected two synthetic aggs; got: {:?}", + result.aggs + ); + // First-encounter (left-to-right DFS) order: SUM(a[*]) then SUM(b[*]). + assert_eq!(synthetic[0].name, "$\u{205A}ltm\u{205A}agg\u{205A}0"); + assert_eq!(synthetic[0].equation_text, "sum(a[*])"); + assert_eq!(source_names(synthetic[0]), vec!["a"]); + assert_eq!(synthetic[1].name, "$\u{205A}ltm\u{205A}agg\u{205A}1"); + assert_eq!(synthetic[1].equation_text, "sum(b[*])"); + assert_eq!(source_names(synthetic[1]), vec!["b"]); +} + +/// AC4.4 (dedup): the same reducer subexpression appearing in two +/// variables' equations (with whitespace/casing differences in the +/// source text) maps to one synthetic agg node referenced by both. +#[test] +fn ast_identical_reducers_dedupe() { + let project = TestProject::new("dedup") + .named_dimension("Region", &["NYC", "Boston"]) + .array_aux("pop[Region]", "100") + // Two different equations both contain SUM(pop[*]); the first is + // spelled with extra spacing and uppercase. + .array_aux("share_a[Region]", "pop / SUM( POP [ * ] )") + .array_aux("share_b[Region]", "pop * 2 / sum(pop[*])"); + + let result = agg_nodes(&project); + + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!( + synthetic.len(), + 1, + "AST-identical reducers must dedupe to one agg; got: {:?}", + result.aggs + ); + assert_eq!(synthetic[0].equation_text, "sum(pop[*])"); + // Both variables reference the same agg index. + let a_idx: Vec = result.by_var.get("share_a").cloned().unwrap_or_default(); + let b_idx: Vec = result.by_var.get("share_b").cloned().unwrap_or_default(); + assert_eq!(a_idx.len(), 1); + assert_eq!(b_idx.len(), 1); + assert_eq!( + a_idx, b_idx, + "both variables must point at the same deduped agg index" + ); +} + +/// Per-element `Ast::Arrayed` target with a different reducer per element: +/// `x[a] = SUM(p[*]); x[b] = MEAN(p[*])` mints two synthetic agg nodes, +/// one per element's reducer. +#[test] +fn per_element_arrayed_target_mints_one_agg_per_element_reducer() { + let project = TestProject::new("per_elem") + .named_dimension("D", &["a", "b"]) + .array_aux("p[D]", "1") + .array_with_ranges_direct( + "x", + vec!["D".into()], + vec![("a", "SUM(p[*])"), ("b", "MEAN(p[*])")], + None, + ); + + let result = agg_nodes(&project); + + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!( + synthetic.len(), + 2, + "per-element reducers must mint one agg per element; got: {:?}", + result.aggs + ); + let texts: std::collections::HashSet<&str> = + synthetic.iter().map(|a| a.equation_text.as_str()).collect(); + assert!(texts.contains("sum(p[*])"), "missing sum(p[*]): {texts:?}"); + assert!( + texts.contains("mean(p[*])"), + "missing mean(p[*]): {texts:?}" + ); + // Both are owned by `x`. + let x_idx = result.by_var.get("x").cloned().unwrap_or_default(); + assert_eq!(x_idx.len(), 2); +} + +/// Determinism: the same model built twice (or with variables declared in +/// a different order) yields identical agg names assigned to the same +/// subexpressions. +#[test] +fn enumeration_is_deterministic_under_variable_reordering() { + // Two synthetic aggs: SUM(a[*]) and SUM(b[*]). Whichever variable + // happens to be visited first is irrelevant -- we always visit in + // canonical-name sorted order, and within an equation left-to-right. + let build = |order_a_first: bool| { + let mut p = TestProject::new("determinism") + .named_dimension("Region", &["NYC", "Boston"]) + .array_aux("a[Region]", "10") + .array_aux("b[Region]", "20"); + // `q` references SUM(a[*]) and SUM(b[*]); `r` references the same + // pair. We add them in different orders to confirm the result is + // identical. + if order_a_first { + p = p + .scalar_aux("q", "SUM(a[*]) + SUM(b[*])") + .scalar_aux("r", "SUM(a[*]) * SUM(b[*])"); + } else { + p = p + .scalar_aux("r", "SUM(a[*]) * SUM(b[*])") + .scalar_aux("q", "SUM(a[*]) + SUM(b[*])"); + } + agg_nodes(&p) + }; + + let r1 = build(true); + let r2 = build(false); + assert_eq!( + r1.aggs, r2.aggs, + "enumeration must be deterministic regardless of declaration order" + ); + assert_eq!(r1.synthetic_by_key, r2.synthetic_by_key); + // Specifically: SUM(a[*]) -> agg 0, SUM(b[*]) -> agg 1 (a < b, and + // within q's equation SUM(a[*]) precedes SUM(b[*])). + assert_eq!( + r1.agg_for_key("sum(a[*])").map(|a| a.name.clone()), + Some("$\u{205A}ltm\u{205A}agg\u{205A}0".to_string()) + ); + assert_eq!( + r1.agg_for_key("sum(b[*])").map(|a| a.name.clone()), + Some("$\u{205A}ltm\u{205A}agg\u{205A}1".to_string()) + ); +} + +/// A model with no reducers produces an empty result. +#[test] +fn model_without_reducers_has_no_aggs() { + let project = TestProject::new("no_reducers") + .stock("population", "100", &["births"], &["deaths"], None) + .flow("births", "population * 0.1", None) + .flow("deaths", "population * 0.05", None) + .scalar_const("rate", 0.1); + + let result = agg_nodes(&project); + assert!( + result.aggs.is_empty(), + "model without reducers must have no aggs; got: {:?}", + result.aggs + ); + assert!(result.synthetic_by_key.is_empty()); + assert!(result.by_var.is_empty()); +} + +/// A reducer over a *scalar* source is not hoisted (the parser would +/// normally reject it anyway, but be defensive). +#[test] +fn reducer_over_scalar_source_is_not_hoisted() { + // `SUM(s)` where `s` is scalar -- pathological, but must not mint an + // agg. (We also keep a real arrayed reducer to confirm the + // enumerator still finds the legitimate one.) + let project = TestProject::new("scalar_reducer") + .named_dimension("Region", &["NYC", "Boston"]) + .scalar_aux("s", "5") + .array_aux("pop[Region]", "100") + .scalar_aux("y", "SUM(s) + SUM(pop[*])"); + + let result = agg_nodes(&project); + // Only the arrayed reducer is recognized. + assert!( + result.agg_for_key("sum(pop[*])").is_some(), + "the arrayed reducer must be recognized; got: {:?}", + result.aggs + ); + assert!( + result.agg_for_key("sum(s)").is_none(), + "a reducer over a scalar source must not be hoisted; got: {:?}", + result.aggs + ); +} + +/// SIZE is not hoisted -- its link score is always 0, matching +/// `try_cross_dimensional_link_scores`'s `Some(vec![])` for SIZE. +#[test] +fn size_reducer_is_not_hoisted() { + let project = TestProject::new("size_reducer") + .named_dimension("Region", &["NYC", "Boston"]) + .array_aux("pop[Region]", "100") + .scalar_aux("n", "SIZE(pop[*])"); + + let result = agg_nodes(&project); + assert!( + result.aggs.is_empty(), + "SIZE must not be hoisted as an agg; got: {:?}", + result.aggs + ); +} + +/// AC4.1: a reducer over an explicit *slice* used as a sub-expression +/// (`x[r] = ... + SUM(pop[NYC, *])`) IS hoisted into a synthetic agg -- +/// the `read_slice` descriptor records which rows it reads +/// (`[Pinned(nyc), Reduced]` over `pop`'s `[Region, Age]` axes), so the +/// element-graph reroute and the per-element reducer link scores route +/// only those rows. `result_dims` is `[]` here: there is no `Iterated` +/// axis (the `Region` on the target `x` is broadcast; the read is a +/// single row). The `pop[NYC, Adult]` `Direct` reference is separate -- +/// not part of the agg. +#[test] +fn slice_reducer_subexpression_is_hoisted() { + let project = TestProject::new("slice_subexpr") + .named_dimension("Region", &["NYC", "Boston"]) + .named_dimension("Age", &["Adult", "Child"]) + .array_aux_direct("pop", vec!["Region".into(), "Age".into()], "10", None) + .array_aux_direct( + "x", + vec!["Region".into()], + "pop[NYC, Adult] + SUM(pop[NYC, *])", + None, + ); + + let result = agg_nodes(&project); + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!( + synthetic.len(), + 1, + "a slice-reducer subexpression must mint one synthetic agg; got: {:?}", + result.aggs + ); + assert_eq!(synthetic[0].name, "$\u{205A}ltm\u{205A}agg\u{205A}0"); + // `expr2_to_string` puts a space after the comma in a multi-index + // subscript -- assert the canonical text it actually produces. + assert_eq!(synthetic[0].equation_text, "sum(pop[nyc, *])"); + assert_eq!(source_names(synthetic[0]), vec!["pop"]); + assert_eq!( + synthetic[0].canonical_read_slice(), + vec![ + AxisRead::Pinned("nyc".to_string()), + AxisRead::Reduced { subset: None } + ] + ); + assert!( + synthetic[0].result_dims.is_empty(), + "no Iterated axis -- result dims must be empty; got: {:?}", + synthetic[0].result_dims + ); + assert!( + result + .aggs_in_var("x") + .any(|a| a.name == "$\u{205A}ltm\u{205A}agg\u{205A}0") + ); +} + +/// AC4.2: a *partial*-reduce slice over an iterated dimension used as a +/// sub-expression (`x[D1] = ... + SUM(matrix[D1, *])`, `matrix[D1, D2]`, +/// `x` A2A over `D1`) mints an arrayed synthetic agg over `D1`: +/// `read_slice = [Iterated(d1), Reduced]`, `result_dims = [D1]`. The +/// element graph routes `matrix[d1, d2] → agg[d1]`. +#[test] +fn sliced_reducer_over_iterated_dim_mints_arrayed_agg() { + let project = TestProject::new("iterated_slice_subexpr") + .named_dimension("D1", &["a", "b"]) + .named_dimension("D2", &["x", "y"]) + .array_aux_direct("matrix", vec!["D1".into(), "D2".into()], "1", None) + .array_aux_direct( + "x", + vec!["D1".into()], + "matrix[a, x] + SUM(matrix[D1, *])", + None, + ); + + let result = agg_nodes(&project); + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!( + synthetic.len(), + 1, + "an iterated-dim slice-reducer subexpression must mint one synthetic agg; got: {:?}", + result.aggs + ); + assert_eq!( + synthetic[0].canonical_read_slice(), + vec![ + AxisRead::Iterated { + dim: "d1".to_string(), + source_dim: "d1".to_string() + }, + AxisRead::Reduced { subset: None } + ] + ); + assert_eq!(synthetic[0].result_dims, vec!["D1".to_string()]); + assert_eq!(source_names(synthetic[0]), vec!["matrix"]); + // `expr2_to_string` canonicalizes the iterated dim name lowercase. + assert_eq!(synthetic[0].equation_text, "sum(matrix[d1, *])"); +} + +/// #514: a *mixed* read slice -- `Iterated` + `Pinned` + `Reduced` axes +/// on one source. `matrix3d[D1, Region, Age]`, `x` A2A over `D1`, +/// `x[D1] = ... + SUM(matrix3d[D1, NYC, *])`: the first axis is iterated +/// over the target's `D1`, the second is pinned to the literal `NYC`, +/// the third (wildcard) is reduced ⇒ `read_slice = [Iterated(d1), +/// Pinned(nyc), Reduced]`, `result_dims = [D1]` (only the iterated axis +/// shapes the agg). Mints one arrayed synthetic agg over `D1`. +#[test] +fn mixed_pinned_iterated_reduced_slice_mints_arrayed_agg() { + let project = TestProject::new("mixed_slice_subexpr") + .named_dimension("D1", &["a", "b"]) + .named_dimension("Region", &["NYC", "Boston"]) + .named_dimension("Age", &["Adult", "Child"]) + .array_aux_direct( + "matrix3d", + vec!["D1".into(), "Region".into(), "Age".into()], + "1", + None, + ) + .array_aux_direct( + "x", + vec!["D1".into()], + "matrix3d[a, NYC, Adult] + SUM(matrix3d[D1, NYC, *])", + None, + ); + + let result = agg_nodes(&project); + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!( + synthetic.len(), + 1, + "a mixed pinned/iterated/reduced slice must mint one synthetic agg; got: {:?}", + result.aggs + ); + assert_eq!( + synthetic[0].canonical_read_slice(), + vec![ + AxisRead::Iterated { + dim: "d1".to_string(), + source_dim: "d1".to_string() + }, + AxisRead::Pinned("nyc".to_string()), + AxisRead::Reduced { subset: None } + ] + ); + assert_eq!(synthetic[0].result_dims, vec!["D1".to_string()]); + assert_eq!(source_names(synthetic[0]), vec!["matrix3d"]); + assert_eq!(synthetic[0].equation_text, "sum(matrix3d[d1, nyc, *])"); +} + +/// #514: a multi-source reducer whose arrayed args agree on their read +/// slice -- `total = 1 + SUM(a[*] + b[*])`, `a`, `b` both over `D`. The +/// reducer's argument expression references two arrayed sources; each +/// reads its whole extent (`[Reduced]`), the slices agree, so one +/// synthetic agg is minted carrying that combined slice and *both* source +/// variables. +#[test] +fn multi_source_reducer_agreeing_slices_mints_one_agg() { + let project = TestProject::new("multi_source_reducer") + .named_dimension("D", &["p", "q"]) + .array_aux_direct("a", vec!["D".into()], "1", None) + .array_aux_direct("b", vec!["D".into()], "2", None) + .scalar_aux("total", "1 + SUM(a[*] + b[*])"); + + let result = agg_nodes(&project); + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!( + synthetic.len(), + 1, + "a multi-source reducer with agreeing slices must mint one synthetic agg; got: {:?}", + result.aggs + ); + assert_eq!( + synthetic[0].canonical_read_slice(), + vec![AxisRead::Reduced { subset: None }] + ); + assert!(synthetic[0].result_dims.is_empty()); + // `sources` lists every arrayed model variable in the argument + // (sorted by name), each carrying the IDENTICAL canonical slice -- + // invariant I1's identical-co-source form (T2 of the + // shape-expressiveness design: acceptance is identical-only, so + // per-source slices cannot yet differ). + assert_eq!(source_names(synthetic[0]), vec!["a", "b"]); + for s in &synthetic[0].sources { + assert_eq!( + s.read_slice, + vec![AxisRead::Reduced { subset: None }], + "every arrayed co-source must carry the canonical slice; got {:?} for {}", + s.read_slice, + s.var + ); + } +} + +/// #514 (negative guard): a multi-source reducer whose arrayed args read +/// *incompatible* slices is NOT hoisted -- `total = 1 + SUM(a[*] + b[*])` +/// where `a` is over `D1` and `b` is over `D2` (disjoint dims, so +/// `[Reduced]` for `a`'s one axis vs `[Reduced]` for `b`'s -- the slices +/// have the same shape but the *sources* differ in dimensionality; more +/// to the point, a 1-axis `a[*]` and a 2-axis `b[*, *]` disagree on +/// length). Use clearly-different ranks to force the disagreement: `a` +/// over `D1`, `b` over `D1 x D2`. `combined_read_slice` returns `None`, +/// so no agg is minted for this reducer. +#[test] +fn multi_source_reducer_disagreeing_slices_is_not_hoisted() { + let project = TestProject::new("multi_source_disagree") + .named_dimension("D1", &["p", "q"]) + .named_dimension("D2", &["x", "y"]) + .array_aux_direct("a", vec!["D1".into()], "1", None) + .array_aux_direct("b", vec!["D1".into(), "D2".into()], "2", None) + .scalar_aux("total", "1 + SUM(a[*] + b[*, *])"); + + let result = agg_nodes(&project); + assert!( + result + .aggs + .iter() + .all(|ag| !ag.reads_var("a") && !ag.reads_var("b")), + "a multi-source reducer whose args read incompatible slices must not be hoisted; \ + got: {:?}", + result.aggs + ); + assert!(result.synthetic_by_key.is_empty()); +} + +/// GH #534: a sliced reducer whose iterated index lines up with the +/// source's row axis via a *positional dimension mapping* +/// (`matrix[Region, D2]`, `State` over `{s1, s2}` with a `State→Region` +/// mapping, target A2A over `State` with `... + SUM(matrix[State, *])`) +/// IS hoisted: the `Iterated` axis carries the (target, source) dim +/// pair, `result_dims` is the TARGET's iterated dim (`State` -- the +/// dimension the agg variable is arrayed over), and the emitters remap +/// each source row to its positionally-corresponding slot. +/// (`classify_iterated_dim_shape`'s own mapped branch -- a +/// whole-equation-iterated subscript, not a sliced reducer argument -- +/// is a separate path and stays `Bare`; see +/// `db::ltm_ir_tests::ir_mapped_iterated_dim_subscript_is_bare`.) +#[test] +fn mapped_iterated_dim_sliced_reducer_is_hoisted_with_pair() { + let project = TestProject::new("mapped_iterated_slice") + .named_dimension("Region", &["r1", "r2"]) + .named_dimension("D2", &["x", "y"]) + .named_dimension_with_mapping("State", &["s1", "s2"], "Region") + .array_aux_direct("matrix", vec!["Region".into(), "D2".into()], "1", None) + .array_aux_direct( + "out", + vec!["State".into()], + "matrix[r1, x] + SUM(matrix[State, *])", + None, + ); + + let result = agg_nodes(&project); + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!( + synthetic.len(), + 1, + "a positionally-mapped sliced reducer must mint one synthetic agg; got: {:?}", + result.aggs + ); + assert_eq!( + synthetic[0].canonical_read_slice(), + vec![ + AxisRead::Iterated { + dim: "state".to_string(), + source_dim: "region".to_string() + }, + AxisRead::Reduced { subset: None } + ] + ); + assert_eq!( + synthetic[0].result_dims, + vec!["State".to_string()], + "the agg's result axis is the TARGET equation's iterated dim" + ); + assert_eq!(source_names(synthetic[0]), vec!["matrix"]); + assert_eq!(synthetic[0].equation_text, "sum(matrix[state, *])"); +} + +/// GH #534 (conservative gate, element-mapped): a sliced reducer over an +/// EXPLICIT element-mapped pair stays un-hoisted -- the executed A2A +/// lowering resolves mapped references positionally and ignores the +/// element map (GH #756), so `mapped_element_correspondence` declines +/// and the reference keeps its conservative shape. +#[test] +fn element_mapped_sliced_reducer_is_not_hoisted() { + let project = TestProject::new("element_mapped_slice") + .named_dimension("Region", &["r1", "r2"]) + .named_dimension("D2", &["x", "y"]) + .named_dimension_with_element_mapping( + "State", + &["s1", "s2"], + "Region", + &[("s1", "r2"), ("s2", "r1")], + ) + .array_aux_direct("matrix", vec!["Region".into(), "D2".into()], "1", None) + .array_aux_direct( + "out", + vec!["State".into()], + "1 + SUM(matrix[State, *])", + None, + ); + + let result = agg_nodes(&project); + assert!( + result.aggs.iter().all(|a| !a.reads_var("matrix")), + "an element-mapped sliced reducer must not be hoisted; got: {:?}", + result.aggs + ); + assert!(result.synthetic_by_key.is_empty()); +} + +/// GH #757 (flipped from the GH #534-era conservative pin): a sliced +/// reducer whose POSITIONAL mapping is declared only in the REVERSE +/// direction (on the source's `Region` toward `State`) is now hoisted -- +/// `classify_axis_access`'s mapped arm gates on +/// `iterated_axis_slot_elements` / `mapped_element_correspondence`, +/// which accepts both declaration directions (the compiler's +/// `translate_via_mapping` resolves both, so declining one direction +/// was pure over-conservatism). The slice and `result_dims` are +/// identical to the forward-declared twin. +#[test] +fn reverse_declared_mapped_sliced_reducer_is_hoisted() { + let project = TestProject::new("reverse_mapped_slice") + .named_dimension_with_mapping("Region", &["r1", "r2"], "State") + .named_dimension("D2", &["x", "y"]) + .named_dimension("State", &["s1", "s2"]) + .array_aux_direct("matrix", vec!["Region".into(), "D2".into()], "1", None) + .array_aux_direct( + "out", + vec!["State".into()], + "1 + SUM(matrix[State, *])", + None, + ); + + let result = agg_nodes(&project); + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!( + synthetic.len(), + 1, + "the reverse-declared positionally-mapped sliced reducer must be hoisted; got: {:?}", + result.aggs + ); + assert_eq!( + synthetic[0].canonical_read_slice(), + vec![ + AxisRead::Iterated { + dim: "state".to_string(), + source_dim: "region".to_string() + }, + AxisRead::Reduced { subset: None } + ] + ); + assert_eq!(synthetic[0].result_dims, vec!["State".to_string()]); +} + +/// GH #534: the whole-RHS twin -- `out[State] = SUM(matrix[State,*])` +/// over a positionally-mapped pair mints a SYNTHETIC agg, an exception +/// to the variable-is-the-agg rule for whole-RHS reducers: the +/// variable-backed link-score path (`try_cross_dimensional_link_scores`) +/// matches result axes against source axes by name, so a remapped pair +/// falls off it onto the `Wildcard` per-shape partial, whose +/// PREVIOUS-wrapping mangles the iterated index into a non-compiling +/// `matrix[PREVIOUS(state),*]` (silently stubbed to 0). The synthetic +/// agg gives the whole-RHS case the same remapped two-half scoring as +/// an inline mapped reducer. +#[test] +fn whole_rhs_mapped_partial_reduce_mints_synthetic_agg() { + let project = TestProject::new("whole_rhs_mapped") + .named_dimension("Region", &["r1", "r2"]) + .named_dimension("D2", &["x", "y"]) + .named_dimension_with_mapping("State", &["s1", "s2"], "Region") + .array_aux_direct("matrix", vec!["Region".into(), "D2".into()], "1", None) + .array_aux_direct("out", vec!["State".into()], "SUM(matrix[State, *])", None); + + let result = agg_nodes(&project); + assert!( + result.aggs.iter().all(|a| a.is_synthetic), + "a whole-RHS MAPPED reducer must mint a synthetic agg (not variable-backed); got: {:?}", + result.aggs + ); + let agg = result + .aggs_in_var("out") + .next() + .expect("expected a synthetic agg owned by `out`"); + assert_eq!(agg.name, "$\u{205A}ltm\u{205A}agg\u{205A}0"); + assert_eq!( + agg.canonical_read_slice(), + vec![ + AxisRead::Iterated { + dim: "state".to_string(), + source_dim: "region".to_string() + }, + AxisRead::Reduced { subset: None } + ] + ); + assert_eq!(agg.result_dims, vec!["State".to_string()]); +} + +/// GH #764 (T4): the whole-RHS BROADCAST twin -- `out[D1,D3] = +/// SUM(matrix[D1,*])`, `result_dims` (`[D1]`) a strict subset of the +/// owner's dims (`[D1,D3]`) -- mints a SYNTHETIC agg, generalizing the +/// GH #534 carve-out: the variable-backed per-`(row, slot)` machinery +/// requires each slot to name a complete `to` element, which a +/// broadcast slot does not. The synthetic agg is arrayed over +/// `result_dims` and rides the two-half emitters + the GH #528 +/// agg-to-target projection. +#[test] +fn whole_rhs_broadcast_partial_reduce_mints_synthetic_agg() { + let project = TestProject::new("whole_rhs_broadcast_764") + .named_dimension("D1", &["a", "b"]) + .named_dimension("D2", &["x", "y"]) + .named_dimension("D3", &["p", "q"]) + .array_aux_direct("matrix", vec!["D1".into(), "D2".into()], "1", None) + .array_aux_direct( + "out", + vec!["D1".into(), "D3".into()], + "SUM(matrix[D1, *])", + None, + ); + + let result = agg_nodes(&project); + assert!( + result.aggs.iter().all(|a| a.is_synthetic), + "a whole-RHS BROADCAST reducer must mint a synthetic agg (not variable-backed); \ + got: {:?}", + result.aggs + ); + let agg = result + .aggs_in_var("out") + .next() + .expect("expected a synthetic agg owned by `out`"); + assert_eq!(agg.name, "$\u{205A}ltm\u{205A}agg\u{205A}0"); + assert_eq!( + agg.canonical_read_slice(), + vec![ + AxisRead::Iterated { + dim: "d1".to_string(), + source_dim: "d1".to_string() + }, + AxisRead::Reduced { subset: None } + ] + ); + assert_eq!(agg.result_dims, vec!["D1".to_string()]); + assert_eq!(source_names(agg), vec!["matrix"]); +} + +/// GH #764 (T4): the whole-RHS PERMUTED twin -- `out[D2,D1] = +/// SUM(cube[D1,D2,*])`, `result_dims` (`[D1,D2]`, slice order) in a +/// different order than the owner's dims (`[D2,D1]`) -- mints a +/// SYNTHETIC agg too: variable-backed slot coordinates are in +/// `Iterated`-axis order, which would mis-subscript `to`. Slots of the +/// synthetic agg are keyed by `result_dims` order, and the GH #528 +/// projection reorders per target element. +#[test] +fn whole_rhs_permuted_partial_reduce_mints_synthetic_agg() { + let project = TestProject::new("whole_rhs_permuted_764") + .named_dimension("D1", &["a", "b"]) + .named_dimension("D2", &["x", "y"]) + .named_dimension("D3", &["p", "q"]) + .array_aux_direct( + "cube", + vec!["D1".into(), "D2".into(), "D3".into()], + "1", + None, + ) + .array_aux_direct( + "out", + vec!["D2".into(), "D1".into()], + "SUM(cube[D1, D2, *])", + None, + ); + + let result = agg_nodes(&project); + assert!( + result.aggs.iter().all(|a| a.is_synthetic), + "a whole-RHS PERMUTED reducer must mint a synthetic agg (not variable-backed); \ + got: {:?}", + result.aggs + ); + let agg = result + .aggs_in_var("out") + .next() + .expect("expected a synthetic agg owned by `out`"); + assert_eq!( + agg.result_dims, + vec!["D1".to_string(), "D2".to_string()], + "result_dims stay in Iterated-axis (slice) order, not the owner's declared order" + ); + assert_eq!( + agg.canonical_read_slice(), + vec![ + AxisRead::Iterated { + dim: "d1".to_string(), + source_dim: "d1".to_string() + }, + AxisRead::Iterated { + dim: "d2".to_string(), + source_dim: "d2".to_string() + }, + AxisRead::Reduced { subset: None } + ] + ); +} + +/// GH #764 ∩ GH #765 (T4): a non-aligned whole-RHS reduce that ALSO +/// carries a `Pinned` axis (`out[D1,D2] = SUM(cube[D1,nyc,*])` over +/// `cube[D1,Region,D2]`) mints a synthetic agg whose slice keeps the +/// `Pinned` axis -- so the synthetic-half emitters (which are +/// Pinned-correct via `read_slice_rows`) score only the read rows. +/// Pre-T4 this shape rode the OLD full-cartesian link-score +/// derivation, scoring unread (`boston`) rows. +#[test] +fn whole_rhs_broadcast_pinned_mix_mints_synthetic_agg() { + let project = TestProject::new("whole_rhs_broadcast_pinned_764") + .named_dimension("D1", &["a", "b"]) + .named_dimension("Region", &["nyc", "boston"]) + .named_dimension("D2", &["x", "y"]) + .array_aux_direct( + "cube", + vec!["D1".into(), "Region".into(), "D2".into()], + "1", + None, + ) + .array_aux_direct( + "out", + vec!["D1".into(), "D2".into()], + "SUM(cube[D1, nyc, *])", + None, + ); + + let result = agg_nodes(&project); + assert!( + result.aggs.iter().all(|a| a.is_synthetic), + "a Pinned-bearing non-aligned whole-RHS reducer must mint a synthetic agg; got: {:?}", + result.aggs + ); + let agg = result + .aggs_in_var("out") + .next() + .expect("expected a synthetic agg owned by `out`"); + assert_eq!(agg.result_dims, vec!["D1".to_string()]); + assert_eq!( + agg.canonical_read_slice(), + vec![ + AxisRead::Iterated { + dim: "d1".to_string(), + source_dim: "d1".to_string() + }, + AxisRead::Pinned("nyc".to_string()), + AxisRead::Reduced { subset: None } + ] + ); +} + +/// GH #534: `iterated_axis_slot_elements` -- identity for the literal +/// case, the positional preimage for a mapped pair, `None` for an +/// element-mapped or unmapped pair. +#[test] +fn iterated_axis_slot_elements_cases() { + use crate::datamodel::{Dimension as DmDimension, DimensionMapping}; + use crate::dimensions::DimensionsContext; + + let named = |name: &str, elems: &[&str], mappings: Vec| { + let mut d = DmDimension::named( + name.to_string(), + elems.iter().map(|e| e.to_string()).collect(), + ); + d.mappings = mappings; + d + }; + let positional = DimensionMapping { + target: "Region".to_string(), + element_map: vec![], + }; + let element_mapped = DimensionMapping { + target: "Region".to_string(), + element_map: vec![ + ("s1".to_string(), "r2".to_string()), + ("s2".to_string(), "r1".to_string()), + ], + }; + + let region_elems = vec!["r1".to_string(), "r2".to_string()]; + + // Literal: identity (no dim_ctx lookups consulted). + let ctx = DimensionsContext::from(&[ + named("Region", &["r1", "r2"], vec![]), + named("State", &["s1", "s2"], vec![positional.clone()]), + ]); + assert_eq!( + iterated_axis_slot_elements("region", "region", ®ion_elems, &ctx), + Some(region_elems.clone()) + ); + + // Positional mapping: source row r1 feeds slot s1, r2 feeds s2 + // (index-identity under the positional correspondence). + assert_eq!( + iterated_axis_slot_elements("state", "region", ®ion_elems, &ctx), + Some(vec!["s1".to_string(), "s2".to_string()]) + ); + + // Explicit element map: declined (GH #756 positional-only gate). + let ctx_elem = DimensionsContext::from(&[ + named("Region", &["r1", "r2"], vec![]), + named("State", &["s1", "s2"], vec![element_mapped]), + ]); + assert_eq!( + iterated_axis_slot_elements("state", "region", ®ion_elems, &ctx_elem), + None + ); + + // Unmapped pair: declined. + let ctx_unmapped = DimensionsContext::from(&[ + named("Region", &["r1", "r2"], vec![]), + named("State", &["s1", "s2"], vec![]), + ]); + assert_eq!( + iterated_axis_slot_elements("state", "region", ®ion_elems, &ctx_unmapped), + None + ); +} + +/// AC4.4 (the carve-out): a reducer over a *dynamic* index +/// (`x[Region] = SUM(pop[idx, *])`, `idx` a scalar aux -- a non-literal +/// index) is NOT statically describable: `compute_read_slice` returns +/// `None` for the `idx` axis, so the reducer is not hoisted and its +/// reference stays on the conservative path. Pin this narrow case. +#[test] +fn dynamic_index_reducer_subexpression_is_not_hoisted() { + let project = TestProject::new("dynamic_index_reducer") + .named_dimension("Region", &["NYC", "Boston"]) + .named_dimension("Age", &["Adult", "Child"]) + .array_aux_direct("pop", vec!["Region".into(), "Age".into()], "10", None) + .scalar_aux("idx", "1") + .array_aux_direct("x", vec!["Region".into()], "SUM(pop[idx, *])", None); + + let result = agg_nodes(&project); + assert!( + result.aggs.iter().all(|a| !a.reads_var("pop")), + "a dynamic-index reducer must not be hoisted; got: {:?}", + result.aggs + ); + assert!(result.synthetic_by_key.is_empty()); +} + +/// AC4.2 (positive guard): a whole-RHS slice/partial reduce +/// (`agg[D1] = SUM(matrix[D1, *])`) IS recognized -- but as a +/// variable-backed agg, not a synthetic one (covered by +/// `whole_rhs_arrayed_partial_reduce_is_its_own_agg`); and an all- +/// wildcard reducer subexpression (`SUM(matrix[*, *])`, no literal pin) +/// is still hoisted as a synthetic agg with an all-`Reduced` slice. +#[test] +fn full_wildcard_reducer_subexpression_is_still_hoisted() { + // `SUM(matrix[*, *])` (all-wildcard, no literal pin) is a full + // reduce and IS hoistable as a synthetic agg. + let project = TestProject::new("full_wildcard_subexpr") + .named_dimension("D1", &["a", "b"]) + .named_dimension("D2", &["x", "y"]) + .array_aux_direct("matrix", vec!["D1".into(), "D2".into()], "1", None) + .scalar_aux("y", "5 + SUM(matrix[*, *])"); + + let result = agg_nodes(&project); + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!( + synthetic.len(), + 1, + "an all-wildcard reducer subexpression must mint one synthetic agg; got: {:?}", + result.aggs + ); + assert_eq!(source_names(synthetic[0]), vec!["matrix"]); + assert_eq!( + synthetic[0].canonical_read_slice(), + vec![ + AxisRead::Reduced { subset: None }, + AxisRead::Reduced { subset: None } + ] + ); + assert!(synthetic[0].result_dims.is_empty()); +} + +/// AC1.2 + GH #982: the consolidated reducer table classifies every array +/// reducer the LTM machinery cares about, and all THREE derived predicates +/// agree with the table row for row. +/// +/// The rows come from [`REDUCER_DECISION_TABLE`], whose 11 entries are +/// derived from `reducer_kind_from_name`'s own arms (see its doc): every +/// arm, every arity guard in both directions, and the catch-all. Each row +/// asserts the name-keyed decider, the builtin-keyed decider, the arity +/// the builtin form reports, and the three consumers -- so the `SIZE` / +/// `RANK` inversion between `reducer_collapses_to_scalar` ("does this +/// collapse to a scalar?") and `builtin_routes_through_agg` ("did an agg +/// get minted for it?") is pinned in both directions on both rows, and an +/// edit to either predicate reds here rather than drifting silently +/// (GH #982). +#[test] +fn reducer_kind_classifies_every_array_reducer() { + for row in REDUCER_DECISION_TABLE { + let name = row.name; + let arity = row.arity; + let builtin = row.builtin(); + assert_eq!( + builtin_reducer_arity(&builtin), + arity, + "{name}/{arity}: the builtin form must report the row's arity" + ); + assert_eq!( + reducer_kind_from_name(name, arity), + row.kind, + "{name}/{arity}: reducer_kind_from_name" + ); + assert_eq!( + reducer_kind(&builtin), + row.kind, + "{name}/{arity}: reducer_kind" + ); + assert_eq!( + reducer_collapses_to_scalar(name, arity), + row.collapses_to_scalar, + "{name}/{arity}: reducer_collapses_to_scalar" + ); + assert_eq!( + reducer_is_hoistable(&builtin), + row.is_hoistable, + "{name}/{arity}: reducer_is_hoistable" + ); + assert_eq!( + builtin_routes_through_agg(&builtin), + row.routes_through_agg, + "{name}/{arity}: builtin_routes_through_agg" + ); + } + + // The table keys `stddev`/`rank`/`size`/`sum` at one arity each, but + // `reducer_kind_from_name` ignores arity for them -- only + // `mean`/`min`/`max` carry an `arity == 1` guard. Spot-check one, so a + // stray arity guard added to an arity-insensitive arm is caught. + assert_eq!( + reducer_kind_from_name("stddev", 7), + Some(ReducerKind::Nonlinear) + ); + assert_eq!( + reducer_kind_from_name("rank", 2), + Some(ReducerKind::Nonlinear) + ); +} + +/// GH #776: a RANK subexpression mints an ARRAY-valued synthetic agg, +/// not a scalar reducer agg. Its result dims are the ranked argument's +/// non-pinned axes, so a bare one-dimensional source ranks over Region. +#[test] +fn rank_subexpression_mints_array_valued_synthetic_agg() { + let project = TestProject::new("rank_array_agg") + .named_dimension("Region", &["north", "south"]) + .array_aux("pop[Region]", "100") + .array_aux("scale[Region]", "pop[Region] * 0.01") + .array_aux("grow[Region]", "scale[Region] * RANK(pop, 1)"); + + let result = agg_nodes(&project); + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!( + synthetic.len(), + 1, + "RANK must mint one synthetic aggregate node; got: {:?}", + result.aggs + ); + assert!(synthetic[0].array_valued_rank); + assert_eq!(synthetic[0].result_dims, vec!["Region"]); + assert_eq!(source_names(synthetic[0]), vec!["pop"]); +} + +/// GH #796 review: multi-source RANK result dims must not depend on +/// `HashMap` iteration order. When several sources share the canonical +/// rank slice but carry differently named mapped axes, choose the first +/// source in canonical source-name order -- the same order `AggSource` +/// emission uses. +#[test] +fn rank_multi_source_result_dims_use_sorted_source_order() { + let project = TestProject::new("rank_multi_source_dim_order") + .named_dimension("Region", &["r1", "r2"]) + .named_dimension_with_mapping("State", &["s1", "s2"], "Region") + // Declare `b` first to make the expected order source-name-based, + // not model declaration order. + .array_aux("b[State]", "1") + .array_aux("a[Region]", "2") + .array_aux("r[Region]", "RANK(a[*] + b[*], 1)"); + + let result = agg_nodes(&project); + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!( + synthetic.len(), + 1, + "multi-source RANK must mint one synthetic aggregate node; got: {:?}", + result.aggs + ); + assert!(synthetic[0].array_valued_rank); + assert_eq!(source_names(synthetic[0]), vec!["a", "b"]); + assert_eq!(synthetic[0].result_dims, vec!["Region"]); +} + +/// GH #796 review: array-valued RANK over a proper StarRange returns the +/// ranked subdimension view. Its synthetic helper must be dimensioned over +/// `Core`, not the parent `Region`, or helper slots and source halves drift. +#[test] +fn rank_star_range_result_dims_preserve_subdimension() { + let project = TestProject::new("rank_star_range_subdimension") + .named_dimension("Region", &["a", "b", "c"]) + .named_dimension("Core", &["a", "b"]) + .array_aux("arr[Region]", "10") + .array_aux("ranking[Core]", "RANK(arr[*:Core], 1)"); + + let result = agg_nodes(&project); + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!( + synthetic.len(), + 1, + "subdimension RANK must mint one synthetic aggregate node; got: {:?}", + result.aggs + ); + assert!(synthetic[0].array_valued_rank); + assert_eq!(synthetic[0].result_dims, vec!["Core"]); +} + +/// GH #796 review: the source-to-RANK slot helper is shared by element +/// graph edges and link-score names. It must fan active/iterated axes out +/// across every RANK output slot and use result-dimension element names, +/// not source-dimension names, for mapped sibling sources. +#[test] +fn rank_output_slots_use_result_dimension_elements() { + let active_dim_read = vec![AxisRead::Iterated { + dim: "region".to_string(), + source_dim: "region".to_string(), + }]; + assert_eq!( + rank_output_slot_parts_for_row( + &active_dim_read, + &[vec!["north".to_string(), "south".to_string()]], + &["north".to_string()], + ), + Some(vec![vec!["north".to_string()], vec!["south".to_string()]]) + ); + + let mapped_source_read = vec![AxisRead::Reduced { subset: None }]; + assert_eq!( + rank_output_slot_parts_for_row( + &mapped_source_read, + &[vec!["r1".to_string(), "r2".to_string()]], + &[], + ), + Some(vec![vec!["r1".to_string()], vec!["r2".to_string()]]) + ); + + let context_plus_ranked_axis = vec![ + AxisRead::Iterated { + dim: "region".to_string(), + source_dim: "region".to_string(), + }, + AxisRead::Reduced { subset: None }, + ]; + assert_eq!( + rank_output_slot_parts_for_row( + &context_plus_ranked_axis, + &[ + vec!["north".to_string(), "south".to_string()], + vec!["x".to_string(), "y".to_string()], + ], + &["north".to_string()], + ), + Some(vec![ + vec!["north".to_string(), "x".to_string()], + vec!["north".to_string(), "y".to_string()] + ]) + ); +} + +/// GH #776 whole-RHS form: `r[Region] = RANK(pop, 1)` uses the same +/// synthetic array-valued helper as the inline spelling, rather than +/// becoming a variable-backed scalar reducer agg. +#[test] +fn rank_whole_rhs_mints_synthetic_not_variable_backed() { + let project = TestProject::new("rank_whole_rhs") + .named_dimension("Region", &["north", "south"]) + .array_aux("pop[Region]", "100") + .array_aux("r[Region]", "RANK(pop, 1)"); + + let result = agg_nodes(&project); + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!( + synthetic.len(), + 1, + "whole-RHS RANK must mint one synthetic aggregate node; got: {:?}", + result.aggs + ); + assert!(synthetic[0].array_valued_rank); + assert!(result.aggs.iter().all(|a| a.is_synthetic)); +} + +/// GH #766: a StarRange naming a dimension that is NEITHER the axis's +/// own dimension NOR a proper subdimension of it (at best a mid-edit +/// inconsistency) DECLINES the hoist -- it must not silently widen to +/// the full extent. The reducer stays on the conservative path. +#[test] +fn star_range_non_subdimension_declines_hoist() { + let project = TestProject::new("star_range_decline") + .named_dimension("Region", &["a", "b", "c"]) + .named_dimension("Other", &["p", "q"]) + .array_aux("arr[Region]", "10") + .scalar_aux("x", "1 + MEAN(arr[*:Other])"); + + let result = agg_nodes(&project); + assert!( + result.aggs.is_empty(), + "a non-subdimension StarRange must decline the hoist; got: {:?}", + result.aggs + ); +} + +/// GH #766: a StarRange over the axis's OWN dimension (`SUM(arr[*:Region])` +/// where `arr` is declared over `Region`) is the full extent -- +/// `Reduced{subset: None}`, byte-identical to a plain `*`. +#[test] +fn star_range_own_dimension_is_full_extent() { + let project = TestProject::new("star_range_own_dim") + .named_dimension("Region", &["a", "b", "c"]) + .array_aux("arr[Region]", "10") + .scalar_aux("x", "1 + SUM(arr[*:Region])"); + + let result = agg_nodes(&project); + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!(synthetic.len(), 1, "got: {:?}", result.aggs); + assert_eq!( + synthetic[0].canonical_read_slice(), + vec![AxisRead::Reduced { subset: None }] + ); +} + +/// GH #766: a StarRange over a PROPER subdimension carries the +/// subdimension's elements as the `Reduced` subset (canonical names, in +/// subdimension-declared order, resolved via `SubdimensionRelation`). +#[test] +fn star_range_proper_subdimension_carries_subset() { + let project = TestProject::new("star_range_subset") + .named_dimension("Region", &["a", "b", "c"]) + .named_dimension("Core", &["a", "b"]) + .array_aux("arr[Region]", "10") + .scalar_aux("x", "1 + MEAN(arr[*:Core])"); + + let result = agg_nodes(&project); + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!(synthetic.len(), 1, "got: {:?}", result.aggs); + assert_eq!( + synthetic[0].canonical_read_slice(), + vec![AxisRead::Reduced { + subset: Some(vec!["a".to_string(), "b".to_string()]) + }] + ); + assert!(synthetic[0].result_dims.is_empty()); +} + +/// GH #766 (composition): a subset StarRange composes with an iterated +/// axis -- `out[D1] = 1 + SUM(matrix[D1, *:SubD2])` hoists a synthetic +/// agg whose slice is `[Iterated(d1), Reduced{subset}]` and whose +/// `result_dims` carry `D1`. +#[test] +fn star_range_subset_composes_with_iterated_axis() { + let project = TestProject::new("star_range_iterated_subset") + .named_dimension("D1", &["a", "b"]) + .named_dimension("D2", &["x", "y", "z"]) + .named_dimension("SubD2", &["x", "y"]) + .array_aux_direct("matrix", vec!["D1".into(), "D2".into()], "1", None) + .array_aux_direct( + "out", + vec!["D1".into()], + "1 + SUM(matrix[D1, *:SubD2])", + None, + ); + + let result = agg_nodes(&project); + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!(synthetic.len(), 1, "got: {:?}", result.aggs); + assert_eq!( + synthetic[0].canonical_read_slice(), + vec![ + AxisRead::Iterated { + dim: "d1".to_string(), + source_dim: "d1".to_string() + }, + AxisRead::Reduced { + subset: Some(vec!["x".to_string(), "y".to_string()]) + } + ] + ); + assert_eq!(synthetic[0].result_dims, vec!["D1".to_string()]); +} + +/// Test helper: resolve the named dimensions of a synced project into +/// `Dimension` objects (for the gate's `to_dims` argument). +fn resolve_dims( + db: &SimlinDb, + project: crate::db::SourceProject, + names: &[&str], +) -> Vec { + let dim_ctx = crate::db::project_dimensions_context(db, project); + names + .iter() + .map(|n| { + dim_ctx + .get(&crate::common::CanonicalDimensionName::from_raw(n)) + .unwrap_or_else(|| panic!("dimension {n} resolves")) + .clone() + }) + .collect() +} + +/// GH #766 x T3: a VARIABLE-BACKED partial reduce whose slice carries a +/// SUBSET (`out[D1] = SUM(matrix[D1,*:SubD2])` as the whole RHS) is +/// ACCEPTED by the reduce gate: `try_cross_dimensional_link_scores` +/// derives co-reduced rows from the same `read_slice_rows`, so the +/// subset edges pair with subset divisors. (Pre-T3 the slice was +/// excluded onto the loud conservative regime because the score +/// derivation enumerated the full cartesian.) +#[test] +fn variable_backed_subset_slice_is_accepted_by_reduce_gate() { + let project = TestProject::new("vb_subset_accepted") + .named_dimension("D1", &["a", "b"]) + .named_dimension("D2", &["x", "y", "z"]) + .named_dimension("SubD2", &["x", "y"]) + .array_aux_direct("matrix", vec!["D1".into(), "D2".into()], "1", None) + .array_aux_direct("out", vec!["D1".into()], "SUM(matrix[D1, *:SubD2])", None); + + let datamodel = project.build_datamodel(); + let db = SimlinDb::default(); + let sync = sync_from_datamodel(&db, &datamodel); + let result = enumerate_agg_nodes(&db, sync.models["main"].source, sync.project); + + // The variable-backed agg exists and carries the subset... + let agg = result + .aggs_in_var("out") + .find(|a| a.name == "out") + .expect("expected a variable-backed agg owned by `out`"); + assert!(matches!( + &agg.canonical_read_slice()[1], + AxisRead::Reduced { subset: Some(s) } if s == &["x".to_string(), "y".to_string()] + )); + + // ...and the gate admits it (T3 of the shape-expressiveness design). + let to_dims = resolve_dims(&db, sync.project, &["d1"]); + let accepted = variable_backed_reduce_agg(result, "matrix", "out", &to_dims) + .expect("the subset-bearing aligned slice must be admitted by the reduce gate"); + assert_eq!(accepted.name, "out"); +} + +/// GH #765 x T3: a VARIABLE-BACKED Pinned-mixed aligned slice +/// (`outf[D1] = MEAN(cube[D1,x,*])`) is ACCEPTED by the reduce gate -- +/// the T1-era Pinned exclusion is deleted atomically with the +/// `read_slice_rows` derivation swap. +#[test] +fn reduce_gate_accepts_pinned_mixed_aligned_slice() { + let project = TestProject::new("gate_pinned_mixed") + .named_dimension("D1", &["a", "b"]) + .named_dimension("D2", &["x", "y"]) + .named_dimension("D3", &["p", "q"]) + .array_aux_direct( + "cube", + vec!["D1".into(), "D2".into(), "D3".into()], + "1", + None, + ) + .array_aux_direct("outf", vec!["D1".into()], "MEAN(cube[D1, x, *])", None); + + let datamodel = project.build_datamodel(); + let db = SimlinDb::default(); + let sync = sync_from_datamodel(&db, &datamodel); + let result = enumerate_agg_nodes(&db, sync.models["main"].source, sync.project); + + let to_dims = resolve_dims(&db, sync.project, &["d1"]); + let accepted = variable_backed_reduce_agg(result, "cube", "outf", &to_dims) + .expect("the Pinned-mixed aligned slice must be admitted by the reduce gate"); + assert_eq!(accepted.name, "outf"); +} + +/// Section 6 (scalar owner): a scalar-result Pinned slice +/// (`total = SUM(pop[nyc,*])`, `to_dims` empty) is admitted -- the slot +/// is the bare `total` node, so `emit_agg_routed_edges` emits exactly +/// the read rows into `to`, matching the per-read-row scores. +#[test] +fn reduce_gate_accepts_scalar_owner_pinned_slice() { + let project = TestProject::new("gate_scalar_owner_pinned") + .named_dimension("Region", &["nyc", "boston"]) + .named_dimension("D2", &["p", "q"]) + .array_aux_direct("pop", vec!["Region".into(), "D2".into()], "1", None) + .scalar_aux("total", "SUM(pop[nyc, *])"); + + let datamodel = project.build_datamodel(); + let db = SimlinDb::default(); + let sync = sync_from_datamodel(&db, &datamodel); + let result = enumerate_agg_nodes(&db, sync.models["main"].source, sync.project); + + let accepted = variable_backed_reduce_agg(result, "pop", "total", &[]) + .expect("the scalar-owner Pinned slice must be admitted by the reduce gate"); + assert_eq!(accepted.name, "total"); +} + +/// Section 6 (inert skip): a PURE full-extent scalar reduce +/// (`total = SUM(pop[*])`) stays OUT of the gate -- the reference +/// walker's reduction edges are already the true reads, so routing it +/// through the gate would change nothing and is skipped to keep the +/// diff inert (byte-identity). +#[test] +fn reduce_gate_declines_pure_full_extent_slice() { + let project = TestProject::new("gate_full_extent") + .named_dimension("Region", &["nyc", "boston"]) + .array_aux("pop[Region]", "1") + .scalar_aux("total", "SUM(pop[*])"); + + let datamodel = project.build_datamodel(); + let db = SimlinDb::default(); + let sync = sync_from_datamodel(&db, &datamodel); + let result = enumerate_agg_nodes(&db, sync.models["main"].source, sync.project); + + assert!( + variable_backed_reduce_agg(result, "pop", "total", &[]).is_none(), + "a pure full-extent slice keeps the reference walker's edges (inert skip)" + ); +} + +/// GH #777: an ARRAYED-owner scalar-result Pinned slice +/// (`share[Region] = SUM(pop[nyc,*])` -- no `Iterated` axis, arrayed +/// `to`) is ADMITTED: the single scalar reducer value broadcasts over +/// the owner's dims, and the per-(read-row, full-target-element) +/// machinery (`emit_agg_routed_edges`' broadcast fan-out + +/// `try_cross_dimensional_link_scores`' broadcast-reduce branch) names +/// every slot. +#[test] +fn reduce_gate_admits_arrayed_owner_scalar_result_slice() { + let project = TestProject::new("gate_broadcast_pinned") + .named_dimension("Region", &["nyc", "boston"]) + .named_dimension("D2", &["p", "q"]) + .array_aux_direct("pop", vec!["Region".into(), "D2".into()], "1", None) + .array_aux_direct("share", vec!["Region".into()], "SUM(pop[nyc, *])", None); + + let datamodel = project.build_datamodel(); + let db = SimlinDb::default(); + let sync = sync_from_datamodel(&db, &datamodel); + let result = enumerate_agg_nodes(&db, sync.models["main"].source, sync.project); + + let to_dims = resolve_dims(&db, sync.project, &["region"]); + let accepted = variable_backed_reduce_agg(result, "pop", "share", &to_dims) + .expect("the arrayed-owner broadcast slice must be admitted (GH #777)"); + assert_eq!(accepted.name, "share"); + assert!( + accepted.result_dims.is_empty(), + "the broadcast reducer's result is scalar (no Iterated axis); got: {:?}", + accepted.result_dims + ); +} + +/// GH #764 boundary (T4): a partial reduce BROADCAST over extra target +/// dims (`out[D1,D3] = SUM(matrix[D1,*])` -- `result_dims` a strict +/// subset of `to`'s dims) never reaches the variable-backed gate at all +/// anymore: T4's minting condition routes it to a SYNTHETIC agg, so +/// `variable_backed_reduce_agg` finds no variable-backed candidate (its +/// Iterated-arm alignment check stays as defense). +#[test] +fn reduce_gate_declines_broadcast_result_dims() { + let project = TestProject::new("gate_broadcast_result") + .named_dimension("D1", &["a", "b"]) + .named_dimension("D2", &["x", "y"]) + .named_dimension("D3", &["p", "q"]) + .array_aux_direct("matrix", vec!["D1".into(), "D2".into()], "1", None) + .array_aux_direct( + "out", + vec!["D1".into(), "D3".into()], + "SUM(matrix[D1, *])", + None, + ); + + let datamodel = project.build_datamodel(); + let db = SimlinDb::default(); + let sync = sync_from_datamodel(&db, &datamodel); + let result = enumerate_agg_nodes(&db, sync.models["main"].source, sync.project); + + let to_dims = resolve_dims(&db, sync.project, &["d1", "d3"]); + assert!( + variable_backed_reduce_agg(result, "matrix", "out", &to_dims).is_none(), + "a broadcast whole-RHS reduce must have no variable-backed agg (GH #764)" + ); + assert!( + result.aggs.iter().all(|a| a.is_synthetic), + "T4 mints a synthetic agg for the broadcast shape; got: {:?}", + result.aggs + ); +} + +/// GH #766 / invariant I3 (uniqueness of the full-extent form): a +/// StarRange naming a SAME-CARDINALITY "subdimension" -- including a +/// permuted alias of the axis's element set (`Alias = [c, a, b]` over +/// `Region = [a, b, c]`: containment + equal size means the same element +/// SET) -- normalizes to `Reduced{subset: None}`, never a `Some` subset +/// covering the whole axis. Reduction order is irrelevant (the reduced +/// rows are a set), and keeping the full-extent representation unique +/// means downstream byte-identity does not depend on which spelling the +/// modeler used. +#[test] +fn star_range_same_cardinality_alias_normalizes_to_full_extent() { + let project = TestProject::new("star_range_alias") + .named_dimension("Region", &["a", "b", "c"]) + .named_dimension("Alias", &["c", "a", "b"]) + .array_aux("arr[Region]", "10") + .scalar_aux("x", "1 + SUM(arr[*:Alias])"); + + let result = agg_nodes(&project); + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!(synthetic.len(), 1, "got: {:?}", result.aggs); + assert_eq!( + synthetic[0].canonical_read_slice(), + vec![AxisRead::Reduced { subset: None }], + "a whole-axis alias must normalize to the unique full-extent form" + ); +} + +/// GH #766 (indexed dimensions): a StarRange over an INDEXED +/// subdimension (`SubIndex(3)` with declared `parent = Index(5)`, which +/// maps to the parent's first 3 elements) resolves the subset through +/// the same `SubdimensionRelation` path as named dimensions -- the +/// subset elements are the canonical indexed names `"1".."3"`, matching +/// `dimension_element_names`'s output. +#[test] +fn star_range_indexed_subdimension_carries_subset() { + let project = TestProject::new("star_range_indexed_subdim") + .indexed_dimension("Index", 5) + .indexed_subdimension("SubIndex", 3, "Index") + .array_aux("arr[Index]", "10") + .scalar_aux("x", "1 + MEAN(arr[*:SubIndex])"); + + let result = agg_nodes(&project); + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!(synthetic.len(), 1, "got: {:?}", result.aggs); + assert_eq!( + synthetic[0].canonical_read_slice(), + vec![AxisRead::Reduced { + subset: Some(vec!["1".to_string(), "2".to_string(), "3".to_string()]) + }] + ); +} + +// --- T2 (shape-expressiveness design): per-source `AggNode` invariant +// pins -- the per-source REPRESENTATION invariants (I2, I3b, sorted +// ordering) and the declines `accept_source_slices` enforces. I1's +// *feeder clause* pins (deferred from T2 to avoid the GH #739 vacuity +// trap) landed with T5's RED fixtures, in the section above. + +/// T2 / I3b ordering: `sources` is sorted by canonical variable name +/// regardless of AST occurrence order -- `SUM(b[*] + a[*])` (with `b` +/// first in the argument) still yields `[a, b]`, so salsa cache +/// equality and downstream emission order never depend on how the +/// modeler spelled the argument. +#[test] +fn multi_source_sources_are_sorted_by_var_name() { + let project = TestProject::new("sorted_sources") + .named_dimension("D", &["p", "q"]) + .array_aux_direct("a", vec!["D".into()], "1", None) + .array_aux_direct("b", vec!["D".into()], "2", None) + .scalar_aux("total", "1 + SUM(b[*] + a[*])"); + + let result = agg_nodes(&project); + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!(synthetic.len(), 1, "got: {:?}", result.aggs); + assert_eq!( + source_names(synthetic[0]), + vec!["a", "b"], + "sources must be sorted by canonical name, not AST occurrence order" + ); +} + +/// T2 / I3b dedup: the same variable referenced twice with the SAME +/// slice (`SUM(a[*] + a[*])`) collapses to ONE `AggSource` -- the +/// by-name downstream consumers (`aggs_in_var` routing, the +/// half-emitters) key `sources` on the variable name, so a duplicate +/// entry would make them ambiguous. +#[test] +fn duplicate_var_same_slice_collapses_to_one_source() { + let project = TestProject::new("dup_var_same_slice") + .named_dimension("D", &["p", "q"]) + .array_aux_direct("a", vec!["D".into()], "1", None) + .scalar_aux("total", "1 + SUM(a[*] + a[*])"); + + let result = agg_nodes(&project); + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!(synthetic.len(), 1, "got: {:?}", result.aggs); + assert_eq!( + source_names(synthetic[0]), + vec!["a"], + "a variable read twice with the same slice is one AggSource" + ); + assert_eq!( + synthetic[0].sources[0].read_slice, + vec![AxisRead::Reduced { subset: None }] + ); +} + +/// T2 / I3b decline: the same variable referenced with two DIFFERENT +/// slices (`SUM(a[*] + a[p])` -- `[Reduced]` vs `[Pinned(p)]`) declines +/// the hoist -- since T5, `accept_source_slices`' per-variable +/// one-slice check (the I3b clause); the pin keeps I3b from regressing +/// under the widened per-source acceptance. +#[test] +fn duplicate_var_with_conflicting_slices_declines_hoist() { + let project = TestProject::new("dup_var_conflicting") + .named_dimension("D", &["p", "q"]) + .array_aux_direct("a", vec!["D".into()], "1", None) + .scalar_aux("total", "1 + SUM(a[*] + a[p])"); + + let result = agg_nodes(&project); + assert!( + result.aggs.iter().all(|ag| !ag.reads_var("a")), + "one variable with two different slices must decline the hoist; got: {:?}", + result.aggs + ); + assert!(result.synthetic_by_key.is_empty()); +} + +/// T2 / I1 decline (GREEN characterization): two co-sources with +/// DIFFERING `Reduced` subsets (`SUM(a[*:Sub1] + b[*:Sub2])`) decline +/// the hoist -- their co-reduced rows per slot would disagree, so no +/// canonical slice exists. Enforced by `accept_source_slices`' +/// co-source-identity clause (subset is part of `AxisRead` equality). +#[test] +fn differing_reduced_subsets_decline_hoist() { + let project = TestProject::new("differing_subsets") + .named_dimension("Region", &["a", "b", "c"]) + .named_dimension("Sub1", &["a", "b"]) + .named_dimension("Sub2", &["b", "c"]) + .array_aux("p[Region]", "1") + .array_aux("q[Region]", "2") + .scalar_aux("total", "1 + SUM(p[*:Sub1] + q[*:Sub2])"); + + let result = agg_nodes(&project); + assert!( + result + .aggs + .iter() + .all(|ag| !ag.reads_var("p") && !ag.reads_var("q")), + "co-sources with differing Reduced subsets must decline the hoist; got: {:?}", + result.aggs + ); + assert!(result.synthetic_by_key.is_empty()); +} + +/// T2 / I1 positive twin: two co-sources with the SAME `Reduced` subset +/// (`SUM(p[*:Sub] + q[*:Sub])`) hoist one agg whose every source +/// carries the identical subset-bearing canonical slice. +#[test] +fn agreeing_reduced_subsets_hoist_with_shared_subset() { + let project = TestProject::new("agreeing_subsets") + .named_dimension("Region", &["a", "b", "c"]) + .named_dimension("Sub", &["a", "b"]) + .array_aux("p[Region]", "1") + .array_aux("q[Region]", "2") + .scalar_aux("total", "1 + SUM(p[*:Sub] + q[*:Sub])"); + + let result = agg_nodes(&project); + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!(synthetic.len(), 1, "got: {:?}", result.aggs); + assert_eq!(source_names(synthetic[0]), vec!["p", "q"]); + let expected = vec![AxisRead::Reduced { + subset: Some(vec!["a".to_string(), "b".to_string()]), + }]; + for s in &synthetic[0].sources { + assert_eq!(s.read_slice, expected, "source {} slice", s.var); + } +} + +/// T2 / I2 + the scalar-feeder representation: a scalar feeder of a +/// hoisted reducer (`scale` in `SUM(pop[*] * scale)`, GH #737) IS a +/// source -- the routing filter and the element graph's scalar-feeder +/// arm key on membership -- and carries an EMPTY read slice (one +/// `AxisRead` per axis, and a scalar has none), while the arrayed +/// co-source's slice has one entry per its declared axis. +#[test] +fn scalar_feeder_source_carries_empty_slice() { + let project = TestProject::new("scalar_feeder") + .named_dimension("Region", &["NYC", "Boston"]) + .array_aux("pop[Region]", "100") + .scalar_aux("scale", "0.5") + .scalar_aux("total", "1 + SUM(pop[*] * scale)"); + + let result = agg_nodes(&project); + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!(synthetic.len(), 1, "got: {:?}", result.aggs); + assert_eq!(source_names(synthetic[0]), vec!["pop", "scale"]); + // I2: one AxisRead per the source's OWN declared axes. + assert_eq!( + synthetic[0].source_read_slice("pop"), + vec![AxisRead::Reduced { subset: None }], + "the arrayed co-source's slice has one entry per its axis" + ); + assert!( + synthetic[0].source_read_slice("scale").is_empty(), + "a scalar feeder has no axes, so its slice is empty" + ); + // The canonical slice skips the feeder's empty slice. + assert_eq!( + synthetic[0].canonical_read_slice(), + vec![AxisRead::Reduced { subset: None }] + ); + // And the defensive non-source lookup is the empty slice too. + assert!(synthetic[0].source_read_slice("absent").is_empty()); + assert!(synthetic[0].reads_var("scale")); + assert!(!synthetic[0].reads_var("absent")); +} + +// -- T5 / GH #767: the I1 FEEDER clause ------------------------------- +// +// These pins were deliberately deferred from T2 (pinning them before the +// acceptance widened would have been vacuous -- the GH #739 trap). They +// land with T5's RED fixtures: an iterated-dim projection feeder is +// accepted as an `AggSource` with ITS OWN slice, the canonical slice is +// the co-source (Reduced-bearing) slice regardless of source sort order, +// and everything outside the projection rule still declines. + +/// T5 / I1 feeder clause (GH #767): the iterated-dim-feeder reducer +/// `1 + SUM(matrix[D1,*] * frac[D1])` (inline => synthetic) IS hoisted: +/// `matrix` is the co-source carrying the canonical +/// `[Iterated, Reduced]` slice, `frac` is a projection feeder carrying +/// its OWN `[Iterated]` slice. `frac` sorts BEFORE `matrix`, so this +/// also pins the `canonical_read_slice` contract fix: the canonical +/// slice is the first slice WITH a `Reduced` axis, never an +/// alphabetically-first feeder slice. +#[test] +fn iterated_dim_feeder_projection_hoists_with_per_source_slices() { + let project = TestProject::new("feeder_projection") + .named_dimension("D1", &["r1", "r2"]) + .named_dimension("D2", &["c1", "c2"]) + .array_aux_direct("matrix", vec!["D1".into(), "D2".into()], "5", None) + .array_aux("frac[D1]", "0.5") + .array_aux("growth[D1]", "1 + SUM(matrix[D1, *] * frac[D1])"); + + let result = agg_nodes(&project); + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!( + synthetic.len(), + 1, + "the projection-feeder reducer must be hoisted (GH #767); got: {:?}", + result.aggs + ); + let agg = synthetic[0]; + assert_eq!(source_names(agg), vec!["frac", "matrix"]); + assert_eq!( + agg.source_read_slice("frac"), + vec![AxisRead::Iterated { + dim: "d1".to_string(), + source_dim: "d1".to_string() + }], + "the feeder carries its OWN projection slice" + ); + assert_eq!( + agg.source_read_slice("matrix"), + vec![ + AxisRead::Iterated { + dim: "d1".to_string(), + source_dim: "d1".to_string() + }, + AxisRead::Reduced { subset: None } + ], + "the co-source carries the canonical slice" + ); + // The contract fix: even though `frac` sorts first, the canonical + // slice is the Reduced-bearing co-source slice. + assert_eq!(agg.canonical_read_slice(), agg.source_read_slice("matrix")); + assert_eq!(agg.result_dims, vec!["D1".to_string()]); +} + +/// T5 / I1: the WHOLE-RHS form of the feeder shape +/// (`growth[D1] = SUM(matrix[D1,*] * frac[D1])`, the GH #743/#767 +/// fixture) is VARIABLE-BACKED -- the canonical (co-source) slice is +/// aligned with the owner's dims, so the variable IS the agg and no +/// synthetic is minted. +#[test] +fn iterated_dim_feeder_whole_rhs_is_variable_backed() { + let project = TestProject::new("feeder_whole_rhs") + .named_dimension("D1", &["r1", "r2"]) + .named_dimension("D2", &["c1", "c2"]) + .array_aux_direct("matrix", vec!["D1".into(), "D2".into()], "5", None) + .array_aux("frac[D1]", "0.5") + .array_aux("growth[D1]", "SUM(matrix[D1, *] * frac[D1])"); + + let result = agg_nodes(&project); + assert!(result.synthetic_by_key.is_empty(), "got: {:?}", result.aggs); + let vb: Vec<&AggNode> = result.aggs.iter().filter(|a| !a.is_synthetic).collect(); + assert_eq!(vb.len(), 1, "got: {:?}", result.aggs); + assert_eq!(vb[0].name, "growth"); + assert_eq!(source_names(vb[0]), vec!["frac", "matrix"]); + assert_eq!(vb[0].result_dims, vec!["D1".to_string()]); + assert!(vb[0].source_is_projection_feeder("frac")); + assert!(!vb[0].source_is_projection_feeder("matrix")); +} + +/// T5 / I1: a feeder combined with a SCALAR feeder still hoists -- +/// `SUM(matrix[D1,*] * frac[D1] * scale)` has three sources, the scalar +/// one with an empty slice (it is NOT a projection feeder: the +/// changed-last machinery for scalar feeders is `generate_scalar_feeder_ +/// to_agg_equation`, not the per-row form). +#[test] +fn projection_feeder_and_scalar_feeder_combo_hoists() { + let project = TestProject::new("feeder_combo") + .named_dimension("D1", &["r1", "r2"]) + .named_dimension("D2", &["c1", "c2"]) + .array_aux_direct("matrix", vec!["D1".into(), "D2".into()], "5", None) + .array_aux("frac[D1]", "0.5") + .scalar_aux("scale", "2") + .array_aux("growth[D1]", "1 + SUM(matrix[D1, *] * frac[D1] * scale)"); + + let result = agg_nodes(&project); + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!(synthetic.len(), 1, "got: {:?}", result.aggs); + let agg = synthetic[0]; + assert_eq!(source_names(agg), vec!["frac", "matrix", "scale"]); + assert!(agg.source_read_slice("scale").is_empty()); + assert!(agg.source_is_projection_feeder("frac")); + assert!(!agg.source_is_projection_feeder("scale")); +} + +/// T5 / I1 (review MINOR-5): a PINNED-bearing CANONICAL slice is within +/// the feeder clause's scope -- the clause keys only on the canonical +/// slice's Iterated target dims, so `SUM(cube[D1, c1, *] * frac[D1])` +/// hoists with canonical `[Iterated, Pinned(c1), Reduced]` and the +/// feeder's own `[Iterated]` projection. (It is the FEEDER's slice that +/// must be Iterated-only, not the canonical one.) +#[test] +fn pinned_bearing_canonical_with_feeder_hoists() { + let project = TestProject::new("pinned_canonical_feeder") + .named_dimension("D1", &["r1", "r2"]) + .named_dimension("D2", &["c1", "c2"]) + .named_dimension("D3", &["k1", "k2"]) + .array_aux_direct( + "cube", + vec!["D1".into(), "D2".into(), "D3".into()], + "5", + None, + ) + .array_aux("frac[D1]", "0.5") + .array_aux("growth[D1]", "1 + SUM(cube[D1, c1, *] * frac[D1])"); + + let result = agg_nodes(&project); + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!(synthetic.len(), 1, "got: {:?}", result.aggs); + let agg = synthetic[0]; + assert_eq!( + agg.source_read_slice("cube"), + vec![ + AxisRead::Iterated { + dim: "d1".to_string(), + source_dim: "d1".to_string() + }, + AxisRead::Pinned("c1".to_string()), + AxisRead::Reduced { subset: None } + ] + ); + assert!(agg.source_is_projection_feeder("frac")); + assert_eq!(agg.result_dims, vec!["D1".to_string()]); +} + +/// T5 / I1 decline: a no-`Reduced` source with a PINNED axis is NOT a +/// projection feeder (the design's clause: a feeder slice consists ONLY +/// of `Iterated` axes) -- the hoist declines. +#[test] +fn feeder_with_pinned_axis_declines_hoist() { + let project = TestProject::new("feeder_pinned") + .named_dimension("D1", &["r1", "r2"]) + .named_dimension("D2", &["c1", "c2"]) + .array_aux_direct("matrix", vec!["D1".into(), "D2".into()], "5", None) + .array_aux_direct("w", vec!["D1".into(), "D2".into()], "0.5", None) + .array_aux("growth[D1]", "1 + SUM(matrix[D1, *] * w[D1, c1])"); + + let result = agg_nodes(&project); + assert!( + result.aggs.iter().all(|a| !a.reads_var("matrix")), + "a Pinned-axis no-Reduced source must decline the hoist; got: {:?}", + result.aggs + ); +} + +/// T5 / I1 decline: a feeder whose Iterated dims are a PROPER SUBSET of +/// the canonical slice's Iterated dims declines -- its rows are not 1:1 +/// with the agg result slots (one feeder row would feed every slot it +/// projects from, a broadcast the per-`(row, slot)` machinery cannot +/// name). Documented residual: the design's I1 wording ("drawn from the +/// canonical Iterated target-dim set") is implemented as ordered +/// EQUALITY for exactly this reason. +#[test] +fn feeder_with_subset_iterated_dims_declines_hoist() { + let project = TestProject::new("feeder_subset") + .named_dimension("D1", &["r1", "r2"]) + .named_dimension("D2", &["c1", "c2"]) + .named_dimension("D3", &["x", "y"]) + .array_aux_direct( + "cube", + vec!["D1".into(), "D2".into(), "D3".into()], + "5", + None, + ) + .array_aux("w[D1]", "0.5") + .array_aux_direct( + "growth", + vec!["D1".into(), "D2".into()], + "1 + SUM(cube[D1, D2, *] * w[D1])", + None, + ); + + let result = agg_nodes(&project); + assert!( + result.aggs.iter().all(|a| !a.reads_var("cube")), + "a proper-subset feeder must decline the hoist; got: {:?}", + result.aggs + ); +} + +/// T5 / I1 decline: a feeder whose Iterated dims are a PERMUTATION of +/// the canonical order declines -- `read_slice_rows` derives slot +/// coordinates in the source's axis order, so a permuted feeder's slots +/// would mis-name the agg's `result_dims`-ordered slots. +#[test] +fn feeder_with_permuted_iterated_dims_declines_hoist() { + let project = TestProject::new("feeder_permuted") + .named_dimension("D1", &["r1", "r2"]) + .named_dimension("D2", &["c1", "c2"]) + .named_dimension("D3", &["x", "y"]) + .array_aux_direct( + "cube", + vec!["D1".into(), "D2".into(), "D3".into()], + "5", + None, + ) + .array_aux_direct("w", vec!["D2".into(), "D1".into()], "0.5", None) + .array_aux_direct( + "growth", + vec!["D1".into(), "D2".into()], + "1 + SUM(cube[D1, D2, *] * w[D2, D1])", + None, + ); + + let result = agg_nodes(&project); + assert!( + result.aggs.iter().all(|a| !a.reads_var("cube")), + "a permuted feeder must decline the hoist; got: {:?}", + result.aggs + ); +} + +/// T5 / I1 decline: a MAPPED Iterated axis (GH #534) anywhere in the +/// combination declines the feeder clause -- pinning the slot element +/// into the equation text reads the TARGET-dim element, which is not +/// the source row a mapped reference reads, so the changed-last feeder +/// equation would mis-pin. The mapped sliced reducer WITHOUT a feeder +/// stays hoisted (the GH #534 path is unchanged). +#[test] +fn mapped_iterated_axis_with_feeder_declines_hoist() { + let project = TestProject::new("feeder_mapped") + .named_dimension("Region", &["r1", "r2"]) + .named_dimension("D2", &["x", "y"]) + .named_dimension_with_mapping("State", &["s1", "s2"], "Region") + .array_aux_direct("matrix", vec!["Region".into(), "D2".into()], "1", None) + .array_aux_direct("frac", vec!["State".into()], "0.5", None) + .array_aux_direct( + "out", + vec!["State".into()], + "1 + SUM(matrix[State, *] * frac[State])", + None, + ); + + let result = agg_nodes(&project); + assert!( + result.aggs.iter().all(|a| !a.reads_var("matrix")), + "a mapped iterated axis with a feeder must decline the hoist; got: {:?}", + result.aggs + ); +} + +/// T5 / I3b decline: the same variable appearing as both a co-source +/// and a feeder-shaped reference (`SUM(matrix[D1,*] * matrix[D1,c1])`) +/// declines -- one variable, two different slices. +#[test] +fn duplicate_var_as_co_source_and_feeder_declines_hoist() { + let project = TestProject::new("dup_co_source_feeder") + .named_dimension("D1", &["r1", "r2"]) + .named_dimension("D2", &["c1", "c2"]) + .array_aux_direct("matrix", vec!["D1".into(), "D2".into()], "5", None) + .array_aux("growth[D1]", "1 + SUM(matrix[D1, *] * matrix[D1, c1])"); + + let result = agg_nodes(&project); + assert!( + result.aggs.iter().all(|a| !a.reads_var("matrix")), + "one variable with co-source AND feeder slices must decline; got: {:?}", + result.aggs + ); +} + +/// T5 / I1 decline: two CO-SOURCES (both Reduced-bearing) with +/// differing slices still decline, exactly as before the feeder clause +/// -- the clause widens acceptance only for no-`Reduced` projections. +#[test] +fn co_sources_with_differing_slices_still_decline() { + let project = TestProject::new("co_source_differ") + .named_dimension("D1", &["r1", "r2"]) + .named_dimension("D2", &["c1", "c2"]) + .array_aux_direct("a", vec!["D1".into(), "D2".into()], "1", None) + .array_aux_direct("b", vec!["D2".into(), "D1".into()], "2", None) + .array_aux("growth[D1]", "1 + SUM(a[D1, *] + b[*, D1])"); + + let result = agg_nodes(&project); + assert!( + result + .aggs + .iter() + .all(|ag| !ag.reads_var("a") && !ag.reads_var("b")), + "co-sources with differing slices must still decline; got: {:?}", + result.aggs + ); +} + +/// PR #784 review (P3), purely defensive: every arrayed reducer source +/// has a `per_var` slice by construction (`collect_var_refs` and +/// `collect_arrayed_source_slices` walk the identical reference +/// surface), but if that invariant ever broke, [`agg_sources`] must +/// DECLINE the hoist (`None` -- the reference stays on the conservative +/// Direct path) rather than silently substituting the CANONICAL slice: +/// for a projection feeder (whose slice differs from canonical by +/// design, GH #767) that substitution would mislabel the feeder as a +/// co-source and corrupt the per-`(row, slot)` link scores downstream. +#[test] +fn agg_sources_declines_when_arrayed_source_lacks_per_var_slice() { + let project = TestProject::new("agg_sources_invariant") + .named_dimension("D1", &["r1", "r2"]) + .array_aux("pop[D1]", "1") + .scalar_aux("scale", "2") + .scalar_aux("total", "1 + SUM(pop[*] * scale)"); + let datamodel = project.build_datamodel(); + let db = SimlinDb::default(); + let sync = sync_from_datamodel(&db, &datamodel); + let model = sync.models["main"].source; + let variables = crate::db::reconstruct_model_variables(&db, model, sync.project); + let dm_dims = crate::db::project_datamodel_dims(&db, sync.project); + let dim_ctx = crate::db::project_dimensions_context(&db, sync.project); + let ctx = AggWalkCtx { + variables: &variables, + target_iterated_dims: &[], + dm_dims: dm_dims.as_slice(), + dim_ctx, + }; + let canonical = vec![AxisRead::Reduced { subset: None }]; + + // The invariant broken by hand: `pop` (arrayed) absent from `per_var`. + let broken = CombinedReadSlices { + canonical: canonical.clone(), + per_var: HashMap::new(), + }; + assert_eq!( + agg_sources(vec!["pop".to_string()], &broken, &ctx), + None, + "a missing per-var slice for an arrayed source must decline the \ + hoist, never substitute the canonical slice" + ); + + // The intact invariant: each source carries its own slice; a scalar + // source still gets the empty slice. + let intact = CombinedReadSlices { + canonical: canonical.clone(), + per_var: HashMap::from([("pop".to_string(), canonical.clone())]), + }; + let sources = agg_sources(vec!["scale".to_string(), "pop".to_string()], &intact, &ctx) + .expect("an intact per-var map must build the sources"); + assert_eq!( + sources, + vec![ + AggSource { + var: "pop".to_string(), + read_slice: canonical, + }, + AggSource { + var: "scale".to_string(), + read_slice: vec![], + }, + ] + ); +} + +/// GH #983: every recognized reducer's classification is CARRIED on the +/// SYNTHETIC node it decided, so no emitter has to recover it by +/// re-parsing [`AggNode::equation_text`]. +/// +/// Scope, stated because "every recognized reducer" is only one of the two +/// axes here: this covers all seven reducers on the SYNTHETIC producer arm, +/// which is the arm both readers actually reach. `register_agg`'s +/// variable-backed arm stores a `reducer` too and no test covers it, because +/// no reader consumes it -- see [`AggNode::reducer`]'s doc. +/// +/// The rows are the reducer set itself, not a sample. `reducer_kind` is +/// the only admission test, and [`crate::ltm_augment`]'s +/// `classify_builtin_if_references_source` -- the function that reads the +/// kind, name and body back off `AggNode::reducer` -- destructures exactly +/// SEVEN `BuiltinFn` variants (`Sum`, `Mean`, `Min`, `Max`, `Stddev`, +/// `Rank`, `Size`) and calls `unreachable!()` on the rest, so those seven +/// are the whole space and this table has seven rows. Each row states the +/// three facts the emitters read: whether a node is minted at all, and (if +/// so) the `ReducerKind` and uppercase name the carried builtin classifies +/// to. +/// +/// `SIZE` is the one row that mints nothing (`ReducerKind::Constant` is +/// never hoisted -- its link score is always 0), and `RANK` is the one +/// row that mints an ARRAY-valued node; both are properties of the +/// enumerator this table would notice changing. +#[test] +fn every_reducer_carries_its_classification_on_the_agg_node() { + use crate::ltm_augment::{ReducerKind, classify_reducer_in_builtin}; + + // (equation for `out`, out's dims, expected (kind, name, array_valued_rank)) + struct Row { + equation: &'static str, + out_dims: &'static [&'static str], + expected: Option<(ReducerKind, &'static str, bool)>, + } + let rows = [ + Row { + equation: "1 + SUM(pop[*])", + out_dims: &[], + expected: Some((ReducerKind::Linear, "SUM", false)), + }, + Row { + equation: "1 + MEAN(pop[*])", + out_dims: &[], + expected: Some((ReducerKind::Linear, "MEAN", false)), + }, + Row { + equation: "1 + MIN(pop[*])", + out_dims: &[], + expected: Some((ReducerKind::Nonlinear, "MIN", false)), + }, + Row { + equation: "1 + MAX(pop[*])", + out_dims: &[], + expected: Some((ReducerKind::Nonlinear, "MAX", false)), + }, + Row { + equation: "1 + STDDEV(pop[*])", + out_dims: &[], + expected: Some((ReducerKind::Nonlinear, "STDDEV", false)), + }, + Row { + // Array-valued: the node is arrayed over the ranked axis, so + // its consumer must be arrayed too. + equation: "1 + RANK(pop[*], 1)", + out_dims: &["Region"], + expected: Some((ReducerKind::Nonlinear, "RANK", true)), + }, + Row { + // `ReducerKind::Constant`: recognized, never hoisted. + equation: "1 + SIZE(pop[*])", + out_dims: &[], + expected: None, + }, + ]; + + for Row { + equation, + out_dims, + expected, + } in rows + { + let mut project = TestProject::new("carried") + .named_dimension("Region", &["nyc", "boston"]) + .array_aux("pop[Region]", "1"); + project = if out_dims.is_empty() { + project.scalar_aux("out", equation) + } else { + project.array_aux_direct( + "out", + out_dims.iter().map(|d| (*d).to_string()).collect(), + equation, + None, + ) + }; + let aggs = agg_nodes(&project); + let synthetic: Vec<&AggNode> = aggs.aggs.iter().filter(|a| a.is_synthetic).collect(); + + let Some((kind, name, array_valued_rank)) = expected else { + assert!( + synthetic.is_empty(), + "{equation}: a Constant reducer must mint no aggregate node" + ); + continue; + }; + assert_eq!( + synthetic.len(), + 1, + "{equation}: expected exactly one synthetic aggregate node" + ); + let agg = synthetic[0]; + assert_eq!( + agg.array_valued_rank, array_valued_rank, + "{equation}: array_valued_rank" + ); + + let classified = classify_reducer_in_builtin(&agg.reducer, "pop", true) + .unwrap_or_else(|| panic!("{equation}: the carried builtin must classify")); + assert_eq!(classified.kind, kind, "{equation}: kind"); + assert_eq!(classified.name, name, "{equation}: name"); + assert!( + classified.is_bare, + "{equation}: an aggregate node's equation IS the reducer call" + ); + // The body is the reducer's array argument, taken from the AST the + // enumerator walked rather than re-parsed from `equation_text`. + assert_eq!( + crate::ast::print_eqn(&classified.body), + "pop[*]", + "{equation}: body" + ); + } +} + +/// GH #983: the carried reducer is stored in +/// [`crate::ast::Expr2::strip_loc_and_bounds`] form, so a `Loc`-only edit +/// leaves the salsa-cached `AggNodesResult` equal. +/// +/// **This is one of the two ways the backdating claim can fail, and it +/// measures only this one.** The other is float non-reflexivity on a NaN +/// literal, which normalization cannot touch and which is closed at the +/// root by [`crate::ast::Literal`]'s bit-pattern equality; +/// [`a_nan_literal_in_a_reducer_does_not_defeat_agg_backdating`] pins that +/// arm. +/// +/// The two spellings differ ONLY in a leading term that mints no +/// aggregate node of its own -- a `SIZE` call, which is recognized as a +/// reducer but never hoisted -- so `SUM(pop[*])` moves to a different byte +/// offset and nothing else about the enumeration changes. The whole +/// `AggNodesResult` must therefore compare EQUAL, which is exactly +/// salsa's backdating criterion and so is the invalidation claim: an edit +/// that changes neither the reducer nor its sources must not invalidate +/// this query's consumers, as it did not before the node carried an AST +/// at all. +/// +/// It is the `Loc` half of the normalization that this measures. The +/// `ArrayBounds` half is inert TODAY and cannot be measured, because +/// `db::analysis::reconstruct_model_variables` -- the source of the ASTs +/// this query walks -- lowers against an EMPTY model scope, so +/// `Expr2Context::get_dimensions` resolves nothing and no bound is ever +/// allocated. The leading `SIZE(other[*])` is chosen over a bare constant +/// so that this test starts measuring the bounds half the moment that +/// stops being true: its arrayed argument would take the first temp id +/// and push `pop[*]` to the second. +/// +/// The second assertion is a weaker independent check on the same +/// property (re-normalizing the stored builtin is a no-op); it catches +/// normalization being dropped entirely but, being idempotence, cannot +/// by itself catch the normalization being made too weak. That is what +/// the equality assertion above is for. +#[test] +fn the_carried_reducer_is_normalized_so_offset_only_edits_backdate() { + let build = |leading: &str| { + TestProject::new("normalized") + .named_dimension("Region", &["nyc", "boston"]) + .array_aux("pop[Region]", "1") + .array_aux("other[Region]", "2") + .scalar_aux("out", &format!("{leading} + SUM(pop[*])")) + }; + let before = agg_nodes(&build("1")); + let after = agg_nodes(&build("SIZE(other[*])")); + + let sum_agg = |r: &AggNodesResult| { + r.aggs + .iter() + .find(|a| a.equation_text == "sum(pop[*])") + .cloned() + .expect("the SUM subexpression must be hoisted") + }; + assert_eq!( + before, after, + "an edit that only moves the reducer's byte offsets must leave the \ + enumerated aggregate nodes equal, so salsa backdates" + ); + let agg = sum_agg(&before); + assert_eq!( + agg.reducer + .clone() + .map(crate::ast::Expr2::strip_loc_and_bounds) + .strip_own_locs(), + agg.reducer, + "the stored reducer must already be normalized" + ); +} + +// The third channel by which `AggNode::reducer` (GH #983) could make the +// salsa-cached, `PartialEq`-compared `AggNodesResult` unequal to an +// identical rebuild: the `f64` on `Expr2::Const`, which +// `strip_loc_and_bounds` cannot normalize away (there is no rewrite that +// removes a NaN literal without changing what the equation means). +// +// It is closed at the ROOT rather than here: `ast::Literal` compares float +// literals by BIT PATTERN, so `nan == nan` and a bit-identical AST is +// equal to itself (GH #987/#981). Salsa's backdating criterion is exactly +// that equality, so without the root fix every revision bump -- from any +// unrelated edit anywhere in the project -- re-executed +// `model_element_causal_edges`, `model_ltm_reference_sites` and +// `model_ltm_variables` for any model with a `nan` in a hoisted reducer. +// +// This test was written inverted (`assert_ne!`) as the characterization of +// that defect; flipping it is how the root fix demonstrates it reached a +// real consumer rather than only its own unit test. +#[test] +fn a_nan_literal_in_a_reducer_does_not_defeat_agg_backdating() { + let build = || { + TestProject::new("nan_backdating") + .named_dimension("Region", &["nyc", "boston"]) + .array_aux("pop[Region]", "1") + .scalar_aux("out", "1 + SUM(pop[*] * nan)") + }; + + // Guard against a vacuous pass: the reducer really is hoisted, so the + // NaN really does ride on a stored `AggNode::reducer`. + let enumerated = agg_nodes(&build()); + let synthetic: Vec<&AggNode> = enumerated.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!( + synthetic.len(), + 1, + "the NaN-bearing reducer must still be hoisted for this to measure anything" + ); + + assert_eq!( + agg_nodes(&build()), + agg_nodes(&build()), + "two enumerations of an identical NaN-bearing model must compare \ + equal, so `enumerate_agg_nodes` backdates for this model like any \ + other" + ); +} + +/// The full ENUMERATION of [`classify_axis_access`]'s verdicts for the +/// element-vs-dimension-name collision, plus a per-`IndexExpr2`-variant +/// control row (GH #986). +/// +/// XMILE permits a dimension to declare an element whose name is also a +/// dimension name (`Bucket = [region, old]` beside a `Region` dimension), +/// and this classifier used to break the tie the wrong way: it asked +/// "is the name one of the target's iterated dimensions?" FIRST, so +/// `effect[Region, region]` was read as an iteration over `Region` rather +/// than as `Bucket`'s `region` ELEMENT. `compiler::subscript`'s +/// `normalize_subscripts3` resolves it the other way round, and the +/// simulation is the authority: a classifier that disagrees describes rows +/// the simulation never reads. +/// +/// The two collision rows are the whole point, and they fail differently -- +/// which is why both are here rather than one standing in for the pair: +/// +/// - UNMAPPED (`bucket` has no mapping to `region`): the old order found no +/// correspondence and returned `None`, so the reference fell to the +/// conservative cross-product. Over-broad, not wrong -- a precision gap; +/// - MAPPED (`bucket` positionally mapped to `region`): the old order took +/// the mapped branch and returned `Iterated` over a dimension the compiler +/// never iterates there, so read slices and element edges named rows the +/// simulation does not read. That one is a correctness hazard, and it is +/// the reason the fix is not just a precision improvement. +/// +/// The remaining rows are the enumeration this is derived from rather than +/// sampled: the function's match has SIX arms (`Wildcard`, `StarRange`, +/// `Range | DimPosition`, `Expr(Var)`, `Expr(Const)`, `Expr(_)`) and every one +/// has a row, with `Expr(Var)` fanning out over all FIVE ways a name can +/// resolve -- an element of the axis, an iterated dimension matching the axis by +/// name, an iterated dimension matching it through a positional mapping, an +/// iterated dimension with no usable correspondence, and a name that is neither. +/// +/// One arm is covered elsewhere rather than here: `StarRange` has three +/// outcomes (the axis's own dimension, a PROPER subdimension, and a name that +/// is neither), and the subdimension pair is GH #766's, pinned by the +/// `*_star_range_*` tests in this file over real models. The row below states +/// the own-dimension outcome, which is the one a name-resolution change could +/// disturb. +#[test] +fn classify_axis_access_resolves_a_colliding_name_element_first() { + use crate::ast::{Expr2, IndexExpr2, Loc}; + use crate::common::{CanonicalDimensionName, Ident}; + use crate::datamodel; + + // `bucket`'s FIRST element is named `region`, exactly like the `region` + // DIMENSION the target iterates. `mapped` decides whether `bucket` also + // carries a positional mapping to `region`, which is what separates the + // old code's two failure modes. + let ctx_with = |mapped: bool| { + let mut bucket = datamodel::Dimension::named( + "bucket".to_string(), + vec!["region".to_string(), "old".to_string()], + ); + if mapped { + bucket.mappings = vec![datamodel::DimensionMapping { + target: "region".to_string(), + element_map: vec![], + }]; + } + // Iterated by the target and POSITIONALLY mapped to `region`, so it + // lines up with a `region` axis by correspondence rather than by name. + let mut mapped_dim = datamodel::Dimension::named( + "mapped_dim".to_string(), + vec!["m1".to_string(), "m2".to_string()], + ); + mapped_dim.mappings = vec![datamodel::DimensionMapping { + target: "region".to_string(), + element_map: vec![], + }]; + crate::dimensions::DimensionsContext::from( + [ + datamodel::Dimension::named( + "region".to_string(), + vec!["nyc".to_string(), "boston".to_string()], + ), + mapped_dim, + // Iterated by the target and mapped to nothing -- the row where + // the correspondence declines. + datamodel::Dimension::named( + "lonely".to_string(), + vec!["l1".to_string(), "l2".to_string()], + ), + bucket, + ] + .as_slice(), + ) + }; + let bucket_axis = |ctx: &crate::dimensions::DimensionsContext| { + ctx.get(&CanonicalDimensionName::from_raw("bucket")) + .expect("the fixture declares `bucket`") + .clone() + }; + let var_idx = |name: &str| IndexExpr2::Expr(Expr2::Var(Ident::new(name), None, Loc::default())); + // The target iterates `region` plus the two dimensions the mapped rows need: + // `mapped_dim` (positionally mapped to `region`) and `lonely` (mapped to + // nothing). + let iterated = [ + "region".to_string(), + "mapped_dim".to_string(), + "lonely".to_string(), + ]; + + // --- the collision, in both mapping states --------------------------- + for mapped in [false, true] { + let ctx = ctx_with(mapped); + assert_eq!( + classify_axis_access(&var_idx("region"), &bucket_axis(&ctx), &iterated, &ctx), + Some(AxisRead::Pinned("region".to_string())), + "a name the axis declares as an ELEMENT must resolve to that \ + element even when it also names an iterated dimension \ + (mapped = {mapped}) -- that is what `normalize_subscripts3` does" + ); + // The control: the same axis's other element, which collides with + // nothing. A fix that special-cased the colliding name would not + // change this row, so it cannot stand in for the row above. + assert_eq!( + classify_axis_access(&var_idx("old"), &bucket_axis(&ctx), &iterated, &ctx), + Some(AxisRead::Pinned("old".to_string())), + "the non-colliding control element (mapped = {mapped})" + ); + } + // Non-vacuity: the collision must be real in both directions, or the + // rows above are testing an ordinary element lookup. + let mapped_ctx = ctx_with(true); + assert!(mapped_ctx.is_dimension_name("region")); + assert!( + mapped_ctx + .mapped_element_correspondence( + &CanonicalDimensionName::from_raw("region"), + &CanonicalDimensionName::from_raw("bucket"), + ) + .is_some(), + "the mapped row needs a usable `region`/`bucket` correspondence, or \ + it duplicates the unmapped row" + ); + + // --- the rest of the space ------------------------------------------- + let ctx = ctx_with(false); + let region_axis = ctx + .get(&CanonicalDimensionName::from_raw("region")) + .expect("the fixture declares `region`") + .clone(); + // A name the axis does NOT declare, matching the axis's own dimension: + // the ordinary iterated form. + assert_eq!( + classify_axis_access(&var_idx("region"), ®ion_axis, &iterated, &ctx), + Some(AxisRead::Iterated { + dim: "region".to_string(), + source_dim: "region".to_string(), + }), + "an iterated dimension matching the axis by name" + ); + // Iterated, matching the axis through a POSITIONAL mapping rather than by + // name: `mapped_dim` is not `region`, but the correspondence lines them up. + assert_eq!( + classify_axis_access(&var_idx("mapped_dim"), ®ion_axis, &iterated, &ctx), + Some(AxisRead::Iterated { + dim: "mapped_dim".to_string(), + source_dim: "region".to_string(), + }), + "an iterated dimension matching the axis through a positional mapping" + ); + // Iterated, with NO usable correspondence to this axis: declines. + assert_eq!( + classify_axis_access(&var_idx("lonely"), ®ion_axis, &iterated, &ctx), + None, + "an iterated dimension with no usable correspondence to this axis \ + declines, keeping the reference on the conservative path" + ); + // A name that is neither an element of the axis nor iterated. + assert_eq!( + classify_axis_access(&var_idx("idx"), ®ion_axis, &iterated, &ctx), + None, + "a variable read selecting the element is not statically describable" + ); + // The remaining `IndexExpr2` variants, none of which poses the + // name-resolution question. + assert_eq!( + classify_axis_access( + &IndexExpr2::Wildcard(Loc::default()), + ®ion_axis, + &iterated, + &ctx + ), + Some(AxisRead::Reduced { subset: None }), + "a wildcard reduces the whole axis" + ); + assert_eq!( + classify_axis_access( + &IndexExpr2::StarRange(CanonicalDimensionName::from_raw("region"), Loc::default()), + ®ion_axis, + &iterated, + &ctx + ), + Some(AxisRead::Reduced { subset: None }), + "`*:D` over the axis's own dimension is the full extent" + ); + assert_eq!( + classify_axis_access( + &IndexExpr2::Range( + Expr2::Const( + "1".to_string(), + crate::ast::Literal::new(1.0), + Loc::default() + ), + Expr2::Const( + "2".to_string(), + crate::ast::Literal::new(2.0), + Loc::default() + ), + Loc::default() + ), + ®ion_axis, + &iterated, + &ctx + ), + None, + "a range is not statically describable per axis" + ); + assert_eq!( + classify_axis_access( + &IndexExpr2::DimPosition(2, Loc::default()), + ®ion_axis, + &iterated, + &ctx + ), + None, + "an `@N` position is declined for hoisting" + ); + assert_eq!( + classify_axis_access( + &IndexExpr2::Expr(Expr2::Const( + "2".to_string(), + crate::ast::Literal::new(2.0), + Loc::default() + )), + ®ion_axis, + &iterated, + &ctx + ), + None, + "a numeric literal into a NAMED dimension names no element" + ); + assert_eq!( + classify_axis_access( + &IndexExpr2::Expr(Expr2::Op2( + crate::ast::BinaryOp::Add, + Box::new(Expr2::Var(Ident::new("idx"), None, Loc::default())), + Box::new(Expr2::Const( + "1".to_string(), + crate::ast::Literal::new(1.0), + Loc::default() + )), + None, + Loc::default() + )), + ®ion_axis, + &iterated, + &ctx + ), + None, + "a compound index expression is not statically describable" + ); +} + +/// The ORACLE behind [`classify_axis_access_resolves_a_colliding_name_element_first`]: +/// what the executed simulation actually reads for a colliding index name. +/// +/// The classifier's job is to describe the reference the simulation +/// performs, so "element-first is right" is a claim about this engine and is +/// checked by running it rather than asserted. `Bucket = [old, region]` +/// declares its colliding element at position 1 while `Region`'s `nyc` sits +/// at position 0, which is what makes the two readings distinguishable: +/// element-first reads `effect[nyc, region]`, and the dimension-name reading +/// would resolve `Region`'s current element `nyc` against `Bucket` (which +/// declares no `nyc`) and read a different row -- or fail to resolve at all. +/// +/// The second half asserts the IR now DESCRIBES that read: the `effect -> t` +/// site classifies `PerElement` with the `Bucket` axis `Pinned` to the +/// literal element. Before GH #986 the classifier took the dimension-name +/// branch, found no `Region`/`Bucket` correspondence and declined, so the +/// reference fell to the conservative cross-product instead. +#[test] +fn a_colliding_index_name_reads_the_axis_element_in_the_simulation() { + use crate::db::{RefShape, compile_project_incremental, ltm_ir::model_ltm_reference_sites}; + + let project = TestProject::new("colliding_element_name") + .with_sim_time(0.0, 1.0, 1.0) + .named_dimension("Region", &["nyc", "boston"]) + // `region` is an ELEMENT of `Bucket` and also the name of a + // DIMENSION -- the collision XMILE permits. It sits at position 1 so + // the two readings pick different rows. + .named_dimension("Bucket", &["old", "region"]) + .array_with_ranges_direct( + "regw", + vec!["Region".to_string()], + vec![("nyc", "1"), ("boston", "2")], + None, + ) + .array_with_ranges_direct( + "bucketw", + vec!["Bucket".to_string()], + vec![("old", "3"), ("region", "4")], + None, + ) + .array_aux_direct( + "effect", + vec!["Region".to_string(), "Bucket".to_string()], + "regw[Region] * 10 + bucketw[Bucket]", + None, + ) + .array_aux_direct( + "t", + vec!["Region".to_string()], + "effect[Region, region]", + None, + ) + .build_datamodel(); + + let db = SimlinDb::default(); + let sync = sync_from_datamodel(&db, &project); + let compiled = compile_project_incremental(&db, sync.project, "main") + .expect("the colliding-name model compiles"); + let mut vm = crate::vm::Vm::new(compiled.clone()).expect("VM creation should succeed"); + vm.run_to_end().expect("simulation should run"); + let results = vm.into_results(); + let at = |name: &str| -> f64 { + let off = compiled + .offsets + .iter() + .find(|(k, _)| k.as_str() == name) + .unwrap_or_else(|| panic!("no slot named {name}")) + .1; + results.iter().next_back().expect("a saved step")[*off] + }; + // Non-vacuity: the two candidate reads must differ, or the oracle + // cannot tell the orders apart. + assert_ne!(at("effect[nyc,region]"), at("effect[nyc,old]")); + assert_eq!( + at("t[nyc]"), + at("effect[nyc,region]"), + "`effect[Region, region]` must read `Bucket`'s `region` ELEMENT -- \ + the axis's own element namespace wins inside square brackets" + ); + assert_eq!(at("t[boston]"), at("effect[boston,region]")); + + // ...and the reference-site IR describes exactly that read. + let sites = model_ltm_reference_sites(&db, sync.models["main"].source, sync.project); + let shapes: Vec<&RefShape> = sites + .sites + .get(&("effect".to_string(), "t".to_string())) + .map(|v| v.iter().map(|s| &s.shape).collect()) + .unwrap_or_default(); + assert_eq!( + shapes, + vec![&RefShape::PerElement { + axes: vec![ + AxisRead::Iterated { + dim: "region".to_string(), + source_dim: "region".to_string(), + }, + AxisRead::Pinned("region".to_string()), + ], + }], + "the classified site must pin the `Bucket` axis to its literal \ + element, matching the read the VM just performed" + ); +} diff --git a/src/simlin-engine/src/ltm_augment.rs b/src/simlin-engine/src/ltm_augment.rs index 0c5afdc5d..95ef55a29 100644 --- a/src/simlin-engine/src/ltm_augment.rs +++ b/src/simlin-engine/src/ltm_augment.rs @@ -54,10 +54,10 @@ pub(crate) use with_lookup::{ #[path = "ltm_augment_post_transform.rs"] mod post_transform; -pub(crate) use post_transform::per_element_row_for_target; #[cfg(test)] pub(crate) use post_transform::substitute_reducers_in_equation; use post_transform::{PerElementRefCtx, qualify_axis_element, substitute_reducers_in_expr0}; +pub(crate) use post_transform::{dep_element_pins, per_element_row_for_target}; /// Context for recognizing GH #511 iterated-dimension source references in /// the partial-equation builder: the live source's declared dimension names @@ -195,7 +195,10 @@ struct WrapOutcome { /// ([`wrap_non_matching_in_previous`] / [`wrap_index_non_matching_in_previous`]), /// bundled so the deep recursion threads one `&WrapCtx` instead of a long /// positional argument list. Every field is a `Copy` reference, so the body -/// destructures it back into the historically-named locals with no clones. +/// destructures it back into the historically-named locals with no clones -- +/// and the whole struct is `Copy`, which lets the `LOOKUP` table-argument +/// descent rebind exactly one field (see [`OccurrenceLookup::empty`]). +#[derive(Clone, Copy)] struct WrapCtx<'a> { /// The source variable whose live shape is held out of `PREVIOUS` /// wrapping; all its other occurrences (and every other-dep reference) @@ -677,54 +680,59 @@ fn wrap_non_matching_in_previous( ctx.occ, path, &mut out.missing_occurrence, - // This call lags (or freezes) everything inside it, - // subscript index reads included, so a runtime table - // index in there is already ceteris-paribus. - true, ), None => call, }; } // A LOOKUP call's first argument names a graphical-function table - // (a lookup-only variable, or the WITH-LOOKUP self-reference); it - // is static data the compiler resolves to a table id, not a causal - // value reference. Wrapping it in PREVIOUS produces + // (a lookup-only variable, or the WITH-LOOKUP self-reference); the + // table HEAD is static data the compiler resolves to a table id, not + // a causal value reference. Wrapping it in PREVIOUS produces // `lookup(PREVIOUS(table), ...)`, which cannot compile (a // table-only variable has no value slot), so the whole link-score // fragment silently zeroes -- the failure mode behind WRLD3's - // identically-zero table-mediated link scores. Hold the table - // argument verbatim and transform only the index argument(s). + // identically-zero table-mediated link scores. Hold the HEAD + // verbatim. + // + // Its subscript INDEX expressions are a different thing entirely + // (GH #984): `compiler::codegen::extract_table_info`'s + // `Expr::Subscript` arm builds the element offset out of the live + // index `Expr`s, so `LOOKUP(g[idx], x)` reads `idx` at the CURRENT + // step. Held verbatim, every partial of that target varied with + // `idx`, misattributing its movement to whichever source the partial + // isolates -- a wrong number with no diagnostic. So the indices go + // through the wrap's own index pass like any other other-dep. if matches!( name.to_ascii_lowercase().as_str(), "lookup" | "lookup_forward" | "lookup_backward" ) && !args.is_empty() { - // The table arg (child 0) is held verbatim; only the index - // arg(s) are wrapped. Child indices match the `Expr2` walk, - // which counts the skipped `LookupTable` slot as child 0. + // Child indices match the `Expr2` walk, which counts the skipped + // `LookupTable` slot as child 0. let new_args = args .into_iter() .enumerate() .map(|(i, a)| { if i == 0 { - // Static table data, not a causal value read: held - // verbatim by the wrap, but still row-pinned (the - // pin-only descent) so a `PerElement` lowering does - // not leave a dimension-name subscript behind. + let a = freeze_lookup_table_indices( + a, + ctx, + out, + &child_path(path, i), + frozen, + ); + // The head is still held verbatim, but a `PerElement` + // lowering must reach every source reference inside + // the argument (the head itself can BE the source): + // an un-pinned dimension-name subscript cannot + // resolve in a scalar fragment. match ctx.pin { Some(pin_ctx) => post_transform::pin_only_source_refs( a, pin_ctx, - ctx.occ, + &OccurrenceLookup::empty(), &child_path(path, i), &mut out.missing_occurrence, - // The wrap holds the table arg VERBATIM, so - // it inherits the enclosing freeze and - // nothing more: a runtime index in a bare - // table argument stays live (declined), the - // same argument inside a frozen subtree is - // already lagged (kept). - frozen, ), None => a, } @@ -780,8 +788,6 @@ fn wrap_non_matching_in_previous( ctx.occ, path, &mut out.missing_occurrence, - // Frozen whole, one line below. - true, ), None => reducer, }; @@ -852,6 +858,112 @@ fn wrap_non_matching_in_previous( } } +/// Freeze the runtime value reads in a `LOOKUP` TABLE argument's subscript +/// indices, leaving the table HEAD (and every static element selector) exactly +/// as written (GH #984). +/// +/// This is the whole of the fix, and it is one rule for every caller: the +/// wrap's own index pass ([`wrap_index_non_matching_in_previous`]) decides what +/// a table index is, exactly as it does for an ordinary subscript. Static +/// selectors -- a literal element, a dimension name, `@N`, a numeric literal -- +/// hit its guards and come back untouched; a genuine value read (`idx`, +/// `idx + 1`) reaches the recursive wrap and comes back frozen. +/// +/// Four deliberate choices: +/// +/// - only an `Expr0::Subscript` has indices to freeze. A bare `Var` table is +/// already static, and anything else is `BadTable` at codegen (the table +/// argument must select exactly one element), so there is nothing here to +/// decide about it; +/// - the descent's dep set is WIDENED with the idents appearing in these +/// indices, and without that the freeze does not fire at all. The wrap freezes +/// an ident only when it is in `other_deps`, and `other_deps` comes from +/// `variable::classify_dependencies`, whose `BuiltinContents::LookupTable` arm +/// records the table's ident and **never walks the table expression** -- so an +/// index variable referenced ONLY inside a table argument (the issue's own +/// `LOOKUP(g[idx], x)`) is not a dependency and was returned live. Widening is +/// scoped to this argument's own indices, so nothing else in the equation sees +/// the added names, and the element / dimension-name guards in +/// [`wrap_index_non_matching_in_previous`] run BEFORE the `other_deps` check -- +/// so adding an element or dimension name to the set cannot make it wrap. (The +/// dep set itself is left alone: dropping a table index from a variable's +/// dependencies is arguably an engine bug in its own right, but it is a +/// runlist-ordering question, not this rule's, and fixing it there is a +/// separate change); +/// - element QUALIFICATION is suppressed (`skip_element_qualification`). A +/// literal element index is static either way, and leaving it verbatim keeps +/// this change to the reads it is about: on the `PerElement` path the row +/// pinning already qualifies it from the source's own declared dims (which is +/// the only qualification that is right for an ambiguous element name), and on +/// the other paths the target's own equation carries the same spelling; +/// - inside an enclosing FREEZE nothing is done. `frozen` here means the whole +/// `LOOKUP` sits in a subtree the wrap is about to lag (a wrapped other-dep's +/// subscript index is the only way to get here -- a pre-existing +/// `PREVIOUS`/`INIT` and a whole-frozen reducer never reach this arm at all), +/// so the index read is already ceteris-paribus and a second `PREVIOUS` would +/// read two steps back. +/// +/// Because this discharges the index everywhere the arm runs -- for ANY index +/// ident, not only one that happens to be a dependency elsewhere -- and the +/// enclosing freeze discharges it everywhere the arm does not, +/// `post_transform::pin_dimension_name_indices` no longer has to REFUSE a table +/// argument carrying a runtime index -- it keeps it and says nothing. That +/// refusal, its `frozen` plumbing, and the warned skip it produced are deleted. +fn freeze_lookup_table_indices( + arg: Expr0, + ctx: &WrapCtx<'_>, + out: &mut WrapOutcome, + path: &[u16], + frozen: bool, +) -> Expr0 { + let Expr0::Subscript(ident, indices, loc) = arg else { + return arg; + }; + if frozen { + return Expr0::Subscript(ident, indices, loc); + } + // Every ident under these indices is a read the wrap must be able to freeze, + // whether or not the dependency extractor knows about it (see the rustdoc). + let mut index_deps: HashSet> = ctx.other_deps.clone(); + for idx in &indices { + let mut add = |e: &Expr0| { + index_deps.extend(expr_reference_idents(e).into_iter().map(|n| Ident::new(&n))); + }; + match idx { + IndexExpr0::Expr(e) => add(e), + IndexExpr0::Range(l, r, _) => { + add(l); + add(r); + } + // Wildcard / star-range / `@N` carry no `Expr0`. + _ => {} + } + } + // The IR records no occurrence anywhere under a table argument, so the + // descent uses an empty lookup rather than this slot's -- see + // [`OccurrenceLookup::empty`] for why a real one would be unsound. + let table_ctx = WrapCtx { + occ: &OccurrenceLookup::empty(), + other_deps: &index_deps, + ..*ctx + }; + let indices = indices + .into_iter() + .enumerate() + .map(|(i, idx)| { + wrap_index_non_matching_in_previous( + idx, + &table_ctx, + out, + true, + &child_path(path, i), + frozen, + ) + }) + .collect(); + Expr0::Subscript(ident, indices, loc) +} + /// If `index` is a bare identifier that unambiguously names a dimension /// element (per `dims_ctx`), return its qualified `dimension·element` form; /// otherwise `None`. @@ -926,6 +1038,20 @@ fn wrap_index_non_matching_in_previous( { return qualified; } + // An ALREADY-`dim·element`-qualified index is a static element selector + // whatever the qualification mode: it carries its own dimension and resolves + // to a constant offset. `qualify_element_index` returns it verbatim too, but + // that call is suppressed on the paths that own qualification themselves, so + // the guard has to stand on its own. It was latent before GH #984 -- the + // suppressed path only wrapped an ident present in `other_deps`, and no + // variable is named `dim·elem` -- and became reachable when the table-index + // freeze started widening that set with the index's own idents. + if let IndexExpr0::Expr(Expr0::Var(name, _)) = &index + && let Some(ctx) = dims_ctx + && ctx.lookup(&canonicalize(name.as_str())).is_some() + { + return index; + } // An index that names a dimension element which *cannot* be qualified // (declared by multiple dimensions at different positions, e.g. C-LEARN's // region elements) is still left verbatim rather than PREVIOUS-wrapped. @@ -2128,26 +2254,60 @@ fn pin_iterated_dim_indices(expr: Expr0, dims: &[String], parts: &[String]) -> O }) } -/// Replace every bare `Var(id)` reference in `equation_text` where `id` -/// (canonicalized) is in `idents` with `Subscript(id, [element])`, -/// pinning that variable to `element`. +/// Where ONE dep is pinned in a scalar per-element link-score equation: the +/// element it reads at that target element, as one +/// `(canonical dimension name, element spelling)` pair per dimension **that dep +/// declares** and that the target element projects onto, in the dep's own +/// declaration order. +/// +/// Carrying the dep's OWN dimensions rather than the target's is the whole +/// content of GH #974. A bare arrayed reference inside an apply-to-all body +/// reads its own axes' coordinates matched by dimension NAME, so a dep +/// declaring a strict subset of the target's dimensions (`w[Age]` under +/// `growth[Region,Age]`) must be pinned over that subset, and a dep declaring +/// the same dimensions in another order (`w[Age,Region]`) must be pinned in ITS +/// order. Pinning both with the target's full tuple produced an over-arity +/// subscript in the first case (a fragment that fails to compile, so the score +/// reads a constant 0) and a compilable, silently TRANSPOSED read in the +/// second. The projection itself is `post_transform::dep_element_pins`; this +/// type is just how the answer travels to the rewrite. +#[derive(Clone)] +pub(crate) struct DepElementPin { + /// The resolved axes, in the dep's declaration order. + pub(crate) axes: Vec<(String, String)>, + /// Whether `axes` covers EVERY dimension the dep declares. Only a complete + /// pin can subscript a BARE reference, which must be spelled at the dep's + /// full arity; an incomplete one still substitutes the dimension-name + /// indices of an already-subscripted reference, which is all that reference + /// needs. + pub(crate) complete: bool, +} + +/// Replace every reference to a pinned dep in `equation_text` with that dep's +/// own element subscript, per [`pins`](DepElementPin). /// /// Used when collapsing a scalar-source -> arrayed-target link score into /// per-target-element scalar variables: the target's A2A equation body -/// references arrayed deps that share the target's dimension *bare* (the +/// references arrayed deps that share the target's dimensions *bare* (the /// A2A expansion subscripts them at runtime), but a *scalar* per-element -/// link-score variable must spell out the subscript. `idents` is the set -/// of those deps (the caller computes it -- it needs to know which deps -/// are arrayed and share the target's dimension). -/// -/// `Subscript` nodes are left untouched: a dep that already carries an -/// explicit subscript (an `Ast::Arrayed` per-element slot wrote -/// `population[NYC]`) is already element-pinned, and double-subscripting -/// would be nonsensical. Function-name identifiers and identifiers not in -/// `idents` are likewise left alone. The result is re-printed in the -/// canonical equation format (via parse + `print_eqn`). -/// -/// Returns `Ok(equation_text)` unchanged when `idents` is empty (nothing to +/// link-score variable must spell out the subscript. `pins` says which deps +/// those are and which element each one reads -- the caller computes it, +/// because deciding that needs the deps' declared dimensions and the project's +/// dimension mappings. +/// +/// A bare `Var(id)` reference to a COMPLETELY-pinned dep becomes +/// `Subscript(id, )`. An already-`Subscript`ed reference keeps its +/// literal element indices, but an index naming one of the dep's OWN +/// dimensions (`dep[Region]`, the A2A iterated reference form) reads "the +/// current element" -- which in a per-element scalar equation is exactly the +/// element being pinned -- so it is substituted whether or not the pin is +/// complete. Left unpinned, such an index is unresolvable in scalar context +/// and forces a synthesized helper aux per occurrence (GH #654: ~27k of +/// C-LEARN's ~30k residual helpers came from this form). Function-name +/// identifiers and identifiers absent from `pins` are left alone. The result is +/// re-printed in the canonical equation format (via parse + `print_eqn`). +/// +/// Returns `Ok(equation_text)` unchanged when `pins` is empty (nothing to /// pin -- a legitimate no-op), and `Err([`PartialEquationError`])` when the /// (already-PREVIOUS-wrapped) partial text fails to re-parse. The latter is /// loud rather than a silent lowercased-input fallback for the same reason @@ -2155,93 +2315,74 @@ fn pin_iterated_dim_indices(expr: Expr0, dims: &[String], parts: &[String]) -> O /// not even compile, and a silent wrong equation is worse than skipping the /// score with a warning. /// -/// `element` is a single element name (`"nyc"`, or qualified `"region·nyc"`) -/// for a one-dimensional target, or a comma-joined tuple (`"nyc,adult"` / -/// `"region·nyc,age·adult"`) for a multi-dimensional one -- the same form -/// `db::ltm::cartesian_subscripts` produces (and `qualify_element_csv` -/// qualifies) and the `parse_link_offsets` discovery parser expects on the -/// `to` side. +/// Each element spelling is a single element name (`"nyc"`, or qualified +/// `"region·nyc"`) -- the same form `db::ltm::cartesian_subscripts` produces +/// (and `qualify_element_csv` qualifies) and the `parse_link_offsets` discovery +/// parser expects on the `to` side. pub(crate) fn subscript_idents_at_element( equation_text: &str, - idents: &HashSet>, - element: &str, + pins: &HashMap, DepElementPin>, ) -> Result { - if idents.is_empty() { + if pins.is_empty() { return Ok(equation_text.to_string()); } let Ok(Some(ast)) = Expr0::new(equation_text, LexerType::Equation) else { return Err(PartialEquationError::new(equation_text)); }; - let index_exprs: Vec = element - .split(',') - .map(|e| { - IndexExpr0::Expr(Expr0::Var( - crate::common::RawIdent::new_from_str(e.trim()), - crate::ast::Loc::default(), - )) - }) - .collect(); - // Qualified element parts (`region·nyc`) also pin *dimension-name* - // subscript indices: a dep referenced as `dep[Region]` (the A2A iterated - // form) reads the same element a bare `dep` reference would, so in a - // per-element scalar equation it must be pinned to that element. The map - // is keyed by the canonical dimension name each qualified part names. - let dim_to_index: Vec<(String, IndexExpr0)> = element - .split(',') - .zip(index_exprs.iter()) - .filter_map(|(part, idx_expr)| { - let part = part.trim(); - part.split_once('\u{00B7}') - .map(|(dim, _)| (canonicalize(dim).into_owned(), idx_expr.clone())) - }) - .collect(); - Ok(print_eqn(&subscript_idents_in_expr0( - ast, - idents, - &index_exprs, - &dim_to_index, - ))) + Ok(print_eqn(&subscript_idents_in_expr0(ast, pins))) +} + +/// The `IndexExpr0` a pin entry's element spelling becomes. +fn pin_index(elem: &str) -> IndexExpr0 { + IndexExpr0::Expr(Expr0::Var( + crate::common::RawIdent::new_from_str(elem), + crate::ast::Loc::default(), + )) } fn subscript_idents_in_expr0( expr: Expr0, - idents: &HashSet>, - index_exprs: &[IndexExpr0], - dim_to_index: &[(String, IndexExpr0)], + pins: &HashMap, DepElementPin>, ) -> Expr0 { match expr { Expr0::Const(..) => expr, Expr0::Var(ref ident, loc) => { let canonical = Ident::new(ident.as_str()); - if idents.contains(&canonical) { - Expr0::Subscript(ident.clone(), index_exprs.to_vec(), loc) - } else { - expr + // Only a COMPLETE pin can spell a bare reference: a subscript + // covering some of the dep's axes is not a legal reference at all. + match pins.get(&canonical).filter(|pin| pin.complete) { + Some(pin) => Expr0::Subscript( + ident.clone(), + pin.axes.iter().map(|(_, elem)| pin_index(elem)).collect(), + loc, + ), + None => expr, } } // An already-subscripted reference to a pinned dep: indices that are // *element literals* are already pinned and stay, but an index that - // names one of the pinned DIMENSIONS (`dep[Region]`, the A2A iterated - // reference form) reads "the current element" -- which in a - // per-element scalar equation is exactly the element being pinned. - // Left unpinned, such an index is unresolvable in scalar context and - // forces a synthesized helper aux per occurrence (GH #654: ~27k of - // C-LEARN's ~30k residual helpers came from this form). + // names one of that dep's own DIMENSIONS gets this element's coordinate + // for it. The lookup is over the DEP's dimensions, so a subset-dims or + // reordered dep resolves each of its axes correctly (GH #974) -- and an + // axis the target does not project (`pop[Region, idx]` under a + // `growth[Region]` target, whose `Age` axis the reference pins itself) + // simply has no entry, which is why an INCOMPLETE pin still applies here. Expr0::Subscript(ident, indices, loc) => { let canonical = Ident::new(ident.as_str()); - if !idents.contains(&canonical) || dim_to_index.is_empty() { + let Some(pin) = pins.get(&canonical) else { return Expr0::Subscript(ident, indices, loc); - } + }; let indices = indices .into_iter() .map(|idx| { if let IndexExpr0::Expr(Expr0::Var(name, _)) = &idx { let idx_canonical = canonicalize(name.as_str()); - if let Some((_, pinned)) = dim_to_index + if let Some((_, elem)) = pin + .axes .iter() .find(|(dim, _)| dim.as_str() == idx_canonical.as_ref()) { - return pinned.clone(); + return pin_index(elem); } } idx @@ -2252,55 +2393,23 @@ fn subscript_idents_in_expr0( Expr0::App(UntypedBuiltinFn(name, args), loc) => { let args = args .into_iter() - .map(|a| subscript_idents_in_expr0(a, idents, index_exprs, dim_to_index)) + .map(|a| subscript_idents_in_expr0(a, pins)) .collect(); Expr0::App(UntypedBuiltinFn(name, args), loc) } - Expr0::Op1(op, inner, loc) => Expr0::Op1( - op, - Box::new(subscript_idents_in_expr0( - *inner, - idents, - index_exprs, - dim_to_index, - )), - loc, - ), + Expr0::Op1(op, inner, loc) => { + Expr0::Op1(op, Box::new(subscript_idents_in_expr0(*inner, pins)), loc) + } Expr0::Op2(op, lhs, rhs, loc) => Expr0::Op2( op, - Box::new(subscript_idents_in_expr0( - *lhs, - idents, - index_exprs, - dim_to_index, - )), - Box::new(subscript_idents_in_expr0( - *rhs, - idents, - index_exprs, - dim_to_index, - )), + Box::new(subscript_idents_in_expr0(*lhs, pins)), + Box::new(subscript_idents_in_expr0(*rhs, pins)), loc, ), Expr0::If(cond, then_expr, else_expr, loc) => Expr0::If( - Box::new(subscript_idents_in_expr0( - *cond, - idents, - index_exprs, - dim_to_index, - )), - Box::new(subscript_idents_in_expr0( - *then_expr, - idents, - index_exprs, - dim_to_index, - )), - Box::new(subscript_idents_in_expr0( - *else_expr, - idents, - index_exprs, - dim_to_index, - )), + Box::new(subscript_idents_in_expr0(*cond, pins)), + Box::new(subscript_idents_in_expr0(*then_expr, pins)), + Box::new(subscript_idents_in_expr0(*else_expr, pins)), loc, ), } @@ -2328,13 +2437,14 @@ fn subscript_idents_in_expr0( /// canonical text to its agg name; it is empty for a true scalar `from`. `to_deps` /// is the full dependency set of that equation (computed with the target's AST /// dimensions so element-name subscripts are not mistaken for variables), plus -/// the agg names. `to_deps_to_subscript` is the subset of `to_deps` that -/// must be element-pinned -- the arrayed deps that share the target's -/// dimension (the target self-reference is pinned implicitly via the -/// already-subscripted `to[element]` reference the guard form is built -/// around). The source itself is never in this set: an arrayed-agg source is -/// pinned via its `source_pins` entry instead, since the full target tuple -/// over-subscripts it in the broadcast case (GH #528). +/// the agg names. `to_deps_to_subscript` is the pin table for the subset of +/// `to_deps` that must be element-pinned -- the arrayed deps whose every +/// declared dimension this target element projects onto, each mapped to the +/// element IT reads (see [`DepElementPin`]; the target self-reference is pinned +/// implicitly via the already-subscripted `to[element]` reference the guard +/// form is built around). The source itself is never in this table: an +/// arrayed-agg source is pinned via its `source_pins` entry instead, since an +/// agg has no declared dimensions of its own to project onto (GH #528). /// /// `source_ref_override`: the pre-rendered (quoted, possibly element-pinned) /// reference expression to use for the `Δsource` denominator. `None` uses the @@ -2344,9 +2454,9 @@ fn subscript_idents_in_expr0( /// partial) numerator do; a bare agg reference in a scalar equation would not /// compile and the link score would stub to zero. /// -/// `source_pins`: the per-ident pin map (GH #751) -- for each `(ident, -/// slot)` entry, that ident's references in the partial BODY are pinned to -/// the (qualified) `slot` element tuple. Empty for a true scalar source +/// `source_pins`: the per-ident pin list (GH #751) -- for each `(ident, pin)` +/// entry, that ident's references in the partial BODY are pinned to the +/// (qualified) element tuple `pin` names. Empty for a true scalar source /// (no pinning). The arrayed-agg caller passes one entry per ARRAYED agg /// referenced by the substituted equation: the LIVE agg's entry carries the /// target element's projection onto its `result_dims` axes -- the same slot @@ -2355,13 +2465,13 @@ fn subscript_idents_in_expr0( /// equation) carries the projection onto ITS OWN `result_dims`, so its /// `PREVIOUS(...)` freeze reads one slot instead of the ill-typed bare /// multi-slot reference that failed fragment compile and stubbed the score -/// to 0 (GH #751). Pinning these idents through `to_deps_to_subscript` -/// instead would use the target's FULL element tuple, which over-subscripts -/// an agg in the broadcast case (`agg[D1]` feeding `to[D1,D2]`): the -/// fragment fails to compile and the score is stubbed to a constant 0 -/// (GH #528). For the diagonal case (`result_dims` == `to`'s dims) the -/// projection IS the full tuple, so the equation is unchanged. A SCALAR -/// co-agg needs no entry: its bare `PREVIOUS(name)` freeze compiles as-is. +/// to 0 (GH #751). This is a SEPARATE channel from `to_deps_to_subscript` +/// because an agg is a synthetic aux with no declared dimensions: its slot +/// space is the hoisted reducer's `result_dims`, which the declared-dimension +/// projection behind `to_deps_to_subscript` cannot derive. For the diagonal +/// case (`result_dims` == `to`'s dims) the projection IS the full tuple, so +/// the equation is unchanged. A SCALAR co-agg needs no entry: its bare +/// `PREVIOUS(name)` freeze compiles as-is. /// /// `gf_table_ref` is the implicit WITH-LOOKUP wrap for THIS target element /// (GH #910): the partial is a full re-evaluation of the target's element @@ -2388,9 +2498,9 @@ pub(crate) fn generate_scalar_to_element_equation( to_elem_eqn: &Expr0, reducer_subst: &HashMap, to_deps: &HashSet>, - to_deps_to_subscript: &HashSet>, + to_deps_to_subscript: &HashMap, DepElementPin>, source_ref_override: Option<&str>, - source_pins: &[(Ident, String)], + source_pins: &[(Ident, DepElementPin)], dims_ctx: Option<&crate::dimensions::DimensionsContext>, gf_table_ref: Option<&str>, occ: &OccurrenceLookup<'_>, @@ -2439,16 +2549,18 @@ pub(crate) fn generate_scalar_to_element_equation( return Err(PartialEquationError::unfreezable(&print_eqn(to_elem_eqn))); } let partial = print_eqn(&substitute_reducers_in_expr0(wrapped, reducer_subst)); - let mut partial = subscript_idents_at_element(&partial, to_deps_to_subscript, element)?; + let mut partial = subscript_idents_at_element(&partial, to_deps_to_subscript)?; // Pin each mapped ident's references (the live source's numerator // occurrence and any PREVIOUS-wrapped co-agg freezes alike) to its own - // projected slot -- separate passes from the full-tuple - // `to_deps_to_subscript` pinning above, because each agg's slot is the - // projection onto ITS `result_dims`, not the target's full tuple. The - // passes touch disjoint idents, so application order is irrelevant. - for (ident, slot) in source_pins { - let one: HashSet> = std::iter::once(ident.clone()).collect(); - partial = subscript_idents_at_element(&partial, &one, slot)?; + // projected slot -- separate passes from the `to_deps_to_subscript` pinning + // above, because each agg's slot is the projection onto ITS `result_dims` + // rather than onto its declared dimensions (an agg is not a model variable + // and has none). The passes touch disjoint idents, so application order is + // irrelevant. + for (ident, pin) in source_pins { + let one: HashMap, DepElementPin> = + std::iter::once((ident.clone(), pin.clone())).collect(); + partial = subscript_idents_at_element(&partial, &one)?; } // The wrap goes on LAST because it is not part of the target's equation: // everything above rewrites that equation within its own (gf-input) domain @@ -2503,7 +2615,7 @@ pub(crate) fn generate_per_element_link_equation( element_qualified: &str, to_elem_eqn: &Expr0, to_deps: &HashSet>, - to_deps_to_subscript: &HashSet>, + to_deps_to_subscript: &HashMap, DepElementPin>, from_dims: &[crate::dimensions::Dimension], target_elem_by_dim: &HashMap, target_iterated_dims: &[String], @@ -2527,6 +2639,7 @@ pub(crate) fn generate_per_element_link_equation( row_parts_bare, from_dims, target_elem_by_dim, + target_iterated_dims, dim_ctx: dims_ctx, }; // The ceteris-paribus wrap runs on the target element's OWN equation, @@ -2577,8 +2690,7 @@ pub(crate) fn generate_per_element_link_equation( return Err(PartialEquationError::unfreezable(&print_eqn(to_elem_eqn))); } let partial = print_eqn(&wrapped); - let mut partial = - subscript_idents_at_element(&partial, to_deps_to_subscript, element_qualified)?; + let mut partial = subscript_idents_at_element(&partial, to_deps_to_subscript)?; if let Some(table_ref) = gf_table_ref { partial = format!("LOOKUP({table_ref}, {partial})"); } diff --git a/src/simlin-engine/src/ltm_augment_occurrence.rs b/src/simlin-engine/src/ltm_augment_occurrence.rs index 0dece32a8..c0f196aa8 100644 --- a/src/simlin-engine/src/ltm_augment_occurrence.rs +++ b/src/simlin-engine/src/ltm_augment_occurrence.rs @@ -98,6 +98,32 @@ impl<'a> SlotOccurrences<'a> { } } +impl OccurrenceLookup<'static> { + /// The lookup for a subtree the IR records NOTHING for, by design. + /// + /// There is exactly one such subtree: a `LOOKUP` TABLE argument, which + /// `db::ltm_ir`'s walker skips whole ("a graphical-function table reference + /// is static data, not a causal edge"). The wrap descends into that + /// argument's subscript INDEX expressions anyway, because those ARE runtime + /// value reads and have to be frozen for ceteris paribus (GH #984) -- and + /// descending with the slot's real lookup would be unsound in two ways at + /// once. A path hit under a table argument would be a COLLISION (the paths + /// there belong to no recorded occurrence), and a MISS on a live-source + /// subscript would trip the desync guard, whose premise -- an occurrence for + /// every live-source subscript head -- is exactly what does not hold here. + /// + /// With an empty lookup the guard is inert (it keys on `is_empty`) and every + /// shape lookup misses uniformly, so nothing under a table argument is + /// treated as the live reference: a source read there is frozen, which is + /// the conservative and ceteris-paribus-correct answer for a reference the + /// IR attributes nothing to. The `PerElement` row pinning is unaffected -- + /// it discharges a table argument structurally, by name, needing no + /// occurrence at all. + pub(super) fn empty() -> Self { + OccurrenceLookup { entries: &[] } + } +} + impl<'a> OccurrenceLookup<'a> { /// The occurrence at exactly `path`, if any (a genuine causal reference at /// that node). `None` for a node that is not a recorded occurrence (a diff --git a/src/simlin-engine/src/ltm_augment_pin_tests.rs b/src/simlin-engine/src/ltm_augment_pin_tests.rs index bf150fe8c..c13d36247 100644 --- a/src/simlin-engine/src/ltm_augment_pin_tests.rs +++ b/src/simlin-engine/src/ltm_augment_pin_tests.rs @@ -78,6 +78,7 @@ fn per_element_pin_descends_into_range_endpoints() { row_parts_bare: &row_parts_bare, from_dims: &from_dims, target_elem_by_dim: &target_elem_by_dim, + target_iterated_dims: &target_iterated_dims, dim_ctx: &dim_ctx, }; @@ -105,7 +106,7 @@ fn per_element_pin_descends_into_range_endpoints() { ); let mut unlowerable = false; let lowered = - super::post_transform::pin_only_source_refs(ast, &ctx, &occ, &[], &mut unlowerable, true); + super::post_transform::pin_only_source_refs(ast, &ctx, &occ, &[], &mut unlowerable); assert!( !unlowerable, "the range-bound occurrence IS recorded, so the pin must lower it rather than \ @@ -125,6 +126,190 @@ fn per_element_pin_descends_into_range_endpoints() { ); } +/// The full ENUMERATION of [`super::post_transform::dep_element_pins`]: how a +/// dep's DECLARED dimensions relate to the target element, and what pin each +/// relation yields (GH #974). +/// +/// This is the projection behind every bare-reference pin -- the rule +/// `subscript_idents_at_element` consumes and the rule `pin_bare_source_ref` +/// reuses for the live source -- so the rows are derived from the two +/// enumerations it composes, not from examples: +/// +/// - per AXIS, [`dep_axis_elements`](super::post_transform) has three outcomes: +/// the target ITERATES the dep's own dimension (identity), the target +/// iterates a dimension with a usable positional CORRESPONDENCE to it +/// (`mapped_element_correspondence`), or neither (declined). The declined arm +/// is reached two ways -- an unrelated dimension, and an explicit ELEMENT map, +/// which the correspondence refuses because execution resolves positionally +/// and ignores it (GH #756) -- and both are rows; +/// - per DEP, the axis outcomes combine into three: every axis resolved +/// (`complete`, the only kind that may subscript a bare reference), some +/// resolved (present but incomplete -- it may still substitute a +/// dimension-name index), none resolved (absent from the table entirely). +/// +/// The order rows exist because the pin is spelled in the DEP's declaration +/// order, not the target's: `flip[Age,Region]` under a `growth[Region,Age]` +/// target is the shape whose pre-fix full-target-tuple pin COMPILED and read +/// the wrong element. +/// +/// One arm is deliberately unrowed: `per_element_row_for_target` can also +/// decline an axis whose correspondence has no entry at the target element's +/// index. That is a mid-edit dimension inconsistency (a mapping whose two +/// dimensions differ in size), it declines exactly like the rows here, and no +/// well-formed project reaches it. +#[test] +fn dep_element_pins_projection_enumeration() { + use crate::dimensions::DimensionsContext; + + // `state` is POSITIONALLY mapped to `region`, so a `State` axis reads the + // region coordinate's positional partner; `other` is unrelated to both. + let build_ctx = |element_map: Vec<(String, String)>| { + let mut state = datamodel::Dimension::named( + "state".to_string(), + vec!["west".to_string(), "east".to_string()], + ); + state.mappings = vec![datamodel::DimensionMapping { + target: "region".to_string(), + element_map, + }]; + DimensionsContext::from( + [ + datamodel::Dimension::named( + "region".to_string(), + vec!["nyc".to_string(), "boston".to_string()], + ), + datamodel::Dimension::named( + "age".to_string(), + vec!["young".to_string(), "old".to_string()], + ), + state, + datamodel::Dimension::named( + "other".to_string(), + vec!["o1".to_string(), "o2".to_string()], + ), + ] + .as_slice(), + ) + }; + let region = make_named_dimension("region", &["nyc", "boston"]); + let age = make_named_dimension("age", &["young", "old"]); + let state = make_named_dimension("state", &["west", "east"]); + let other = make_named_dimension("other", &["o1", "o2"]); + + // Target `growth[Region,Age]` at element `(boston, old)` -- both + // coordinates are the SECOND element of their dimension, which is what + // makes the positional correspondence observable (`state·east`). + let target_iterated_dims = vec!["region".to_string(), "age".to_string()]; + let mut target_elem_by_dim: HashMap = HashMap::new(); + target_elem_by_dim.insert("region".to_string(), ("boston".to_string(), 1usize)); + target_elem_by_dim.insert("age".to_string(), ("old".to_string(), 1usize)); + + let pinnable: Vec<(Ident, Vec)> = vec![ + // identity, in the target's own order + (Ident::new("same"), vec![region.clone(), age.clone()]), + // identity, REORDERED -- the silent-wrong-element row + (Ident::new("flip"), vec![age.clone(), region.clone()]), + // a strict SUBSET -- the arity row + (Ident::new("sub"), vec![age.clone()]), + // a positionally MAPPED axis beside an identity one + (Ident::new("mapped"), vec![state.clone(), age.clone()]), + // one axis resolves, one does not -> incomplete + (Ident::new("partial"), vec![region.clone(), other.clone()]), + // nothing resolves -> absent + (Ident::new("unrelated"), vec![other.clone()]), + ]; + + let axes_of = |pins: &HashMap, crate::ltm_augment::DepElementPin>, + name: &str| + -> Option<(Vec<(String, String)>, bool)> { + pins.get(&Ident::::new(name)) + .map(|p| (p.axes.clone(), p.complete)) + }; + let axis = |dim: &str, elem: &str| (dim.to_string(), elem.to_string()); + + let positional = build_ctx(vec![]); + let pins = super::post_transform::dep_element_pins( + &pinnable, + &target_iterated_dims, + &target_elem_by_dim, + &positional, + ); + + assert_eq!( + axes_of(&pins, "same"), + Some(( + vec![ + axis("region", "region\u{B7}boston"), + axis("age", "age\u{B7}old") + ], + true + )), + "a dep declaring the target's own dimensions in the target's order pins \ + to the target element" + ); + assert_eq!( + axes_of(&pins, "flip"), + Some(( + vec![ + axis("age", "age\u{B7}old"), + axis("region", "region\u{B7}boston") + ], + true + )), + "a REORDERED dep must be pinned in ITS OWN declaration order -- the \ + target's tuple would compile and read the transposed element" + ); + assert_eq!( + axes_of(&pins, "sub"), + Some((vec![axis("age", "age\u{B7}old")], true)), + "a subset-dims dep must be pinned over its own single axis, not the \ + target's full tuple" + ); + assert_eq!( + axes_of(&pins, "mapped"), + Some(( + vec![ + axis("state", "state\u{B7}east"), + axis("age", "age\u{B7}old") + ], + true + )), + "a positionally-mapped axis reads the corresponding element of its own \ + dimension (`boston` is Region's second, so State's second is `east`)" + ); + assert_eq!( + axes_of(&pins, "partial"), + Some((vec![axis("region", "region\u{B7}boston")], false)), + "a dep with one unprojectable axis stays in the table (its dimension-name \ + indices are still substitutable) but is NOT complete" + ); + assert_eq!( + axes_of(&pins, "unrelated"), + None, + "a dep no axis of which projects has nothing to rewrite and must be absent" + ); + + // An EXPLICIT element map is declined by `mapped_element_correspondence` + // (execution resolves positionally and ignores it, GH #756), so the same + // `mapped[State,Age]` dep loses its State axis and becomes incomplete. + let element_mapped = build_ctx(vec![ + ("west".to_string(), "boston".to_string()), + ("east".to_string(), "nyc".to_string()), + ]); + let pins = super::post_transform::dep_element_pins( + &pinnable, + &target_iterated_dims, + &target_elem_by_dim, + &element_mapped, + ); + assert_eq!( + axes_of(&pins, "mapped"), + Some((vec![axis("age", "age\u{B7}old")], false)), + "an element-mapped axis must decline: following the map would spell a \ + read the positionally-resolving simulation never performs" + ); +} + /// The shared `(Region, Age)` context for the `PerElement` pin-only-descent /// tests: source `pop[Region, Age]` (`region = [nyc, boston]`, /// `age = [young, old]`), target `growth[Region]` instantiated at @@ -389,7 +574,7 @@ impl PinFixture { ast, deps, // Nothing to element-pin by name: the source's pinning is the wrap's job. - &HashSet::new(), + &HashMap::new(), &self.from_dims, &self.target_elem_by_dim, &self.target_iterated_dims, @@ -592,71 +777,79 @@ fn per_element_pin_lowers_structurally_inside_a_lookup_table_arg() { ); } -/// The loud net BEHIND the structural pin: a BARE table argument whose index the -/// rule cannot discharge statically is declined, and the whole partial abandoned. +/// GH #984, both halves at once: a `LOOKUP` table argument's HEAD stays bare and +/// its runtime INDEX gets frozen. +/// +/// The two halves pull against each other, which is why the issue asks for one +/// test that pins both. The head must NOT be wrapped: a graphical-function table +/// has no value slot, so `lookup(PREVIOUS(table), ...)` cannot compile and the +/// whole fragment silently zeroes -- the WRLD3 failure mode. But the index IS a +/// value read: `codegen::extract_table_info`'s `Expr::Subscript` arm builds the +/// element offset out of the live index `Expr`s, so a live `idx` made EVERY +/// partial of that target vary with `idx`'s current-step movement, attributing it +/// to whichever source the partial isolates. Holding the whole ARGUMENT verbatim +/// bought the first at the price of the second. /// -/// The rule resolves exactly three index forms -- the axis's own dimension name, -/// an element that axis declares, and a numeric literal. Everything else is -/// declined here, for one of two DIFFERENT reasons, and the difference matters -/// because only one of them is unconditional: +/// The rows are the three shapes an index read can take -- a bare variable, a +/// compound expression around one, and a nested source subscript -- so the +/// freeze is shown to reach inside a compound index (`PREVIOUS(idx) + 1`, not +/// `PREVIOUS(idx + 1)`) rather than only replacing whole indices. The static +/// selectors, the unspellable ones, and the frozen-context column are the +/// sibling [`per_element_pin_index_verdict_enumeration`]'s job. /// -/// - a RUNTIME read (`pop[Region, idx]`, `pop[Region, pop[Region, old]]`) COMPILES -/// fine; the problem is ceteris paribus. A BARE `LOOKUP` table argument is not -/// inside a freeze -- the wrap holds it verbatim, since a `PREVIOUS` of a table -/// has no value slot -- so the index stays LIVE, and -/// `codegen::extract_table_info` evaluates it to select the table element. The -/// `young` partial would then vary with the current-step value of `old`, -/// misattributing one row's influence to another. Pinning these purely because -/// it made the fragment COMPILE is what a reviewer caught on this branch. Inside -/// a freeze the same index is already lagged and is KEPT -- see -/// [`per_element_pin_keeps_a_runtime_table_index_inside_a_freeze`]; -/// - the rule having NO ANSWER (`pop[State, old]` on a source declared over -/// `[Region, Age]`) is a compilability verdict and is loud everywhere: resolving -/// it would need to decide that `State` reads `Region` through a positional -/// mapping rather than being a mismatched or transposed axis, and that per-axis -/// CLASSIFICATION is `db::ltm_ir`'s -- which recorded nothing here. +/// Before the fix each of these was a loud DECLINE +/// (`WrapOutcome::missing_occurrence` -> a warned skip), which threw away every +/// per-element score on the edge including the live site's own. Freezing keeps +/// the score AND makes it ceteris-paribus. /// -/// Together with [`per_element_pin_lowers_structurally_inside_a_lookup_table_arg`] -/// and the frozen twin, this is the whole contract: statically-resolvable table -/// arguments are pinned and scored, already-lagged runtime ones are pinned as far -/// as they go, and the rest stays loud instead of emitting either an un-pinned -/// dimension-name subscript (a dropped fragment reading 0) or a plausible wrong -/// score. +/// **`idx` is deliberately NOT in the declared dep set**, and that is the whole +/// difference between a fixture and an oracle here. Production derives the wrap's +/// dep set from `variable::identifier_set`, whose `BuiltinContents::LookupTable` +/// arm does not walk the table expression, so an index variable referenced only +/// inside a table argument is NOT a dependency -- asserted below on the fixture's +/// own equation rather than assumed. Declaring `idx` a dep would make the freeze +/// fire through the ordinary other-dep path and the test would pass on an input +/// production cannot produce, which is exactly how the first version of this fix +/// shipped without firing at all. #[test] -fn per_element_pin_declines_every_non_static_bare_table_arg_index() { - // Each case is `(label, equation, deps, extra project dims)`. `state` is a - // real dimension positionally parallel to `region` -- the shape whose - // resolution would need the mapping classifier. - let cases: [(&str, &str, &[&str], Vec); 3] = [ +fn per_element_pin_freezes_a_runtime_table_arg_index() { + // `(label, table argument, the expected frozen spelling)`. + let cases: [(&str, &str, &str); 3] = [ ( "a variable index -- the simplest runtime element read", - "pop[Region, young] + LOOKUP(pop[Region, idx], input)", - &["input", "idx"], - vec![], + "pop[Region, idx]", + "lookup(pop[region\u{B7}boston, PREVIOUS(idx)]", ), ( - "a nested source read selecting the element at runtime", - "pop[Region, young] + LOOKUP(pop[Region, pop[Region, old]], input)", - &["input"], - vec![], + "a compound index: the freeze reaches the read, not the whole expression", + "pop[Region, idx + 1]", + "lookup(pop[region\u{B7}boston, PREVIOUS(idx) + 1]", ), ( - "another dimension's name, which only the IR can classify", - "pop[Region, young] + LOOKUP(pop[State, old], input)", - &["input"], - vec![datamodel::Dimension::named( - "state".to_string(), - vec!["ny".to_string(), "ma".to_string()], - )], + "a nested source read selecting the element at runtime", + "pop[Region, pop[Region, old]]", + "lookup(pop[region\u{B7}boston, \ + PREVIOUS(pop[region\u{B7}boston, age\u{B7}old])]", ), ]; - for (label, eqn, deps, extra_dims) in cases { - let fx = PinFixture::new(extra_dims); - let (ast, deps, occurrences) = fx.parse(eqn, deps); + for (label, table_arg, expected) in cases { + let fx = PinFixture::new(vec![]); + let eqn = format!("pop[Region, young] + LOOKUP({table_arg}, input)"); + // The dep list is the production one for this equation: `input` is a + // dependency and `idx` is not. Checked against the extractor itself, so + // the fixture cannot drift from what production supplies. + assert!( + !crate::variable::identifier_set(&crate::variable::scalar_ast(&eqn), &[], None) + .contains(&Ident::::new("idx")), + "{label}: production's dep extractor must NOT report a table-only \ + index as a dependency, or this fixture is testing the ordinary \ + other-dep path instead of the table-index freeze" + ); + let (ast, deps, occurrences) = fx.parse(&eqn, &["input"]); // Non-vacuity: the table argument is the reference WITHOUT an occurrence - // (the IR skips it whole), so the decline must come from the structural - // rule rather than from the IR. + // (the IR skips it whole), so the freeze comes from the wrap's own index + // pass rather than from the IR's classification. assert_eq!( PinFixture::source_occurrences(&occurrences), 1, @@ -664,17 +857,24 @@ fn per_element_pin_declines_every_non_static_bare_table_arg_index() { {occurrences:?}" ); let slot_occurrences = SlotOccurrences::new(&occurrences); - let err = fx.generate(&ast, &deps, &slot_occurrences.for_slot(0)); + let text = fx + .generate(&ast, &deps, &slot_occurrences.for_slot(0)) + .unwrap_or_else(|e| { + panic!("{label}: the partial must be emitted, not declined: {e:?}") + }); assert!( - matches!( - err, - Err(PartialEquationError { - kind: PartialEquationErrorKind::UnfreezablePartial, - .. - }) - ), - "{label}: must be a loud Err, never an Ok equation carrying an \ - un-pinned subscript or an unattributed live read: {err:?}" + text.contains(expected), + "{label}: expected the frozen index {expected:?}; got: {text}" + ); + assert!( + !text.contains("lookup(PREVIOUS("), + "{label}: the table HEAD must stay bare -- a PREVIOUS of a table has no \ + value slot; got: {text}" + ); + assert!( + text.contains("pop[boston, young]"), + "{label}: the LIVE occurrence must keep its bare row spelling -- the \ + pre-fix decline threw its score away too; got: {text}" ); } } @@ -706,7 +906,11 @@ fn per_element_pin_keeps_a_runtime_table_index_inside_a_freeze() { ), ] { let fx = PinFixture::new(vec![]); - let (ast, deps, occurrences) = fx.parse(eqn, &["input", "idx", "other"]); + // `idx` is not declared: production does not classify a table-only index + // as a dependency (see `per_element_pin_freezes_a_runtime_table_arg_index`), + // and this test's point is that the enclosing freeze -- not the dep set -- + // is what makes the index ceteris-paribus here. + let (ast, deps, occurrences) = fx.parse(eqn, &["input", "other"]); assert_eq!( PinFixture::source_occurrences(&occurrences), 1, @@ -786,9 +990,11 @@ fn per_element_pin_keeps_a_runtime_table_index_inside_a_freeze() { /// Everything here is already lagged by the enclosing `PREVIOUS`, index reads /// included, so pinning is sufficient and NO further freeze is wanted (a nested /// `PREVIOUS` would read two steps back). That is exactly what distinguishes this -/// from the table-argument case, which is not inside a freeze and is therefore -/// declined -- see -/// [`per_element_pin_declines_a_table_arg_with_a_runtime_index`]. +/// from an UNFROZEN table argument, whose index is a live read and so is frozen +/// rather than merely pinned -- see +/// [`per_element_pin_freezes_a_runtime_table_arg_index`], and +/// [`per_element_pin_keeps_a_runtime_table_index_inside_a_freeze`] for the same +/// argument inside a freeze, where this test's reasoning applies instead. #[test] fn per_element_pin_descends_into_a_source_subscript_index_expression() { let fx = PinFixture::new(vec![]); @@ -843,6 +1049,13 @@ type PinIndexCell<'a> = (&'a str, &'a str, Option<&'a str>, Option<&'a str>); /// `fx`'s own shape -- it is what makes the equation a `PerElement` target at all, /// and it is deliberately not the cell under test. The cell under test is the /// table argument, which the IR records NOTHING for. +/// +/// The declared dep list is `["input"]` and NOT `["input", "idx"]`, matching what +/// `variable::identifier_set` reports for these equations: a variable appearing +/// only as a table-argument subscript index is not a dependency, so a row that +/// declared it would exercise the ordinary other-dep freeze instead of the +/// table-index one. `per_element_pin_freezes_a_runtime_table_arg_index` asserts +/// that against the extractor itself. fn assert_pin_index_verdicts(fx: &PinFixture, live_ref: &str, cases: &[PinIndexCell<'_>]) { // A DIMENSION-name subscript surviving into a scalar fragment is the silent // zero this whole rule exists to prevent, so every cell forbids one on any @@ -861,7 +1074,7 @@ fn assert_pin_index_verdicts(fx: &PinFixture, live_ref: &str, cases: &[PinIndexC } else { format!("{live_ref} + LOOKUP({subscript}, input)") }; - let (ast, deps, occurrences) = fx.parse(&eqn, &["input", "idx"]); + let (ast, deps, occurrences) = fx.parse(&eqn, &["input"]); // Non-vacuity, every cell: the table argument is the reference the IR // records NOTHING for, so the rule is what decides -- not the IR. assert_eq!( @@ -922,9 +1135,10 @@ fn assert_pin_index_verdicts(fx: &PinFixture, live_ref: &str, cases: &[PinIndexC /// landing in the wrong bucket. Each was fixed after someone supplied the /// counterexample. This test exists so the space is stated rather than sampled: /// the rule sorts an index into exactly one of -/// [`super::post_transform::IndexVerdict`]'s four outcomes, and the outcome -/// depends on the index's spelling and (for `RuntimeRead` alone) on whether the -/// subtree is already frozen. Two columns, eighteen rows, no gaps. +/// [`super::post_transform::IndexVerdict`]'s three outcomes -- `Pinned`, `Keep`, +/// `Unspellable` -- purely on the index's spelling, and the WRAP separately +/// decides whether a runtime read needs freezing. Two columns, eighteen rows, no +/// gaps. /// /// "No gaps" is a checkable claim, not a hope: `IndexExpr0` has exactly five /// variants and every one has a row -- `Wildcard`, `StarRange`, `Range`, @@ -952,19 +1166,22 @@ fn assert_pin_index_verdicts(fx: &PinFixture, live_ref: &str, cases: &[PinIndexC /// /// Reading the table: see [`PinIndexCell`]. The unfrozen column is a bare `LOOKUP` /// table argument; the frozen column is the same argument inside a pre-existing -/// `PREVIOUS`. Only the runtime-read rows differ between columns -- that is the -/// whole content of the compilability-vs-ceteris-paribus split. +/// `PREVIOUS`. Only the runtime-read rows differ between columns, and since +/// GH #984 the difference is a FREEZE rather than a refusal: a bare table +/// argument's index is wrapped in `PREVIOUS` by +/// `ltm_augment::freeze_lookup_table_indices`, while the same index inside a +/// pre-existing freeze is already lagged and is left alone. Every static row is +/// identical in both columns, and every unspellable row is loud in both. /// -/// Two rows are marked UNREACHABLE and assert current behavior rather than a +/// Three rows are marked UNREACHABLE and assert current behavior rather than a /// derived requirement, because a compilable model cannot produce them: codegen's -/// `extract_table_info` rejects a range or wildcard table index outright -/// (`BadTable`, "range subscripts not supported in lookup tables"), so the -/// TARGET's own equation would fail to compile long before its link scores are -/// generated. Their frozen cells are therefore honestly uncertain: the rule -/// currently accepts them, and if they ever became reachable the failure would -/// surface as a fragment-compile Warning rather than a warned skip -- loud either -/// way, through a different channel. Left as-is deliberately rather than guessed -/// into `Unspellable`, since no reachable shape distinguishes the two choices. +/// `extract_table_info` rejects a range, wildcard or star-range table index +/// outright (`BadTable`, "range subscripts not supported in lookup tables"), so +/// the TARGET's own equation would fail to compile long before its link scores are +/// generated. If one ever became reachable the failure would surface as a +/// fragment-compile Warning rather than a warned skip -- loud either way, through +/// a different channel. Left as-is deliberately rather than guessed into +/// `Unspellable`, since no reachable shape distinguishes the two choices. /// /// One further row (the numeric literal) is about the RULE's verdict, not about /// downstream compilability: a numeric index into a NAMED dimension is not @@ -1022,18 +1239,17 @@ fn per_element_pin_index_verdict_enumeration() { Some("pop[region\u{B7}boston, 1 + 1]"), ), ( - // A 0-arity BUILTIN index is where the widening deliberately STOPS. - // `reify_0_arity_builtins` turns `time` into an `Expr0::App` before this - // rule sees it, and telling `TIME` (varies every step) from `PI` (does - // not) inside `App` would mean a fourth copy of the builtin - // classification `builtins`/`compiler::invariance` own. So the arm stays - // loud in the bare column -- the CONSERVATIVE direction, a warned skip - // rather than a confident wrong score. A stated boundary, not a gap: if - // this ever needs to pin, the fix is to consult that classification, not - // to guess here. - "a 0-arity builtin index (deliberately still loud)", + // A 0-arity BUILTIN index used to be loud in the bare column, because + // the rule could not tell `TIME` (varies every step) from `PI` (does + // not) inside an `Expr0::App` without a fourth copy of the builtin + // classification `builtins`/`compiler::invariance` own -- so it + // declined conservatively. It no longer has to decide: the rule keeps + // the index either way, and the wrap's index pass leaves a 0-arity + // builtin live exactly as it does everywhere else in a partial (TIME + // is not a dep being isolated, and the guard form reads it live too). + "a 0-arity builtin index", "pop[Region, TIME]", - None, + Some("pop[region\u{B7}boston, time()]"), Some("pop[region\u{B7}boston, time()]"), ), // --- unspellable: a COMPILABILITY verdict, so loud in BOTH contexts ---- @@ -1079,42 +1295,48 @@ fn per_element_pin_index_verdict_enumeration() { None, None, ), - // --- runtime reads: a CETERIS-PARIBUS verdict, so context decides ------ + // --- runtime reads: kept by the rule, FROZEN by the wrap when bare ----- + // The two columns differ by exactly one `PREVIOUS`, which IS the GH #984 + // fix: a bare table argument's index read is lagged, and the same read + // inside a pre-existing freeze is left alone rather than double-lagged. ( "an undeclared bare name -- a variable selecting the element", "pop[Region, idx]", - None, + Some("pop[region\u{B7}boston, PREVIOUS(idx)]"), Some("pop[region\u{B7}boston, idx]"), ), ( "an arithmetic index expression", "pop[Region, idx + 1]", - None, + Some("pop[region\u{B7}boston, PREVIOUS(idx) + 1]"), Some("pop[region\u{B7}boston, idx + 1]"), ), ( "a nested source subscript selecting the element", "pop[Region, pop[Region, young]]", - None, + Some( + "pop[region\u{B7}boston, \ + PREVIOUS(pop[region\u{B7}boston, age\u{B7}young])]", + ), Some("pop[region\u{B7}boston, pop[region\u{B7}boston, age\u{B7}young]]"), ), // --- UNREACHABLE from a compilable model -- see fn docs --------------- ( "a range index (UNREACHABLE: codegen rejects it as a table index)", "pop[Region, 1:2]", - None, + Some("pop[region\u{B7}boston,"), Some("pop[region\u{B7}boston,"), ), ( "a wildcard index (UNREACHABLE: codegen rejects it as a table index)", "pop[Region, *]", - None, + Some("pop[region\u{B7}boston,"), Some("pop[region\u{B7}boston,"), ), ( "a star-range index (UNREACHABLE: codegen rejects it as a table index)", "pop[Region, *:Age]", - None, + Some("pop[region\u{B7}boston,"), Some("pop[region\u{B7}boston,"), ), ]; diff --git a/src/simlin-engine/src/ltm_augment_post_transform.rs b/src/simlin-engine/src/ltm_augment_post_transform.rs index 134771978..716c2aeed 100644 --- a/src/simlin-engine/src/ltm_augment_post_transform.rs +++ b/src/simlin-engine/src/ltm_augment_post_transform.rs @@ -47,7 +47,7 @@ use crate::ast::{Expr0, IndexExpr0, print_eqn}; use crate::builtins::UntypedBuiltinFn; use crate::common::{Canonical, Ident, RawIdent}; -use super::{OccurrenceLookup, qualify_element_csv}; +use super::{DepElementPin, OccurrenceLookup, qualify_element_csv}; use crate::db::ltm_ir::{OccurrenceAxis, OccurrenceSite}; #[cfg(test)] @@ -82,6 +82,11 @@ pub(super) struct PerElementRefCtx<'a> { /// Target-iterated dim (canonical) -> (element of `e` for that dim, /// its index within the dim) -- the projection data `e` supplies. pub(super) target_elem_by_dim: &'a HashMap, + /// The target equation's iterated dimensions, canonical and IN ORDER. + /// Same set as `target_elem_by_dim`'s keys; carried separately because + /// [`dep_row_for_target`]'s mapped search must be deterministic and a + /// `HashMap`'s iteration order is not. + pub(super) target_iterated_dims: &'a [String], pub(super) dim_ctx: &'a crate::dimensions::DimensionsContext, } @@ -135,6 +140,137 @@ pub(crate) fn per_element_row_for_target( .collect() } +/// Per DECLARED dimension of a dep, the element it reads at ONE target element +/// -- `None` for a dimension this target element projects onto neither by name +/// nor through a positional mapping. Bare element names, in the dep's own +/// declaration order. +/// +/// A bare arrayed reference inside an apply-to-all body reads the element the +/// enclosing iteration selects FOR EACH OF ITS OWN AXES, matched by dimension +/// NAME and not by position: `growth[Region,Age] = ... w ...` over a `w[Age]` +/// reads `w[]`, and over a `w[Age,Region]` reads +/// `w[,]` -- the transpose of the target's own tuple. That is the +/// executed behaviour (pinned by +/// `bare_arrayed_dep_is_pinned_over_its_own_declared_dims`' numeric oracle), +/// and it is why pinning such a reference with the target's FULL element tuple +/// is wrong twice over: over- or under-arity when the dep declares a strict +/// subset (a fragment that fails to compile, so the score reads a constant 0), +/// and a compilable, SILENTLY WRONG element when it declares the same +/// dimensions in another order (GH #974). +/// +/// The per-axis question -- which target coordinate projects onto this +/// dimension -- is handed to [`per_element_row_for_target`], the single row +/// derivation, as an [`crate::ltm_agg::AxisRead::Iterated`] pair. That is what +/// makes the identity axis and a positionally-MAPPED one (`w[Region]` read from +/// a `growth[State,..]` body under a `State`/`Region` mapping) one arm instead +/// of two, and it means this accepts exactly the mapped pairs the +/// occurrence-driven pin and `ltm_agg::classify_axis_access` accept. +/// +/// The mapped search walks `target_iterated_dims` in the target's declared +/// ORDER, so a dep dimension reachable from two mapped target dimensions +/// resolves the same way on every run (`target_elem_by_dim` is a `HashMap` +/// and cannot be iterated for this). +fn dep_axis_elements( + dep_dims: &[crate::dimensions::Dimension], + target_iterated_dims: &[String], + target_elem_by_dim: &HashMap, + dim_ctx: &crate::dimensions::DimensionsContext, +) -> Vec> { + use crate::common::CanonicalDimensionName; + use crate::ltm_agg::AxisRead; + dep_dims + .iter() + .map(|dep_dim| { + let source_dim = dep_dim.name().to_string(); + // The target iterating the dep's OWN dimension is the identity + // projection and always wins; only a name the target does not + // iterate goes looking for a mapped partner. + let target_dim = if target_elem_by_dim.contains_key(&source_dim) { + source_dim.clone() + } else { + target_iterated_dims + .iter() + .find(|td| { + dim_ctx + .mapped_element_correspondence( + &CanonicalDimensionName::from_raw(td), + dep_dim.canonical_name(), + ) + .is_some() + })? + .clone() + }; + let axis = AxisRead::Iterated { + dim: target_dim, + source_dim, + }; + per_element_row_for_target(std::slice::from_ref(&axis), target_elem_by_dim, dim_ctx) + .map(|row| row[0].clone()) + }) + .collect() +} + +/// The element a variable declared over `dep_dims` reads at ONE target element, +/// one bare element name per declared dimension -- [`dep_axis_elements`] with +/// every axis resolved. `None` when some declared dimension does not project, +/// because a BARE reference has to be spelled at the dep's full arity or not +/// at all. +pub(crate) fn dep_row_for_target( + dep_dims: &[crate::dimensions::Dimension], + target_iterated_dims: &[String], + target_elem_by_dim: &HashMap, + dim_ctx: &crate::dimensions::DimensionsContext, +) -> Option> { + dep_axis_elements(dep_dims, target_iterated_dims, target_elem_by_dim, dim_ctx) + .into_iter() + .collect() +} + +/// The element-pin table for ONE target element: each dep in `pinnable` mapped +/// to the elements IT reads there ([`DepElementPin`]), qualified in its own +/// dimensions' space so a frozen read compiles to a direct LoadPrev. +/// +/// A dep no axis of which projects is ABSENT from the table entirely (nothing +/// to rewrite). A dep only SOME of whose axes project is present but not +/// [`complete`](DepElementPin::complete): its dimension-name indices are still +/// substituted -- that is the GH #654 helper-aux fix, and it only needs the +/// axes the reference actually spells as dimension names -- while a BARE +/// reference to it is left alone, since no correct full-arity subscript exists. +/// Leaving it bare is the loud direction: a bare multi-slot reference in a +/// scalar fragment fails to compile and surfaces an `Assembly` warning, where +/// the pre-GH #974 full-target-tuple pin silently mis-read the dep whenever the +/// arity happened to match. +/// +/// `pinnable` carries each dep's declared `Dimension`s, resolved ONCE per +/// target equation by the caller; only the projection is per element. +pub(crate) fn dep_element_pins( + pinnable: &[(Ident, Vec)], + target_iterated_dims: &[String], + target_elem_by_dim: &HashMap, + dim_ctx: &crate::dimensions::DimensionsContext, +) -> HashMap, DepElementPin> { + pinnable + .iter() + .filter_map(|(ident, dep_dims)| { + let elems = + dep_axis_elements(dep_dims, target_iterated_dims, target_elem_by_dim, dim_ctx); + let complete = elems.iter().all(Option::is_some); + let axes: Vec<(String, String)> = elems + .iter() + .zip(dep_dims) + .filter_map(|(elem, dim)| { + elem.as_ref() + .map(|e| (dim.name().to_string(), qualify_axis_element(e, dim))) + }) + .collect(); + if axes.is_empty() { + return None; + } + Some((ident.clone(), DepElementPin { axes, complete })) + }) + .collect() +} + /// The per-axis [`crate::ltm_agg::AxisRead`] slice of a live-source subscript /// occurrence, or `None` when the occurrence is not fully describable per axis. /// @@ -280,20 +416,27 @@ pub(super) fn pin_source_subscript_indices( /// Row-pin a BARE `Var` reference to the live source (the mixed /// `Bare`+`PerElement` edge's other site): each axis reads the target element's -/// coordinate for that axis's own dimension (same-element semantics), qualified. +/// coordinate for that axis's own dimension, qualified. +/// +/// The projection is [`dep_row_for_target`]'s -- the same rule that pins a bare +/// arrayed OTHER-dep -- so a positionally-MAPPED axis resolves through the +/// correspondence rather than being declined (GH #974). Before that it matched +/// dimension names only, so `growth[State,Age] = pop[State,young] * pop` over a +/// `pop[Region,Age]` source found no `Region` coordinate, left the bare +/// reference alone, and the wrap froze it into a bare multi-slot +/// `PREVIOUS(pop)` that cannot compile in a scalar fragment -- every +/// per-element score on the edge silently read a constant 0. +/// /// `None` when some axis does not resolve -- the caller then leaves the bare /// reference for the wrap's conservative freeze. pub(super) fn pin_bare_source_ref(ctx: &PerElementRefCtx<'_>) -> Option> { - let bare_axes: Vec = ctx - .from_dims - .iter() - .map(|d| crate::ltm_agg::AxisRead::Iterated { - dim: d.name().to_string(), - source_dim: d.name().to_string(), - }) - .collect(); - per_element_row_for_target(&bare_axes, ctx.target_elem_by_dim, ctx.dim_ctx) - .map(|row| qualified_row_indices(&row, ctx)) + dep_row_for_target( + ctx.from_dims, + ctx.target_iterated_dims, + ctx.target_elem_by_dim, + ctx.dim_ctx, + ) + .map(|row| qualified_row_indices(&row, ctx)) } /// What [`pin_dimension_name_indices`] can say about ONE index of a source @@ -302,22 +445,26 @@ enum IndexVerdict { /// A static selector this rule rewrites: an iterated coordinate, or a literal /// element qualified with its own axis. Pinned(String), - /// A static selector already spelled the way this rule would spell it (a - /// numeric literal, an `@N` position, an already-`dim·elem` name). - Static, + /// Nothing to do -- the index is left exactly as written. Two kinds land + /// here and the rule treats them alike because nothing downstream + /// distinguishes them: a STATIC selector it would spell the same way (a + /// numeric literal, an `@N` position, an already-`dim·elem` name), and a + /// RUNTIME read (`pop[Region, idx]`) which this rule cannot spell and does + /// not need to. Its ceteris-paribus obligation is discharged before this + /// descent runs: by [`crate::ltm_augment::freeze_lookup_table_indices`] on a + /// bare `LOOKUP` table argument (which is why an index reaching here as a + /// bare `Var` at all means the wrap did not run over it), and by the + /// enclosing freeze on the descents the wrap does not enter -- a pre-existing + /// `PREVIOUS`/`INIT`, a whole-frozen reducer (GH #984). + Keep, /// No pin can spell it, because the SHARED row derivation /// ([`per_element_row_for_target`]) cannot resolve the axis: a dimension the /// target does not iterate, an iterated dimension with no usable positional /// correspondence to this source axis (unmapped, element-mapped, or a /// transposition), an index no axis owns. Left alone it keeps a /// DIMENSION-name subscript, which cannot resolve in a scalar fragment, so - /// this is loud ALWAYS -- a compilability verdict, independent of freezing. + /// this one is LOUD -- a compilability verdict. Unspellable, - /// A RUNTIME read selecting the element: a variable (`pop[Region, idx]`) or a - /// nested expression (`pop[Region, pop[Region, old]]`). It COMPILES as it - /// stands, so this is purely a ceteris-paribus question -- see the `frozen` - /// discussion on [`pin_dimension_name_indices`]. - RuntimeRead, } /// Row-pin a source subscript the occurrence IR deliberately records NOTHING @@ -339,19 +486,13 @@ enum IndexVerdict { /// /// - an index the source's axis at that position DECLARES as an element (or an /// already-`dim·elem`-qualified one) is a literal selector, qualified with that -/// axis (`old` -> `age·old`). This is tried FIRST, because it is the order -/// `compiler::subscript`'s own `normalize_subscripts3` resolves a subscript index -/// in: it looks the name up in the axis's `indexed_elements` and only falls back to -/// the dimension-name (A2A `ActiveDimRef`) reading if that misses. The two readings -/// collide when a dimension declares an element whose name is ALSO a dimension the -/// target iterates (`Category = [Region, x]` beside a `Region` dimension), and -/// there the element must win or this rule spells a row the simulation never reads. -/// (`ltm_agg::classify_axis_access` has the opposite order -- GH #986. That is a -/// precision gap rather than a wrong answer TODAY: its dimension-first reading then -/// finds no mapping and declines the axis, so the reference falls to the -/// conservative path instead of being mis-resolved. It is filed rather than fixed -/// here because that classifier feeds reducer hoisting, the reference-shape IR, and -/// element-edge emission, so reordering it moves decisions well outside this rule.) +/// axis (`old` -> `age·old`). Elements are tried FIRST, and that precedence is +/// not this rule's to choose: it is the shared +/// [`crate::dimensions::resolve_axis_index_name`], which +/// `ltm_agg::classify_axis_access` reads too (GH #986 closed the divergence), +/// and which follows the compiler's own `normalize_subscripts3`. See that +/// function for why element-first is the right order and what the XMILE spec +/// does and does not settle about it. /// Qualifying rather than leaving the element bare matters even though it would very /// likely resolve bare (see `wrap_non_matching_in_previous`'s /// `skip_index_qualification`): the wrap's generic `qualify_element_index` cannot @@ -377,36 +518,35 @@ enum IndexVerdict { /// positionally and ignores it), a transposition, a dimension this target does /// not iterate -- is `IndexVerdict::Unspellable`, and it is unspellable because /// the SHARED derivation says so, not because the name differs; -/// - an index that selects a FIXED element is kept verbatim, because it needs no pin -/// and reads nothing at the current step. Three spellings qualify: a numeric -/// literal, arithmetic over numeric literals (`1 + 1` -- see -/// [`index_expr_selects_a_fixed_element`], whose base case is that literal), and an -/// `@N` POSITION index, which `compiler::context`'s subscript lowering resolves to -/// a concrete element offset in scalar context (which a link-score fragment is). -/// Spelling `@N` out as an element name here would be a SECOND implementation of -/// position syntax living outside the compiler that owns it; keeping it verbatim -/// leaves the one. +/// - anything that is not a bare identifier is kept verbatim, because this rule has +/// nothing to say about it. A numeric literal, arithmetic over literals, and an +/// `@N` POSITION index (which `compiler::context`'s subscript lowering resolves +/// to a concrete element offset in scalar context) all select a FIXED element and +/// need no pin -- spelling `@N` out as an element name here would be a SECOND +/// implementation of position syntax living outside the compiler that owns it. A +/// compound expression selecting the element at RUNTIME (`idx + 1`, a nested +/// source read) needs no pin either: it already compiles, and its +/// ceteris-paribus obligation is discharged before this descent runs. So all of +/// them are `IndexVerdict::Keep` and the rule does not have to tell them apart. /// -/// Everything else is one of the two loud verdicts on [`IndexVerdict`], and -/// `frozen` is what separates them. `IndexVerdict::Unspellable` is loud -/// unconditionally: it is a COMPILABILITY verdict. `IndexVerdict::RuntimeRead` is -/// loud only when `frozen` is false, and that is a CETERIS-PARIBUS verdict: +/// The one LOUD verdict is `IndexVerdict::Unspellable`, and it is a COMPILABILITY +/// verdict: a dimension-name subscript that survives into a scalar fragment does +/// not resolve, so the partial is abandoned rather than emitted. /// -/// - a bare `LOOKUP` table argument is NOT inside a freeze -- the wrap holds it -/// verbatim rather than wrapping it, since a `PREVIOUS` of a table has no value -/// slot -- so a runtime index there stays LIVE in the emitted partial. -/// `codegen::extract_table_info` evaluates it to select the table element, so -/// the partial isolating one row would vary with the current-step value of -/// another. A descent that wraps nothing cannot fix that, and a -/// compilable-but-wrong score is worse than none (GH #311 / #661 / #743), so the -/// partial is abandoned; -/// - but the SAME `LOOKUP` nested inside a pre-existing `PREVIOUS`/`INIT`, inside -/// a whole-frozen reducer, or inside a frozen other-dep's subscript, IS already -/// lagged -- index reads included -- so ceteris paribus already holds and -/// refusing would drop a perfectly good score for no reason. -/// -/// So the caller passes the freeze context it alone knows, exactly as the wrap -/// threads its own `frozen` flag beside `path`. +/// This rule used to be loud about a RUNTIME index too, conditionally on an +/// enclosing freeze, because a bare `LOOKUP` table argument is the one place the +/// wrap wrapped nothing -- so the index stayed live, `codegen::extract_table_info` +/// evaluated it at the current step, and the partial isolating one source varied +/// with another's movement. That is GH #984, and it is now fixed at the source +/// rather than refused here: [`crate::ltm_augment::freeze_lookup_table_indices`] +/// puts the table argument's indices through the wrap's own index pass, having +/// first WIDENED that descent's dep set with the indices' own idents -- without +/// which the freeze would not fire at all, since `classify_dependencies` does not +/// walk a table expression and so reports no dependency for an index variable +/// referenced only there. A runtime index therefore arrives here already frozen +/// (or, inside an enclosing freeze, already lagged) for ANY ident, not just one +/// that happens to be a dependency elsewhere. One rule discharges it on every +/// path, this rule keeps it, and the refusal plus its `frozen` plumbing are gone. /// /// There is no `RefShape` here, no live-vs-frozen decision, and this can never /// make the reference live-selectable (the pin-only descent records no @@ -431,166 +571,94 @@ enum IndexVerdict { fn pin_dimension_name_indices( indices: Vec, ctx: &PerElementRefCtx<'_>, - frozen: bool, ) -> (Vec, bool) { let mut discharged = true; let indices = indices .into_iter() .enumerate() .map(|(i, idx)| { - let (name, loc) = match &idx { - // An `@N` POSITION selector is static -- `compiler::context` - // resolves it to a concrete element offset in scalar context -- so - // it neither needs a pin nor reads anything at the current step. - IndexExpr0::DimPosition(..) => return idx, - IndexExpr0::Expr(Expr0::Var(name, loc)) => { - (crate::common::canonicalize(name.as_str()).to_string(), *loc) - } - // A numeric selector, and arithmetic over numeric selectors, is - // static: nothing to pin, nothing to lag. See - // [`index_expr_selects_a_fixed_element`] -- the bare `Const` is just - // its base case. - IndexExpr0::Expr(e) if index_expr_selects_a_fixed_element(e) => return idx, - // Any other compound index expression selects the element at - // runtime. (A range, wildcard or star-range cannot be a table index - // at all -- `codegen::extract_table_info` needs a subscript selecting - // exactly ONE element and rejects anything wider as `BadTable` -- so - // treating them the same way costs nothing.) - _ => { - return verdict_into_index( - IndexVerdict::RuntimeRead, - idx, - frozen, - &mut discharged, - ); - } + // Only a bare identifier can name an element or a dimension, which is + // the only question this rule answers. Everything else -- an `@N` + // position, a numeric literal, a compound expression, a range or + // wildcard (which cannot be a table index at all: + // `codegen::extract_table_info` needs a subscript selecting exactly ONE + // element and rejects anything wider as `BadTable`) -- is `Keep`. + let IndexExpr0::Expr(Expr0::Var(name, loc)) = &idx else { + return idx; }; + let (name, loc) = (crate::common::canonicalize(name.as_str()).to_string(), *loc); let Some(dim) = ctx.from_dims.get(i) else { // Over-arity: no axis owns this index, so nothing names its owner // and there is nothing to resolve it against. - return verdict_into_index(IndexVerdict::Unspellable, idx, frozen, &mut discharged); + discharged = false; + return idx; }; - // A literal element selector of THIS axis qualifies to `dim·elem`; - // `qualify_axis_element` returns the name unchanged for anything the axis - // does not declare, testing exactly the axis's own `indexed_elements` -- - // the same membership `compiler::subscript`'s own resolution tests, and - // it is tried BEFORE the dimension-name reading for the same reason. - let axis_element = qualify_axis_element(&name, dim); + // An already-`dim·elem`-qualified index carries its own dimension, so + // it poses no element-vs-dimension-name question at all: it is static + // and already spelled the way this rule would spell it. let verdict = if ctx.dim_ctx.lookup(&name).is_some() { - IndexVerdict::Static - } else if axis_element != name { - IndexVerdict::Pinned(axis_element) - } else if ctx.target_elem_by_dim.contains_key(&name) { - // The index names one of the TARGET's ITERATED dimensions, so this - // axis reads whatever element this target element projects onto it. - // WHICH element that is -- the identity for a same-named axis, the - // positional correspondence for a mapped one -- is the shared row - // derivation's answer, not this rule's: it declines an unmapped or - // element-mapped pair, and a name-directed pin therefore accepts - // exactly the pairs the occurrence-driven one does. - let axis = crate::ltm_agg::AxisRead::Iterated { - dim: name.clone(), - source_dim: dim.name().to_string(), - }; - match per_element_row_for_target( - std::slice::from_ref(&axis), - ctx.target_elem_by_dim, - ctx.dim_ctx, - ) { - Some(row) => IndexVerdict::Pinned(qualify_axis_element(&row[0], dim)), - None => IndexVerdict::Unspellable, - } - } else if ctx.dim_ctx.is_dimension_name(&name) { - // A dimension name no target coordinate projects onto this axis. - IndexVerdict::Unspellable + IndexVerdict::Keep } else { - // Neither an element of this axis nor a dimension: a variable read - // selecting the element at runtime. - IndexVerdict::RuntimeRead - }; - let verdict = match verdict { - // A pin that changes nothing keeps its own node. - IndexVerdict::Pinned(part) if part == name => IndexVerdict::Static, - other => other, + match crate::dimensions::resolve_axis_index_name(&name, dim, |n| { + ctx.target_elem_by_dim.contains_key(n) + }) { + // A literal element selector of THIS axis, qualified with that + // axis (`old` -> `age·old`). + crate::dimensions::AxisIndexName::Element(elem) => { + IndexVerdict::Pinned(qualify_axis_element(&elem, dim)) + } + // The index names one of the TARGET's ITERATED dimensions, so this + // axis reads whatever element this target element projects onto it. + // WHICH element that is -- the identity for a same-named axis, the + // positional correspondence for a mapped one -- is the shared row + // derivation's answer, not this rule's: it declines an unmapped or + // element-mapped pair, and a name-directed pin therefore accepts + // exactly the pairs the occurrence-driven one does. + crate::dimensions::AxisIndexName::IteratedDim => { + let axis = crate::ltm_agg::AxisRead::Iterated { + dim: name.clone(), + source_dim: dim.name().to_string(), + }; + match per_element_row_for_target( + std::slice::from_ref(&axis), + ctx.target_elem_by_dim, + ctx.dim_ctx, + ) { + Some(row) => IndexVerdict::Pinned(qualify_axis_element(&row[0], dim)), + None => IndexVerdict::Unspellable, + } + } + crate::dimensions::AxisIndexName::Unresolved => { + if ctx.dim_ctx.is_dimension_name(&name) { + // A dimension name no target coordinate projects onto + // this axis. + IndexVerdict::Unspellable + } else { + // Neither an element of this axis nor a dimension: a + // variable read selecting the element at runtime, + // already frozen by the wrap. + IndexVerdict::Keep + } + } + } }; match verdict { + // A pin that changes nothing keeps its own node. + IndexVerdict::Pinned(part) if part == name => idx, IndexVerdict::Pinned(part) => { IndexExpr0::Expr(Expr0::Var(RawIdent::new_from_str(&part), loc)) } - other => verdict_into_index(other, idx, frozen, &mut discharged), + IndexVerdict::Keep => idx, + IndexVerdict::Unspellable => { + discharged = false; + idx + } } }) .collect(); (indices, discharged) } -/// Whether a subscript index expression selects a FIXED element -- built entirely -/// from numeric literals, so it reads nothing at any step and cannot vary with the -/// ceteris-paribus wrap's choice of what to hold live. -/// -/// This is the predicate behind `IndexVerdict::Static` for a compound index. Its -/// base case is the bare `Const` the rule always left alone; `LOOKUP(pop[Region, -/// 1 + 1], x)` is exactly as static as `LOOKUP(pop[Region, 2], x)` and the rule used -/// to decline the first only because its catch-all never looked inside. -/// -/// It is deliberately the NARROWEST sound predicate, not a general invariance test. -/// `compiler::invariance::exprs_are_invariant` is the engine's run-invariance -/// derivation, but it consumes LOWERED `Expr` plus an offset-classification -/// callback, neither of which exists in this position -- and its notion is *wider* -/// than this rule's obligation anyway: `Dt`, `StartTime` and `INIT(x)` of any -/// variable are all run-invariant, yet whether a table index may read a variable's -/// init buffer is an ATTRIBUTION question, and attribution is the wrap's vocabulary, -/// not this rule's. So this decides only the half it can decide alone. -/// -/// The match is exhaustive over `Expr0` on purpose -- no catch-all -- so a new -/// variant forces a decision here rather than silently inheriting `false`: -/// -/// - `Var` reads model state (or names an element, which the caller's own name arms -/// resolved before reaching this predicate); -/// - `Subscript` reads an array element; -/// - `App` is where a 0-arity builtin lands after `reify_0_arity_builtins`, and -/// `TIME` and `PI` are indistinguishable here without re-implementing the builtin -/// classification that `builtins`/`compiler::invariance` own. So the whole arm -/// stays `false`: a `PI`-indexed table declines CONSERVATIVELY (a loud skip, the -/// safe direction) rather than being sorted by a fourth copy of that knowledge. -fn index_expr_selects_a_fixed_element(expr: &Expr0) -> bool { - match expr { - Expr0::Const(..) => true, - Expr0::Op1(_, inner, _) => index_expr_selects_a_fixed_element(inner), - Expr0::Op2(_, lhs, rhs, _) => { - index_expr_selects_a_fixed_element(lhs) && index_expr_selects_a_fixed_element(rhs) - } - Expr0::If(cond, then_e, else_e, _) => { - index_expr_selects_a_fixed_element(cond) - && index_expr_selects_a_fixed_element(then_e) - && index_expr_selects_a_fixed_element(else_e) - } - Expr0::Var(..) | Expr0::Subscript(..) | Expr0::App(..) => false, - } -} - -/// Apply a non-rewriting [`IndexVerdict`]: keep the index as it stands, and clear -/// `discharged` when the verdict is loud in this freeze context (see -/// [`pin_dimension_name_indices`] -- `Unspellable` always, `RuntimeRead` only -/// outside a freeze). -fn verdict_into_index( - verdict: IndexVerdict, - idx: IndexExpr0, - frozen: bool, - discharged: &mut bool, -) -> IndexExpr0 { - match verdict { - IndexVerdict::Pinned(_) | IndexVerdict::Static => {} - IndexVerdict::Unspellable => *discharged = false, - IndexVerdict::RuntimeRead => { - if !frozen { - *discharged = false; - } - } - } - idx -} - /// Row-pin the live source's references inside a subtree the ceteris-paribus /// wrap declines to descend into -- a pre-existing `PREVIOUS`/`INIT` call, the /// GH #517 whole-frozen reducer, or a `LOOKUP` table argument. @@ -634,7 +702,6 @@ pub(super) fn pin_only_source_refs( occ: &OccurrenceLookup<'_>, path: &[u16], unlowerable: &mut bool, - frozen: bool, ) -> Expr0 { match expr { Expr0::Const(..) => expr, @@ -668,7 +735,6 @@ pub(super) fn pin_only_source_refs( occ, &idx_path, unlowerable, - frozen, )), IndexExpr0::Range(l, r, rloc) => IndexExpr0::Range( pin_only_source_refs( @@ -677,7 +743,6 @@ pub(super) fn pin_only_source_refs( occ, &super::child_path(&idx_path, 0), unlowerable, - frozen, ), pin_only_source_refs( r, @@ -685,7 +750,6 @@ pub(super) fn pin_only_source_refs( occ, &super::child_path(&idx_path, 1), unlowerable, - frozen, ), rloc, ), @@ -705,7 +769,7 @@ pub(super) fn pin_only_source_refs( // entirely, so nothing the IR classifies changes spelling. let (indices, discharged) = match node_occ { Some(_) => (indices, true), - None => pin_dimension_name_indices(indices, ctx, frozen), + None => pin_dimension_name_indices(indices, ctx), }; if !discharged { *unlowerable = true; @@ -734,7 +798,6 @@ pub(super) fn pin_only_source_refs( occ, &idx_path, unlowerable, - frozen, )), IndexExpr0::Range(l, r, rloc) => IndexExpr0::Range( pin_only_source_refs( @@ -743,7 +806,6 @@ pub(super) fn pin_only_source_refs( occ, &super::child_path(&idx_path, 0), unlowerable, - frozen, ), pin_only_source_refs( r, @@ -751,7 +813,6 @@ pub(super) fn pin_only_source_refs( occ, &super::child_path(&idx_path, 1), unlowerable, - frozen, ), rloc, ), @@ -767,14 +828,7 @@ pub(super) fn pin_only_source_refs( .into_iter() .enumerate() .map(|(i, a)| { - pin_only_source_refs( - a, - ctx, - occ, - &super::child_path(path, i), - unlowerable, - frozen, - ) + pin_only_source_refs(a, ctx, occ, &super::child_path(path, i), unlowerable) }) .collect(); Expr0::App(UntypedBuiltinFn(name, args), loc) @@ -787,7 +841,6 @@ pub(super) fn pin_only_source_refs( occ, &super::child_path(path, 0), unlowerable, - frozen, )), loc, ), @@ -799,7 +852,6 @@ pub(super) fn pin_only_source_refs( occ, &super::child_path(path, 0), unlowerable, - frozen, )), Box::new(pin_only_source_refs( *r, @@ -807,7 +859,6 @@ pub(super) fn pin_only_source_refs( occ, &super::child_path(path, 1), unlowerable, - frozen, )), loc, ), @@ -818,7 +869,6 @@ pub(super) fn pin_only_source_refs( occ, &super::child_path(path, 0), unlowerable, - frozen, )), Box::new(pin_only_source_refs( *t, @@ -826,7 +876,6 @@ pub(super) fn pin_only_source_refs( occ, &super::child_path(path, 1), unlowerable, - frozen, )), Box::new(pin_only_source_refs( *f, @@ -834,7 +883,6 @@ pub(super) fn pin_only_source_refs( occ, &super::child_path(path, 2), unlowerable, - frozen, )), loc, ), diff --git a/src/simlin-engine/src/ltm_augment_tests.rs b/src/simlin-engine/src/ltm_augment_tests.rs index 6c465ec65..3d4aad2f2 100644 --- a/src/simlin-engine/src/ltm_augment_tests.rs +++ b/src/simlin-engine/src/ltm_augment_tests.rs @@ -47,6 +47,40 @@ fn deps_set(idents: &[&str]) -> HashSet> { idents.iter().map(|s| Ident::new(s)).collect() } +/// Build a [`subscript_idents_at_element`] pin table literally: each dep with +/// the `(dimension, element)` pairs IT is pinned over, in its own declared +/// order. Spelling the dep's own dimensions out is the point -- production +/// derives them from the dep's declaration +/// (`crate::ltm_augment::dep_element_pins`), and the tests below pin what that +/// derivation is obliged to produce. +fn pin_table(pins: &[(&str, &[(&str, &str)])]) -> HashMap, DepElementPin> { + pin_table_with_completeness(pins, true) +} + +/// [`pin_table`] with the `complete` flag chosen explicitly: an INCOMPLETE pin +/// is one whose dep declares a dimension this target element does not project +/// onto, so it may substitute dimension-name indices but may not subscript a +/// bare reference. +fn pin_table_with_completeness( + pins: &[(&str, &[(&str, &str)])], + complete: bool, +) -> HashMap, DepElementPin> { + pins.iter() + .map(|(ident, axes)| { + ( + Ident::new(ident), + DepElementPin { + axes: axes + .iter() + .map(|(dim, elem)| ((*dim).to_string(), (*elem).to_string())) + .collect(), + complete, + }, + ) + }) + .collect() +} + /// Source-dimension element names for the per-shape partial-equation /// tests using a single `Region` dimension with elements `nyc` and /// `boston` (canonical lowercase form, in source-declared order). @@ -300,13 +334,15 @@ fn substitute_reducers_empty_reducers_passes_through_unparseable() { /// (~27k call sites, GH #654). #[test] fn test_subscript_idents_at_element_pins_dimension_name_indices() { - let idents = deps_set(&["reference_emissions", "pct_change"]); + let pins = pin_table(&[ + ("reference_emissions", &[("cop", "cop·oecd_us")]), + ("pct_change", &[("cop", "cop·oecd_us")]), + ]); // Parsed function names round-trip lowercased through print_eqn, so // the expected text spells `previous(...)`. let result = subscript_idents_at_element( "PREVIOUS(reference_emissions[cop]) * (PREVIOUS(pct_change[cop]) / c + 1)", - &idents, - "cop·oecd_us", + &pins, ) .unwrap(); assert_eq!( @@ -320,13 +356,11 @@ fn test_subscript_idents_at_element_pins_dimension_name_indices() { /// a different order) pins each index to the right element. #[test] fn test_subscript_idents_at_element_pins_by_dimension_name() { - let idents = deps_set(&["row_input", "matrix"]); - let result = subscript_idents_at_element( - "row_input[age] + matrix[age,region]", - &idents, - "region·nyc,age·adult", - ) - .unwrap(); + let pins = pin_table(&[ + ("row_input", &[("age", "age·adult")]), + ("matrix", &[("age", "age·adult"), ("region", "region·nyc")]), + ]); + let result = subscript_idents_at_element("row_input[age] + matrix[age,region]", &pins).unwrap(); assert_eq!( result, "row_input[age·adult] + matrix[age·adult, region·nyc]" @@ -334,17 +368,65 @@ fn test_subscript_idents_at_element_pins_by_dimension_name() { } /// Indices that are already element literals (not dimension names) are -/// left untouched, and unqualified pinned elements (no `dim·` part) -/// cannot pin dimension-name indices -- those keep the conservative form. +/// left untouched: only an index naming one of the DEP's own dimensions is +/// the "current element" form the pin substitutes. #[test] fn test_subscript_idents_at_element_leaves_literal_indices() { - let idents = deps_set(&["dep"]); + let pins = pin_table(&[("dep", &[("region", "region·la")])]); // `nyc` is an element literal, not the dimension name `region`. - let result = - subscript_idents_at_element("dep[nyc] + dep[region]", &idents, "region·la").unwrap(); + let result = subscript_idents_at_element("dep[nyc] + dep[region]", &pins).unwrap(); assert_eq!(result, "dep[nyc] + dep[region·la]"); } +/// A BARE reference to a pinned dep is subscripted over the dep's OWN +/// dimensions -- the arm GH #974 is about. +/// +/// Rows, derived from the ways a dep's declared dimensions can relate to the +/// target's (here `growth[Region,Age]`), which is what the projection behind +/// the table has to get right: +/// +/// | dep's dims | pre-#974 full-tuple pin | correct | +/// |---|---|---| +/// | same, same order (`same[Region,Age]`) | `same[region·nyc, age·old]` | identical -- the case that always worked | +/// | same, REORDERED (`flip[Age,Region]`) | `flip[region·nyc, age·old]` -- compiles, reads the WRONG element | `flip[age·old, region·nyc]` | +/// | a strict SUBSET (`w[Age]`) | `w[region·nyc, age·old]` -- arity 2 over a 1-D variable, fragment fails to compile | `w[age·old]` | +/// +/// The reordered row is the one that had no diagnostic at all: both indices +/// resolve numerically, so the partial compiled and silently read +/// `flip[young, boston]`. +#[test] +fn test_subscript_idents_at_element_pins_bare_refs_over_the_deps_own_dims() { + let pins = pin_table(&[ + ("same", &[("region", "region·nyc"), ("age", "age·old")]), + ("flip", &[("age", "age·old"), ("region", "region·nyc")]), + ("w", &[("age", "age·old")]), + ]); + let result = subscript_idents_at_element("same * flip * w", &pins).unwrap(); + assert_eq!( + result, + "same[region·nyc, age·old] * flip[age·old, region·nyc] * w[age·old]" + ); +} + +/// An INCOMPLETE pin -- a dep declaring a dimension this target element does +/// not project onto (`pop[Region,Age]` read from a `growth[Region]` body) -- +/// substitutes the dimension-name index of a SUBSCRIPTED reference but leaves a +/// BARE reference alone. +/// +/// Both halves are load-bearing and pull opposite ways. Substituting the +/// `Region` index is the GH #654 helper-aux fix: a surviving dimension name in +/// a scalar fragment forces a synthesized capture helper that cannot compile. +/// Leaving the bare reference alone is GH #974: there is no correct full-arity +/// subscript to give it, and the pre-fix full-target-tuple pin spelled the +/// arity-1 `pop[region·nyc]` over a 2-D variable. Bare stays bare, which fails +/// to compile LOUDLY rather than reading a wrong element. +#[test] +fn test_subscript_idents_at_element_incomplete_pin_leaves_bare_refs_alone() { + let pins = pin_table_with_completeness(&[("pop", &[("region", "region·nyc")])], false); + let result = subscript_idents_at_element("pop[Region, idx] + pop", &pins).unwrap(); + assert_eq!(result, "pop[region·nyc, idx] + pop"); +} + // -- dimension_element_names tests -- #[test] @@ -1865,6 +1947,101 @@ fn test_partial_equation_lookup_table_arg_not_wrapped() { } } +/// GH #984 on the GENERAL (non-`PerElement`) path: the table argument's HEAD +/// stays bare while its subscript INDEX reads are frozen. +/// +/// The `PerElement` callers thread a row-pinning context and reach the table +/// argument through the pin-only descent; every OTHER caller passes `pin: None`, +/// and for them the arm returned the whole argument completely untouched -- no +/// descent, no freeze, nothing. So `y = LOOKUP(g[idx], x)` made every partial of +/// `y` vary with `idx`'s current-step movement +/// (`compiler::codegen::extract_table_info` builds the element offset out of the +/// live index `Expr`s), attributing it to whichever source the partial isolates. +/// There is no diagnostic behind that: the fragment compiles and the score is +/// just wrong. +/// +/// The rows are derived from what a table argument can be, since that is what +/// the arm dispatches on: a bare `Var` (no indices at all), a subscript with a +/// STATIC selector (nothing to freeze), and a subscript with a runtime read (the +/// defect). Each is checked for all three `LOOKUP` spellings, because the arm +/// matches on the name and a fix that missed one would leave the extrapolating +/// variants wrong. +/// +/// **The dep set is DERIVED, not declared**, and that is what makes this an +/// oracle. Production builds the wrap's `other_deps` from +/// `variable::identifier_set`, whose `BuiltinContents::LookupTable` arm records +/// the table's ident and never walks the table expression -- so `idx` is not a +/// dependency of `y = LOOKUP(g[idx], x)` and the wrap's ordinary +/// `other_deps.contains(..)` freeze can never fire for it. Calling +/// `identifier_set` here rather than hand-writing `deps_set(&[.., "idx", ..])` +/// is the difference between testing the fix and testing an input production +/// cannot produce: the first version of this fix passed a hand-written set and +/// shipped without firing at all. +#[test] +fn test_partial_equation_lookup_table_index_is_frozen() { + // `region`/`nyc` give the static row a real element to be static about. + let dims = crate::dimensions::DimensionsContext::from( + [datamodel::Dimension::named( + "region".to_string(), + vec!["nyc".to_string(), "boston".to_string()], + )] + .as_slice(), + ); + let source = Ident::::new("food"); + + for (label, table_arg, expected_table_arg) in [ + ("a bare table -- nothing to index", "g", "g"), + ( + "a STATIC element selector -- nothing to freeze", + "g[nyc]", + "g[nyc]", + ), + ( + "a runtime index read -- frozen, head still bare", + "g[idx]", + "g[PREVIOUS(idx)]", + ), + ] { + for func in ["lookup", "lookup_forward", "lookup_backward"] { + let equation = format!("{func}({table_arg}, food / subsistence)"); + // Production's dep set for this exact equation, through the exact + // function production calls. + let deps = + crate::variable::identifier_set(&crate::variable::scalar_ast(&equation), &[], None); + let partial = build_partial_equation_shaped( + &equation, + &deps, + &source, + &RefShape::Bare, + &[], + None, + Some(&dims), + ) + .unwrap_or_else(|e| panic!("{label} ({func}): the partial must be emitted: {e:?}")); + assert_eq!( + partial, + format!("{func}({expected_table_arg}, food / PREVIOUS(subsistence))"), + "{label} ({func})" + ); + } + } + + // Non-vacuity for the third row: the derived set really does omit the index + // ident, so the freeze there cannot be coming from the ordinary other-dep + // path. This is the fact the whole test hangs on, so it is asserted rather + // than described. + let deps = crate::variable::identifier_set( + &crate::variable::scalar_ast("lookup(g[idx], food / subsistence)"), + &[], + None, + ); + assert!( + !deps.contains(&Ident::::new("idx")), + "production's dep extractor must NOT report a table-only index as a \ + dependency; got {deps:?}" + ); +} + /// The WITH LOOKUP lowering references the variable's own table as /// `lookup(self_var, input)`. The self-reference is the table holder, so /// it stays bare; the (live) input is held live and other deps wrapped. @@ -2195,22 +2372,21 @@ fn build_partial_equation_shaped_no_deps_to_wrap_is_ok() { /// `subscript_idents_at_element` shares the same loud-failure contract: /// an unparseable (already-partial) equation returns `Err`, while an -/// empty `idents` set is a legitimate no-op that returns the text -/// unchanged. +/// empty pin table is a legitimate no-op that returns the text unchanged. #[test] fn subscript_idents_at_element_parse_error_is_err() { - let idents = deps_set(&["dep"]); + let pins = pin_table(&[("dep", &[("region", "region·nyc")])]); let bad = "dep * * other"; - let result = subscript_idents_at_element(bad, &idents, "region·nyc"); + let result = subscript_idents_at_element(bad, &pins); match result { Err(err) => assert_eq!(err.equation_text, bad), Ok(out) => panic!("a parse failure must be a loud Err; got Ok({out:?})"), } - // Empty idents: nothing to pin, returns the text verbatim (even text + // Empty pin table: nothing to pin, returns the text verbatim (even text // that would not parse is irrelevant -- the function short-circuits). - let noop = subscript_idents_at_element(bad, &deps_set(&[]), "region·nyc") - .expect("empty idents is a no-op, not a parse attempt"); + let noop = subscript_idents_at_element(bad, &pin_table(&[])) + .expect("an empty pin table is a no-op, not a parse attempt"); assert_eq!(noop, bad); } diff --git a/src/simlin-engine/src/variable.rs b/src/simlin-engine/src/variable.rs index 8795f9b60..cb551091c 100644 --- a/src/simlin-engine/src/variable.rs +++ b/src/simlin-engine/src/variable.rs @@ -1215,7 +1215,7 @@ pub fn previous_referenced_idents(ast: &Ast) -> BTreeSet { /// /// Panics on parse or lowering errors -- intended for test use only. #[cfg(test)] -fn scalar_ast(eqn: &str) -> Ast { +pub(crate) fn scalar_ast(eqn: &str) -> Ast { use crate::ast::lower_ast; let (ast, err) = parse_equation( From b2977e89561babd716ff8e6330de67ab35371db9 Mon Sep 17 00:00:00 2001 From: Bobby Powers Date: Tue, 28 Jul 2026 06:57:20 -0700 Subject: [PATCH 08/17] engine: fix two ceteris-paribus defects in the wrap's index handling Two independent defects in how the LTM ceteris-paribus wrap treats a subscript index, both silent wrong numbers. GH #977's collapse was implemented alongside these and withdrawn; see the PR body and the issue. -- GH #975: seed a frozen subscript index with the un-lagged index -- The ceteris-paribus wrap emitted a UNARY PREVIOUS around a frozen dynamic subscript index, which the parser desugars to PREVIOUS(idx, 0). The XMILE spec (section 3.5.6) makes that second parameter the value returned in the first DT, and zero is a sound default for a value but is out of range for every 1-based subscript -- so at t=0 the synthesized capture helper read past the end of its source and computed NaN, and the outer PREVIOUS served that NaN as the score's first live step. A NaN the engine manufactures is indistinguishable on a graph from the modeller's own division by zero, which is what makes this a defect rather than a wart. An index-position freeze now names its own un-lagged operand as its first-DT initial value: at the first DT "the index one step ago" IS the current index, and since it is the index the target's own equation reads at that step, the freeze is never out of range where the model's own read is not. Value positions keep the unary spelling, because 0 is a valid VALUE -- not because it is unobservable; a value-position freeze inside a capture helper is read at step 1 by exactly this route. Only the two walkers that descend into subscript indices need it; the other two document that they never do. Note the issue's own diagnosis was wrong on two counts, both checked rather than assumed: the capture helper already participates in the initials phase, and PREVIOUS does not read a prev_values snapshot seeded from initials -- it returns its declared initial value, exactly as the spec says. The head-lag pin gets a strictly stronger discriminator and its own fixture. The NaN used to be the only evidence the lag existed at all; now the guard checks the helper is finite at t=0 AND reads the LAGGED element at every step, against the model's own simulated series. That needs two Age rows that actually differ, which the shared fixture does not have. -- GH #986's third consumer: resolve against the axis it indexes -- The ceteris-paribus wrap decided whether a bare-identifier subscript index was a static element selector or a runtime read with two PROJECT-WIDE predicates -- dimension_uniquely_containing_element and is_element_of_any_dimension, which range over every dimension in the project. The compiler does not: compiler::subscript::normalize_subscripts3 resolves such an index against the indexed variable's OWN axis, which is what dimensions::resolve_axis_index_name implements and what GH #986 unified two other LTM consumers onto. This was the third consumer and it was missed. The disagreement is a wrong number, not a missed optimization. A model variable whose canonical name happens to be an element of an UNRELATED dimension -- a Scenario = [base, high, low] beside a variable named base -- is a runtime read to the simulation and a static selector to the wrap, so the partial leaves it LIVE and moves with it: a link score reporting real influence for an edge with no causal dependence at all. The qualification step then rewrites the index to an otherdim-qualified form, naming an element of a dimension the indexed variable is not declared over, which still compiles and reads a different slot than the PREVIOUS(target) anchor did. Declaring a dimension that NO equation references is enough to change a score, with no diagnostic in either mode. THE UNIFICATION IS PARTIAL AND THIS DOES NOT PRETEND OTHERWISE. Resolving an index needs the declared Dimension at that index's position, and the wrap takes it from IteratedDimCtx::dep_dims -- the dep-to-declared-dimensions table the db layer already threads for the GH #526 other-dep correspondence check, so this adds a consumer rather than a second table built by a second route. But dep_dims is a table of the TARGET'S ARRAY DEPENDENCIES, not of declared dimensions, and reading "absent from dep_dims" as "no axis exists" is the convenient reading rather than the sound one. Two production paths therefore still take the project-wide fallbacks and still reproduce the defect, and both are named at their call sites rather than left to be rediscovered: every LOOKUP table index, because a graphical-function holder is by construction absent from dep_dims (classify_dependencies records it in referenced_tables and keeps it off the dependency graph, GH #606), so the axis lookup can never resolve there -- it is passed a literal None, because a call that could only ever return None reads as coverage for a path that is not fixed; and generate_per_element_link_equation, which threads dep_dims: None on a comment that was true when other_dep_verdict was that table's only consumer. GH #984 stays open on the first, with its reproduction on the issue. Closing either needs the wrap to get its dimension truth from where the compiler gets it, which is a decision about a function family that deliberately carries none, and so is its own change. What IS fixed is pinned on the emitted text AND on the simulated series, over the arrayed / scalar / A2A / stock-flow emitters, with a control proving the fix does not degenerate into freezing every bare index. The test says in its own rustdoc which path it covers and which it does not, because an earlier revision of it claimed a LOOKUP fixture it never had -- and that false claim is how the LOOKUP gap shipped unnoticed in the first place. One older defect at this site is disclosed rather than folded in: an index frozen inside an already-frozen head is DOUBLE-lagged, reading the head at t-1 indexed at t-2 where the anchor indexed at t-1, so such a partial is not fully ceteris-paribus even once every name resolves. That behaviour is pinned but was never adjudicated -- the pin that freezes it was written to catch a blanket skip of the whole index pass and uses the lag only as its discriminator, while this function's own GH #759 comment calls reading an index two steps back "semantically wrong for a genuinely-dynamic index". It is deferred because it is a semantics question that interacts with GH #975's head-lag pin, not because the current answer is settled. --- src/simlin-engine/CLAUDE.md | 6 +- .../per_element_dynamic_index.txt | 6 +- src/simlin-engine/src/db/ltm_char_tests.rs | 125 ++++++-- src/simlin-engine/src/db/ltm_tests.rs | 268 +++++++++++++++++ src/simlin-engine/src/float.rs | 8 +- src/simlin-engine/src/ltm_augment.rs | 271 +++++++++++++++--- src/simlin-engine/src/ltm_augment_freeze.rs | 66 +++++ .../src/ltm_augment_pin_tests.rs | 14 +- src/simlin-engine/src/ltm_augment_tests.rs | 34 ++- 9 files changed, 713 insertions(+), 85 deletions(-) create mode 100644 src/simlin-engine/src/ltm_augment_freeze.rs diff --git a/src/simlin-engine/CLAUDE.md b/src/simlin-engine/CLAUDE.md index 28d153135..ede89369c 100644 --- a/src/simlin-engine/CLAUDE.md +++ b/src/simlin-engine/CLAUDE.md @@ -91,7 +91,7 @@ Diagnostics (`collect_all_diagnostics`) run on the user's `SourceProject`, so sy - **`src/errors.rs`** - Human-readable error formatting: `FormattedError`/`FormattedErrors`, `FormattedErrorKind`, `UnitErrorKind`. `format_diagnostic()` converts a salsa `Diagnostic` to `FormattedError`; `format_diagnostic_with_datamodel()` adds source snippets from the datamodel. `collect_formatted_errors()` is the bulk entry point that aggregates all diagnostics into a `FormattedErrors` value. `FormattedError.message` is TERMINAL-formatted (source snippet + `~~~` underline + model/variable summary line); the separate `details` field carries just the bare reason for unit errors (e.g. "computed units 'x' don't match specified units") so GUI consumers (via libsimlin's `SimlinErrorDetail.details`) can render it without the snippet noise. `FormattedError.severity` carries the originating `db::DiagnosticSeverity`, and the summary line's severity word is rendered FROM it (`severity_word`) rather than hardcoded (GH #919): a `Warning` -- the conveyor/queue LTM-degraded advisory, the conveyor spec advisories, a unit *consistency*/*inference* mismatch -- reads "warning in model ..." / "units warning in model ...", while an `Error` (an equation error, a unit *definition* syntax error) keeps "error in model ...". The word therefore tracks the diagnostic's severity, not the arm it came from, so no consumer that renders `message` verbatim can present an advisory as a compilation failure. `FormattedErrors::push` is the SOLE place `has_model_errors`/`has_variable_errors` are set, and it counts `Error` severity only -- those flags gate failure-shaped decisions (the CLI's redundant-`NotSimulatable` suppression), so a warning must never flip them. `format_simulation_error` has no `Diagnostic` behind it and is unconditionally `Error`. Canonical implementation shared by both `simlin-mcp` and `libsimlin` (which re-exports from here). - **`src/datamodel.rs`** - Core structures: `Project`, `Model`, `Variable`, `Equation` (including `Arrayed` variant with `default_equation` for EXCEPT semantics and `has_except_default` bool flag), `Dimension` (with `mappings: Vec` replacing the old `maps_to` field, and `parent: Option` for indexed subdimension relationships), `DimensionMapping`, `DataSource`/`DataSourceKind`, `UnitMap`, `MacroSpec` (a macro-marked `Model`'s calling convention: `parameters`/`primary_output`/`additional_outputs`; `Model.macro_spec` is `None` for every ordinary model; `salsa::Update` so it rides directly on the `SourceModel` input). `Model::new_macro(name, params, additional_outputs, body_variables)` is the shared port-synthesis + `MacroSpec`-construction step used by *both* the MDL converter and the XMILE reader: it sets `can_be_module_input` on each formal-parameter body variable (synthesizing a `Flow`/`Aux` placeholder port when absent) so `collect_module_idents` treats the macro as an ordinary sub-model. View element types (`Aux`, `Stock`, `Flow`, `Alias`, `Cloud`) carry an optional `ViewElementCompat` with original Vensim sketch dimensions/bits for MDL roundtrip fidelity. `StockFlow` has an optional `font` string for the Vensim default font spec. - **`src/variable.rs`** - Variable variants (`Stock`, `Flow`, `Aux`, `Module`), `ModuleInput`, `Table` (graphical functions). `classify_dependencies()` is the primary API for extracting dependency categories from an AST in a single walk, returning a `DepClassification` with five sets: `all` (every referenced ident), `init_referenced`, `previous_referenced`, `previous_only` (idents only inside PREVIOUS), and `init_only` (idents only inside INIT/PREVIOUS). `parse_var_with_module_context` accepts a `module_idents` set so `PREVIOUS(module_var)` rewrites through a scalar helper aux instead of `LoadPrev`. Per-element graphical functions: `build_tables` materializes one `Table` per element of an arrayed GF and `reorder_arrayed_element_tables` places each at the element's *flat row-major declared-dimension index*, NOT its `Equation::Arrayed` `elems` Vec position -- because the runtime selects a per-element table by the row-major dimension offset (`vm.rs` `Lookup`/`LookupArray`: `graphical_functions[base_gf + element_offset]`). Elements lacking a GF get an empty placeholder so the index stays aligned. The salsa dependency-table path mirrors this in `db.rs::extract_tables_from_source_var`. -- **`src/dimensions.rs`** - `DimensionsContext` for dimension matching, subdimension detection, and element-level mappings. Supports indexed subdimensions via `parent` field (child maps to first N elements of parent). `has_mapping_to()` checks for element-level dimension mappings between two dimensions. `resolve_axis_index_name(name, axis_dim, target_iterates)` is the engine's SINGLE element-vs-dimension-name precedence for a bare-identifier subscript index -- the axis's own declared elements first, then a dimension the enclosing equation iterates -- matching `compiler::subscript::normalize_subscripts3`; both LTM callers (`ltm_agg::classify_axis_access` and the pin rule) read it, so a colliding element name cannot be resolved two ways (GH #986). `mapped_element_correspondence(iterated_dim, source_dim)` (GH #527) is the reusable element-level correspondence for a mapped dimension pair -- per iterated element, the source element the executed simulation reads (both declaration directions, single-hop only, POSITIONAL mappings only; an explicit element map returns `None` because the executed A2A lowering resolves positionally and ignores it -- GH #753 and the tracked positional-vs-element-map execution inconsistency (GH #756) are the gate for re-enabling; `None` ⇒ callers keep their conservative broadcast, a superset of the true edges). It is what keeps the LTM element graph (`expand_same_element`'s mapped diagonal) and link-score dimensions (`link_score_dimensions` -- whose mapped arm is additionally gated on the edge having a `Bare`-classified site via `model_edge_shapes`, so the arrayed retarget fires exactly when the element graph expands the diagonal rather than the element-mapped `DynamicIndex` cross-product) in lockstep with the classifier (`classify_iterated_dim_shape`, whose mapped arm gates on this same correspondence in BOTH declaration directions since GH #757 -- via `ltm_agg::classify_axis_access` / `iterated_axis_slot_elements` -- so a reverse-declared positional pair classifies `Bare` and gets the diagonal) and -- for positional mappings -- the simulation; the agg machinery's mapped sliced reducers (GH #534) consume it through `ltm_agg::iterated_axis_slot_elements` (the per-source-element preimage inversion), so the same gate governs hoisting and the emitters' slot remap. `SubdimensionRelation` caches parent-child offset mappings for both named (element containment) and indexed (declared parent) dimensions +- **`src/dimensions.rs`** - `DimensionsContext` for dimension matching, subdimension detection, and element-level mappings. Supports indexed subdimensions via `parent` field (child maps to first N elements of parent). `has_mapping_to()` checks for element-level dimension mappings between two dimensions. `resolve_axis_index_name(name, axis_dim, target_iterates)` is the engine's SINGLE element-vs-dimension-name precedence for a bare-identifier subscript index -- the axis's own declared elements first, then a dimension the enclosing equation iterates -- matching `compiler::subscript::normalize_subscripts3`; three LTM consumers read it -- `ltm_agg::classify_axis_access`, the pin rule (`ltm_augment_post_transform::pin_dimension_name_indices`), and SOME of the ceteris-paribus wrap's subscript-index guard (`ltm_augment::index_axis_verdict`, GH #986). The wrap was the last to be unified and the unification is PARTIAL, which is the important part: it had two PROJECT-WIDE predicates (`dimension_uniquely_containing_element` and `is_element_of_any_dimension`, ranging over every dimension in the project) where the compiler ranges over the indexed variable's own axis, so a model variable whose canonical name happens to be an element of an UNRELATED dimension is a runtime read to the simulation and a static element selector to the wrap -- the partial leaves it LIVE and moves with it (a link score reporting real influence for an edge with no causal dependence at all), and the qualification step rewrites the index to `otherdim·name`, an element of a dimension the indexed variable is not declared over, which still compiles and reads a different slot than the `PREVIOUS(target)` anchor did. Declaring a dimension that NO equation references is enough to change a score, with no diagnostic in either mode. The wrap is now THREE call-site families with TWO answers: the arrayed / scalar / A2A / stock-flow emitters resolve against the axis (pinned by `db::ltm_tests::a_colliding_index_name_is_resolved_against_the_axis_it_indexes`, on the emitted text AND the simulated series), while **`generate_per_element_link_equation` and every `LOOKUP` table index still take the project-wide fallbacks and still reproduce the defect** -- the first because it threads `dep_dims: None`, the second because a graphical-function holder is by construction absent from `dep_dims` (`variable::classify_dependencies`' `LookupTable` arm records it in `referenced_tables` and keeps it off the dependency graph, GH #606), so `axis_dim_at` can never resolve its axis. Both gaps are named at their call sites; GH #984 stays open on the second. The axis comes from `IteratedDimCtx::dep_dims`, the dep -> declared-dimensions table already threaded for the GH #526 other-dep correspondence check -- reusing it avoids a second table built by a second route, but it is a table of the TARGET's array dependencies rather than of declared dimensions, and that mismatch is exactly what bounds the fix. A separate, older defect at the same site is also open: an index frozen inside an ALREADY-frozen head is DOUBLE-lagged (the partial reads the head at `t-1` indexed at `t-2`, where the anchor indexed at `t-1`), so such a partial is not fully ceteris-paribus even where every name resolves correctly -- and the wrap's own GH #759 comment calls reading an index two steps back "semantically wrong for a genuinely-dynamic index". It is deferred because it is a semantics question that interacts with GH #975's own head-lag pin, NOT because the current behaviour has been adjudicated: `db::ltm_char_tests::per_element_dynamic_index_scores_preserve_head_lag` freezes the lag but was written to catch a blanket skip of the whole index pass, and uses the lag only as its discriminator. `mapped_element_correspondence(iterated_dim, source_dim)` (GH #527) is the reusable element-level correspondence for a mapped dimension pair -- per iterated element, the source element the executed simulation reads (both declaration directions, single-hop only, POSITIONAL mappings only; an explicit element map returns `None` because the executed A2A lowering resolves positionally and ignores it -- GH #753 and the tracked positional-vs-element-map execution inconsistency (GH #756) are the gate for re-enabling; `None` ⇒ callers keep their conservative broadcast, a superset of the true edges). It is what keeps the LTM element graph (`expand_same_element`'s mapped diagonal) and link-score dimensions (`link_score_dimensions` -- whose mapped arm is additionally gated on the edge having a `Bare`-classified site via `model_edge_shapes`, so the arrayed retarget fires exactly when the element graph expands the diagonal rather than the element-mapped `DynamicIndex` cross-product) in lockstep with the classifier (`classify_iterated_dim_shape`, whose mapped arm gates on this same correspondence in BOTH declaration directions since GH #757 -- via `ltm_agg::classify_axis_access` / `iterated_axis_slot_elements` -- so a reverse-declared positional pair classifies `Bare` and gets the diagonal) and -- for positional mappings -- the simulation; the agg machinery's mapped sliced reducers (GH #534) consume it through `ltm_agg::iterated_axis_slot_elements` (the per-source-element preimage inversion), so the same gate governs hoisting and the emitters' slot remap. `SubdimensionRelation` caches parent-child offset mappings for both named (element containment) and indexed (declared parent) dimensions - **`src/model.rs`** - Model compilation stages (`ModelStage0` -> `ModelStage1` -> `ModuleStage2`). It performs NO dependency analysis of its own: `ModelStage1::set_dependencies` reads the production dependency graph (`db::dep_graph::model_dependency_graph`) once per module instantiation and copies its runlists into `ModuleStage2`. Until GH #568 it ran a second, independent walk here -- its own transitive closure (`all_deps`), its own cross-model output resolution (`module_output_deps`), its own `topo_sort` runlists and its own `CircularDependency` -- and that second gate genuinely disagreed with production's, rejecting the element-acyclic recurrence SCCs `resolve_recurrence_sccs` resolves. There is one gate now, pinned by `project.rs`'s `the_circular_dependency_gate_is_the_production_one` in both the resolves and the rejects direction. `collect_module_idents` pre-scans datamodel variables to identify which names will expand to modules (preventing incorrect `LoadPrev` compilation). Unit checking uses salsa tracked functions in `db.rs`. The two `#[cfg(test)]` `ModelStage0` constructors are the salsa-FREE twin of `db::stages::model_stage0`: `new_in_project(project_models, x_model, ..)` builds a stage from a `datamodel::Model` with no database, resolving module-function calls against the whole project's `MacroRegistry` (which is what makes it a faithful oracle for a model that CALLS a macro defined in a sibling model), and `new(x_model, ..)` is the single-model wrapper the many one-model fixtures use. They derive the module-ident set, the macro registry and the duplicate-ident errors along completely different routes than the query, which is exactly why `db::stages_tests` can use them as an oracle. The former salsa-cached `ModelStage0::new_cached` -- a test-only third copy of the query's construction -- was deleted; its coverage now points at the live query. `enumerate_modules_inner` records a model's instantiation BEFORE descending into it, so that a module cycle terminates: the "have I seen this model" test is the recursion guard, so a model still being walked has to count as seen. (A cycle through `main` already terminated, because `enumerate_modules` records `main` up front; one below `main` did not.) This matches its salsa twin `db::assemble::enumerate_module_instances_inner`. Recording early cannot LOSE an instantiation because the insert is unconditional at every module site -- only the recursion is guarded; the set-of-input-sets value is a separate fact, and it is why the ORDER instantiations arrive in is unobservable. - **`src/project.rs`** - `Project` struct aggregating models. `from_salsa(datamodel, db, source_project, cb)` builds a Project from a pre-synced salsa database: it READS `db::stages::model_stage1` for every project model and clones each memo (the clone is mandatory, not incidental -- `model_deps.take()`, `set_dependencies` and the `model_cb` all mutate it, while a `returns(ref)` memo is shared with every other reader). Each stage is kept paired with the `SourceModel` handle it came from, because `set_dependencies` reads the production dependency graph and that query is keyed on the handle, not on the model's name. It used to build its own whole-project `ModelStage0` map and lower it inline, a second salsa-native copy that had silently drifted from `db::stages` on three fields. Its model order is deterministic because the `topo_sort` seed is sorted first -- `topo_sort` breaks ties by visit order and the seed came from a `HashMap`'s keys (the `Project`-path twin of the GH #595 fix in `db::dep_graph`); the Initials runlists inherit the dependency graph's own determinism since GH #568 unified the two gates. `compiler::Module::new` is the sole reader of `ModelStage1::instantiations`; its callers are `#[cfg(test)] TestProject::build_module` and three direct calls in the `compiler`'s dimension tests, all `#[cfg(test)]`, while production `src/db/assemble.rs` never builds a `compiler::Module` at all. It REFUSES a model whose dependency graph carries a resolved recurrence SCC (`ModelStage1::set_dependencies` records a `NotSimulatable` for one): GH #568 unified the cycle GATE but not the EMITTER, and the per-element interleave `db::assemble::combine_scc_fragment` performs has no monolithic equivalent, so emitting the members whole would read a co-member's element before assigning it -- a silent wrong answer where the second gate used to give an accidental refusal. A model that can REACH a module cycle is gated out of dependency resolution first, exactly as the production entry points gate it (`db::diagnostic::module_cycle_diagnostic`, GH #806): the dependency graph's recurrence-SCC refinement descends into the recursive `model_module_map`, and salsa turns that into an unrecoverable dependency-graph panic -- a process abort under `panic = abort`, reachable from the public `From`. Such a model records the cycle as its error and keeps empty runlists. `from_datamodel(datamodel)` is a convenience wrapper that creates a local DB and syncs. **Neither is reachable from production**: inside the crate every caller is a test, and the only in-repo user of the public `From` impl is the engine's own `tests/integration/simulate_ltm.rs`, feeding the `ltm_finding::discover_loops` convenience wrapper (the shipped analysis path, libsimlin's `simlin_analyze_discover_loops`, goes through `discover_loops_with_graph` and builds no `Project`). Treat it as a monolith kept honest as a test oracle, not as live code. Production compiles via `db::compile_project_incremental` with `ltm_enabled`/`ltm_discovery_mode` on `SourceProject`. - **`src/results.rs`** - `Results` (variable offsets + timeseries data), `Specs` (time/integration config) @@ -182,7 +182,7 @@ The unit subsystem is partial-result throughout: a single bad declaration or one - `tests.rs` - integration-style tests spanning all submodules; preserved as a single file to avoid splitting shared test fixtures. - **`src/ltm_finding.rs`** - Strongest-path loop discovery algorithm (Eberlein & Schoenberg 2020). Post-processes simulation results containing link score synthetic variables to find the most important loops at each timestep. `discover_loops_with_graph(results, causal_graph, stocks, ltm_vars, dims, expansion, sub_model_output_ports, budget)` is the primary entry point: `ltm_vars` and `dims` enable A2A link score expansion (per-element edges from `parse_link_offsets`); when empty, all link scores are treated as scalar. A Bare A2A link score is dimensioned over the TARGET's dims, so `expand_a2a_link_offsets` subscripts the TARGET node per element but PROJECTS the SOURCE node onto `from`'s OWN declared dims -- bare for a scalar feeder (`scale → growth`, GH #790), the same-element diagonal for an equal-dim feeder, the broadcast/partial-collapse form for a lower-dim feeder, and the positionally-mapped diagonal for a `State→Region` pair (GH #527) -- by reusing the element graph's own `db::expand_same_element` rule, so the discovery search graph's node names match `model_element_causal_edges` node-for-node (GH #754; before this, subscripting BOTH endpoints with the score's full tuple minted phantom from-nodes like `scale[a]`/`boost[r,a]`/`x[s]` that named no real node, so every loop through such a feeder dangled and was silently undiscoverable). The from-var declared dims + dimension-mapping context ride in a `LinkExpansionContext` (`declared_dims` + `dim_ctx`) that `analyze_model` builds via the public `analysis::build_link_expansion_context` (the SAME `variable_dimensions` / `project_dimensions_context` queries the element graph reads); the db-less `discover_loops(&Results, &Project)` convenience path passes `LinkExpansionContext::default()` (no A2A expansion runs there). Element-mapped (non-positional) pairs never reach the projection: `db::ltm::link_score_dimensions` declines to retarget them (no Bare A2A score; the GH #758 loud skip fires instead) under the GH #756 positional-only gate. `SearchGraph` provides DFS-based strongest-path traversal from stock nodes. The post-simulation score recompute (path → `FoundLoop`) applies the **per-exit-port pathway selection** (`recompute_module_input_edge_series`, GH #698): a loop edge `x → m` into a module is recomputed by max-abs-selecting over the sub-model's `m·$⁚ltm⁚path⁚{entry}⁚{idx}` pathway scores that end at the exit port the loop reads (recovered from the next link `m → y`), instead of reading the module *composite* (which max-abs-selects across ALL ports and so can pick a wrong-signed port for a multi-output module -- flipping the loop polarity vs exhaustive). The pathway indices come from the module's sub-graph via the same `enumerate_pathways_to_outputs_with_truncation` over the same sorted output-port set the sub-model emitted against, so they match index-for-index (the discovery-mode equivalent of exhaustive's `compute_module_link_overrides`). That port set is NOT re-derived parent-scoped (which would shift indices when another project model reads an extra output port -- GH #698 / PR #705 r3353097150): `discover_loops_with_graph` takes a `SubModelOutputPorts` map (sub-model canonical name -> emitted sorted port set) that `analyze_model` builds from the SAME emission decision via `analysis::build_sub_model_output_ports` -> `db::ltm::sub_model_output_ports` (identity-by-construction); the db-less `discover_loops(&Results, &Project)` convenience path reconstructs the same project-wide-union + stdlib-`output` semantics in `project_sub_model_output_ports`. Falls back to the base composite offset for single-output / pathless / indeterminate-port / sub-model-absent-from-map edges. Because the discovery graph is element-level, the recompute `strip_subscript`s `link.from`/`link.to`/`next.from`/`next.to` before its name matches (arrayed loop nodes carry `[elem]`; mirrors the exhaustive twin's stripping -- PR #705 r3353758167); this is currently latent parity defense, since an arrayed loop through a multi-output module is not yet discoverable end-to-end (a scalar module output feeding an arrayed reader scores a constant-0 link, dropping the loop -- GH #716). Returns a `DiscoveryResult` whose `FoundLoop`s carry per-timestep link/loop/pathway scores, ranked **competitive-first** (`rank_and_filter`): loops sharing their cycle partition with at least one other discovered loop come first, ordered by mean partition-relative importance; loops trivially ALONE in their partition (relative score exactly +/-1 by construction -- zero information, e.g. C-LEARN's isolated gas-uptake decay loops) sort after all competing loops and are dropped first under the `MAX_LOOPS` cap. Each `FoundLoop` carries a result-scoped dense `partition` index into `DiscoveryResult::partitions` (`DiscoveredPartition { stocks, loop_count }`, first-appearance order; NOT stable across runs/edits -- key on the stock set for durable identity), threaded through `analysis::ModelAnalysis::partitions` / `LoopSummary::partition`, the FFI `SimlinDiscoveredPartition` / `SimlinDiscoveredLoop.partition`, and pysimlin `Analysis.partitions` / `Loop.partition`. Also hosts the link-set synthetic-node collapse: `trim_synthetic_aggs_from_loop_links` collapses `$⁚ltm⁚agg⁚{n}` nodes out of a single loop's link *cycle*, and the public `collapse_synthetic_links(Vec)` generalizes that to ALL synthetic/macro/module-internal nodes (`ltm::is_synthetic_node_name`) over an arbitrary link *set* -- each chain `X -> $⁚…internal… -> Y` collapses to one composite edge `X -> Y` with product polarity and the per-timestep max-magnitude path score (the composite link score, LTM ref 6.3/6.4); a purely-internal cycle/source is dropped. `CollapsibleLink { from, to, polarity, score: Option> }` is the abstract shape, so structural-only callers (`score = None`) and LTM-run callers share one impl. libsimlin's `simlin_analyze_get_links(.., include_internal=false)` collapses the `get_links()` set through this; `include_internal=true` returns the raw graph. The `#[cfg(test)]` tests live in the sibling `src/ltm_finding_tests.rs` (mounted via `#[cfg(test)] #[path]`, split out for the per-file line cap). - **`src/ltm_agg.rs`** - Aggregate-node enumeration for LTM. `enumerate_agg_nodes` (salsa-tracked) walks every variable's `Expr2` AST left-to-right depth-first and identifies each maximal array-reducer subexpression (`SUM`/`MEAN`/`MIN`/`MAX`/`STDDEV`/`RANK`/`SIZE`); AST-identical subexpressions (keyed by canonical printed equation text) dedupe to one `AggNode`. Two kinds: **synthetic** (`is_synthetic == true`, the reducer is a sub-expression of a larger equation -- a `$⁚ltm⁚agg⁚{n}` aux is minted) and **variable-backed** (`is_synthetic == false`, the reducer is the entire dt-equation of a scalar/A2A variable like `total_population = SUM(pop[*])` -- the variable itself is the agg; EXCEPT a whole-RHS reducer whose shape the variable-backed machinery cannot express -- a MAPPED iterated axis (GH #534: the name-based link-score path cannot remap, so its `Wildcard` partial would silently stub to 0) or NON-ALIGNED result dims (GH #764 / shape-expressiveness T4: a broadcast over extra owner dims `out[D1,D3] = SUM(matrix[D1,*])` or permuted axes `out[D2,D1] = SUM(cube[D1,D2,*])`, where a per-`(row, slot)` slot cannot name a complete owner element) -- which mints a synthetic agg instead (`variable_backed_shape_is_expressible`, the ONE minting condition), riding the two-half emitters + the GH #528 projection). Each `AggNode` carries a `sources: Vec` -- one entry per source variable, SORTED by canonical name and deduped (salsa cache-equality + emission-order determinism; T2 of the shape-expressiveness design), each with its OWN `read_slice: Vec` (one `AxisRead ∈ {Pinned(elem), Iterated{dim, source_dim}, Reduced{subset}}` per THAT source's axes -- which rows of it the reducer actually reads; a scalar feeder like `scale` in `SUM(pop[*] * scale)` carries an empty slice; `Iterated` carries the (target, source) canonical dim pair, equal for the literal case; `Reduced.subset` is `None` for the full extent or the proper-subdimension element subset for a `SUM(arr[*:Sub])` StarRange, GH #766, decided per axis by `classify_axis_access` -- the single per-axis classifier of the shape-expressiveness design) -- and a `result_dims` (the `Iterated` axes' TARGET dims, datamodel-cased -- empty for a whole-extent or pinned-slice reduce). Under the I1 acceptance (`accept_source_slices`, T5 of the shape-expressiveness design / GH #767) every arrayed CO-SOURCE (`Reduced`-bearing slice) carries the identical *canonical* slice (`AggNode::canonical_read_slice` -- the first `Reduced`-bearing source slice, falling back to the first non-empty for the degenerate no-co-source agg), while an ITERATED-DIM PROJECTION FEEDER -- a source whose slice is all-`Iterated` over exactly the canonical slice's iterated target dims, in order, unmapped (`frac[D1]` in `SUM(matrix[D1,*] * frac[D1])`, per-result-slot constant) -- is accepted with ITS OWN slice (`AggNode::source_is_projection_feeder` is the discriminator); per-source consumers (`emit_agg_routed_edges`, `emit_source_to_agg_link_scores`) read `AggNode::source_read_slice(from)`, and name-keyed consumers use `AggNode::reads_var`. A feeder's link-score half is the per-`(row, slot)` CHANGED-LAST equation (`ltm_augment::generate_iterated_feeder_to_agg_equation`, emitted via `link_scores::iterated_feeder_row_scores` from both the synthetic source-half and `try_cross_dimensional_link_scores`' variable-backed feeder branch): the reducer text pinned to the slot with only the feeder frozen -- the arrayed generalization of the GH #737 scalar-feeder convention, exactly complementary per slot to the co-source rows' changed-first numerators for a bilinear body; the co-source rows' changed-first body partial pins the mismatched-arity feeder dep BY DIM NAME (`pin_body_to_row`'s GH #767 extension) so `PREVIOUS(frac[d1·r])` is held frozen at the row instead of bailing to the delta-ratio fallback. `compute_read_slice` decides hoistability: a whole-extent reduce (`SUM(pop[*])` ⇒ all-`Reduced`), a sliced one (`SUM(pop[NYC,*])` ⇒ `[Pinned(nyc), Reduced]`, `SUM(matrix[D1,*])` over an A2A-`D1` body ⇒ `[Iterated(d1,d1), Reduced]` → an arrayed agg over `D1`), a mixed one (`SUM(matrix3d[D1,NYC,*])` over an A2A-`D1` body ⇒ `[Iterated(d1,d1), Pinned(nyc), Reduced]`), or a positionally-MAPPED sliced one (GH #534: `SUM(matrix[State,*])` over `matrix[Region,D2]` with a positional `State→Region` mapping ⇒ `[Iterated{state, region}, Reduced]`, `result_dims = [State]` -- the agg is arrayed over the TARGET's iterated dim and the three `Iterated`-axis consumers remap each source row to the slot of its positionally-corresponding target element via `iterated_axis_slot_elements`, the preimage inversion of `mapped_element_correspondence`, so the positional-only/element-map gate is inherited) is hoisted; the carve-outs are (a) a reducer over a *dynamic index* (`SUM(pop[idx,*])`, `idx` non-literal ⇒ not statically describable ⇒ not hoisted, reference stays on the conservative path -- `db::ltm_ir` reclassifies it as `DynamicIndex`), (b) an ELEMENT-mapped sliced reducer the correspondence declines (execution resolves positionally and ignores the map, GH #756; a POSITIONAL mapping is accepted in either declaration direction since GH #757) ⇒ `compute_read_slice` returns `None`, conservative -- (c) a multi-source slice combination outside the I1 acceptance -- co-sources with differing slices, one variable read with two different slices (I3b), or a no-`Reduced` source that is not the pure iterated projection (a Pinned-axis mix like `SUM(matrix[D1,*] * w[D1, c1])`, a dim-subset/permuted feeder, or any mapped Iterated axis in a feeder combination) -- `combined_read_slice`/`accept_source_slices` return `None`, (d) a StarRange naming a NON-subdimension of its axis (a mid-edit inconsistency; declined rather than silently widened to the full extent), and (e) `RANK` (GH #771: array-valued, so `reducer_is_hoistable` requires `reducer_collapses_to_scalar` and RANK references stay `Direct` -- a bare arg classifies `Bare`, scored by the GH #742 arrayed-capture path; loops through the rank ORDERING are a documented residual). A multi-source reducer whose arrayed args *agree* (`SUM(a[*] + b[*])`, `a`, `b` over the same dim) mints one agg with one `AggSource` per variable, each carrying the shared canonical slice. `AggNodesResult` exposes `aggs` (first-encounter order), `agg_for_key` (by canonical reducer text), and `aggs_in_var` (which aggs occur in a variable's equation, so the element-graph reroute can ask "which agg of `to` reads `from`?"). A variable-backed reduce with a NON-TRIVIAL statically-describable slice is gated by `variable_backed_reduce_agg` (GH #752, generalized by GH #765 / T3 of the shape-expressiveness design) -- the SAME gate `model_element_causal_edges`' dispatch, `build_element_level_loops`' per-circuit routing, and `try_cross_dimensional_link_scores`' row derivation consume, so edges, loop routing, and scores always cover the identical read rows (all three derive from `read_slice_rows`, invariant I4). Accepted: an ALIGNED partial reduce (`row_sum[D1] = SUM(matrix[D1,*])`, `result_dims` equal to the variable's own dims in order -- Pinned-mixed `outf[D1] = MEAN(cube[D1,x,*])` and subset `out[D1] = MEAN(matrix[D1,*:Sub])` slices included: the divisor is the true read count and unread rows get neither edges nor scores) gets read-slice element edges straight onto the variable's element nodes and per-circuit scalar loops whose both-subscripted `matrix[d1,d2]→row_sum[d1]` links resolve the per-`(row, slot)` scores; a scalar-result Pinned/subset slice on a SCALAR owner (`total = SUM(pop[nyc,*])`, `total = SUM(arr[*:Sub])`) routes the read rows into the bare `to` node with matching per-read-row scores; and the ARRAYED-owner scalar-result Pinned/subset BROADCAST slice (`share[Region] = SUM(pop[nyc,*])`, no `Iterated` axis -- GH #777) fans each read row across the FULL target element set (`emit_agg_routed_edges`' broadcast arm emits `pop[nyc,d2] → share[e]` for every `e`; `emit_broadcast_reduce_link_scores` emits the matching per-(read-row, full-target-element) scalar scores `pop[nyc,d2]→share[e]`, the section-3 `PerElement` rule applied to a variable-backed reducer owner), with loop circuits routed to the per-circuit scalar path via `is_broadcast_reduce_edge` -- the read rows are independent of `to`'s dims, so the RELATED-dim (`share[Region]`) and DISJOINT-dim (`share[D9]`) spellings emit identically. Declined: a pure full-extent variable-backed agg keeps the normal reference walker's edges (the true reads for that shape, inert skip). The GH #764 broadcast/permuted result shapes never reach this gate since T4 -- they mint synthetic aggs at enumeration -- so the gate's Iterated-arm alignment check is defense-in-depth. `scalar_feeder_of_variable_backed_agg` (GH #790) is the scalar sibling of `source_is_projection_feeder`: it composes `variable_backed_reduce_agg` with an empty-slice + genuine-`Reduced`-canonical check to recognize a SCALAR FEEDER of a whole-RHS variable-backed reduce (`scale` in `growth[D1] = SUM(matrix[D1,*] * scale)`), so `try_scalar_to_arrayed_link_scores` can route it to the single Bare A2A changed-last feeder score instead of the uncompilable per-target-element partials. Two predicates here answer what LOOKS like one question -- "is this reference inside a reducer?" -- and their answers are inverted on exactly `SIZE` and `RANK`; the inversion is the DEFINITION of the difference, not a disagreement (GH #982, assessed and left as two predicates). `reducer_collapses_to_scalar` is about the reducer's RESULT TYPE (does the subtree fit in a scalar slot?) and is read by the two freeze/capture gates plus the GH #779 bare-reducer-feeder decline: `SIZE` is a count so it collapses, `RANK` is array-valued so it does not. `builtin_routes_through_agg` is about LTM ROUTING (did `enumerate_agg_nodes` mint a node for this call?) and sets `db::ltm_ir::OccurrenceSite::in_reducer`: `SIZE` is `Constant` and is never hoisted, `RANK` gets an array-valued agg. Both read the ONE `reducer_kind_from_name` table, and `builtin_routes_through_agg` is the disjunction of the enumerator's own two minting branches (`reducer_is_hoistable` and `array_valued_rank_arg`) rather than a restatement of them. The `#[cfg(test)]` `REDUCER_DECISION_TABLE` pins all three derived predicates row by row over every arm of the kind table -- including the agreement gate's name-keyed twin of the routing predicate -- so no cell can move silently. `is_synthetic_agg_name` / `synthetic_agg_name` are the `$⁚ltm⁚agg⁚{n}` name helpers. `classify_axis_access` resolves a bare-identifier index through the shared `dimensions::resolve_axis_index_name` (element-first, GH #986), so it and `ltm_augment_post_transform::pin_dimension_name_indices` cannot disagree about which row a colliding name selects. The `#[cfg(test)]` tests live in the sibling `src/ltm_agg_tests.rs` (split out for the per-file line cap). Each node also carries `reducer: BuiltinFn` -- the reducer call the enumerator classified when it decided the hoist, of which `equation_text` is the printed rendering (GH #983). It is what makes the link-score and polarity emitters parse-free: `ltm_augment::classify_reducer_in_builtin` reads the kind/name/body off it and `ltm::CausalGraph::source_to_agg_polarity` analyses it directly, where both used to print `equation_text`, re-parse it, re-lower it against a freshly built scope and re-derive the classification -- per (agg, source) pair, with both fallible steps returning early and silently zeroing the agg's loop score. It is stored in `Expr2::strip_loc_and_bounds` form, which removes two of the three ways this field could make the salsa-cached `AggNodesResult` compare unequal to an identical rebuild: `Loc` (load-bearing -- two AST-identical occurrences differ in byte offsets, so raw storage would make the dedup winner observable and would stop `enumerate_agg_nodes` backdating across an offset-only edit) and `ArrayBounds` (a guard -- inert today, since `reconstruct_model_variables` lowers against an empty model scope and allocates no bound). Neither reader looks at either. It cannot remove the third -- dropping a `nan` literal would change what the equation means -- so that one is closed at the ROOT instead: `Expr2::Const` holds an `ast::Literal`, compared by BIT PATTERN, so a model whose hoisted reducer contains a `nan` literal backdates like any other (GH #987/#981; with a bare `f64` it never could, since `NaN != NaN`). Pinned by `a_nan_literal_in_a_reducer_does_not_defeat_agg_backdating`. The stored builtin is read only for SYNTHETIC aggs (both readers filter to those); the variable-backed arm's copy is unread today, kept so `AggNode` has one shape. `AggNode`/`AggNodesResult` derive `Eq` (reflexivity is a compile-checked property now that the literal is not a bare `f64`) and `Debug` only under `debug-derive`. -- **`src/ltm_augment.rs`** - Equation generators for LTM synthetic variables: `generate_link_score_equation_for_link` (ceteris-paribus link scores; takes `RefShape` and source dimension elements to drive per-shape PREVIOUS wrapping), `generate_loop_score_variables` (emits one `loop_score` per loop as a dimension-shaped `datamodel::Equation`: `Scalar` for scalar loops, `ApplyToAll` for dimensioned loops whose links resolve through Bare A2A names, and per-slot `Equation::Arrayed` for dimensioned loops backed by per-element circuits via `Loop.slot_links` -- GH #653; relative loop scores are computed post-simulation in `ltm_post.rs`), `build_partial_equation_shaped` (the `#[cfg(test)]` TEXT entry point for the ceteris-paribus wrap; arrayed-per-element-equation (`Ast::Arrayed`) targets get one partial per element assembled into an `Equation::Arrayed`). **Production never parses a target equation**: `wrap_changed_first_ast` takes an `Expr0` lowered straight from the target's `Expr2` by `patch::expr2_to_expr0` (which is what `expr2_to_string` prints, so the former print->reparse was a parse of our own output), and every per-occurrence decision -- access shape, the GH #526 other-dep verdict, the literal-element index guard, and the `PerElement` row pinning -- is a lookup into the `db::ltm_ir` occurrence IR by the structural child-index path the wrap tracks, which equals the occurrence's `SiteId` BY CONSTRUCTION now that both walk the same tree. A subscripted source reference the IR did NOT record -- reachable under a `LOOKUP` TABLE argument, which the walker skips as static data ("not a causal edge", which is right about ATTRIBUTION) -- still has to COMPILE, so the pin-only descent discharges it by NAME (`pin_dimension_name_indices`: an index naming one of the TARGET's iterated dimensions becomes the source element this target element reads on that axis, the same structural substitution `pin_bare_source_ref` performs for a bare `Var`). That is a lowering-completeness rule, not a second classifier: it consults no occurrence, infers no shape, and never makes the reference live-selectable -- it asks the SHARED row derivation `per_element_row_for_target` (hence `DimensionsContext::mapped_element_correspondence`) which element an axis reads, so the identity axis and a positionally-MAPPED one (`effect[State, old]` over an `effect[Region, Age]` source, either declaration direction) are one arm and it accepts exactly the mapped pairs `ltm_agg::classify_axis_access` accepts. An index the source's axis DECLARES as an element is resolved BEFORE that dimension-name reading, and that precedence is not the pin's own: it is the shared `dimensions::resolve_axis_index_name`, which `ltm_agg::classify_axis_access` reads too (GH #986 closed the divergence -- the classifier had the opposite order, so a mapped collision described the axis as `Iterated` over a dimension the compiler never iterates there). Element-first is what `compiler::subscript`'s `normalize_subscripts3` does ("First check if it's a named dimension element (takes priority)"), and the simulation is the authority: the two readings collide when a dimension declares an element whose name is also a dimension name (`Bucket = [old, region]` beside a `Region` dimension), and a describer that breaks the tie the other way names rows the simulation never reads (`a_colliding_index_name_reads_the_axis_element_in_the_simulation` is the numeric oracle; the XMILE spec's footnote [9] settles the adjacent VARIABLE-vs-element pair outright and sections 2.1/3.7.1 argue this pair from the namespace rule -- `resolve_axis_index_name`'s rustdoc says which is which). Everything that is NOT a bare identifier is left verbatim, needing no pin: a numeric literal, arithmetic over literals, an `@N` position (which `compiler::context` resolves to a concrete element offset in scalar context -- spelling it out here would be a second implementation of position syntax), and a compound expression selecting the element at RUNTIME. That last one used to be a conditional REFUSAL, and deleting it is GH #984: the wrap now freezes a `LOOKUP` table argument's index reads itself (`freeze_lookup_table_indices`), so a runtime index arrives here already lagged and the rule keeps it. That freeze WIDENS its own descent's dep set with the index idents, and without that it would not fire at all -- `variable::classify_dependencies`' `BuiltinContents::LookupTable` arm records the table's ident and never walks the table expression, so an index variable referenced only there is not a dependency and the wrap's `other_deps` freeze could never reach it. The widened set is scoped to that argument's indices, and the element / dimension-name guards run before the dep check, so it cannot make a selector wrap. (Leaving the dep set itself alone is deliberate: an index dropped from a variable's dependencies is a runlist-ordering question, not an LTM one.) What the SHARED derivation declines -- an unmapped or element-mapped pair (GH #756), a transposition, a dimension the target does not iterate -- is declined LOUDLY (`WrapOutcome::missing_occurrence` -> warned skip) rather than emitted with its dimension-name subscript intact, which would not resolve in a scalar fragment. The verdict space (`Pinned`, `Keep`, and the one loud `Unspellable`) is ENUMERATED cell by cell in `ltm_augment_pin_tests.rs`'s three verdict enumerations rather than sampled. There is ONE access-shape classifier family, on `Expr2`; the Expr0 mirror is test-support only and lives in `ltm_augment_wrap_test_support.rs` (kept because three wrap unit tests -- unparseable text, empty text, and the Fig. 2 Q4 `SUM(w[from]) + from` shape the engine REJECTS as a model -- cannot be db-backed fixtures), with `ltm_classifier_agreement_tests.rs` proving it matches the IR field for field (`SiteId` path, `shape`, `axes`, `in_reducer`) corpus-wide, `link_score_var_name` (synthetic name helper: Bare gets the canonical `{from}\u{2192}{to}` form, FixedIndex prepends `[elem]` to from; the obsolete per-shape `\u{205A}wildcard`/`\u{205A}dynamic` Wildcard/DynamicIndex suffixes were retired -- those shapes now collapse onto the Bare name, since *every statically-describable* inlined reducer (whole-extent or sliced) is hoisted into a `$⁚ltm⁚agg⁚{n}` node and only a `DynamicIndex` reference -- `arr[i+1]`, a range, or the not-hoistable dynamic-index reducer carve-out `SUM(pop[idx,*])` -- a whole-RHS variable-backed reducer's `Wildcard` argument, or a de-hoisted array-valued reducer's `Wildcard` arg (`RANK(pop[*], 1)`, GH #771) reach this function), `quote_ident` (identifier quoting for equations). Array support: `classify_reducer` (walks target Expr2 AST to identify reducing builtins -- Linear for SUM/MEAN, Nonlinear for MIN/MAX/STDDEV/RANK, Constant for SIZE -- a thin reader of `ltm_agg::reducer_kind`; it also hands back the reducer's array-argument AST as `ClassifiedReducer::body`, lowered from `Expr2` rather than printed, so the body-aware row partials never re-parse it), `generate_element_to_scalar_equation` (per-element link score equations for arrayed-to-scalar edges, used by both the variable-backed-reducer path and the `source[d] → $⁚ltm⁚agg⁚{n}` half) which dispatches on `ReducerKind` -- `generate_linear_partial` (SUM/MEAN algebraic shortcut), `generate_nonlinear_partial` (MIN/MAX nested binary calls; STDDEV the unrolled population-variance `sqrt` ceteris-paribus partial -- divisor `N`, matching `vm.rs::Opcode::ArrayStddev`, with the mean string-inlined; RANK the documented delta-ratio stand-in pinned by `test_generate_rank_keeps_delta_ratio` -- an order statistic, non-differentiable and unreachable via a real model RHS), `generate_scalar_to_element_equation` (per-element link score for the `$⁚ltm⁚agg⁚{n} → target[e]` half; takes a `source_ref_override: Option<&str>` so a multi-slot arrayed agg's `Δsource` denominator carries the projected `agg[]` subscript instead of the bare agg name, which wouldn't compile as a scalar), `substitute_reducers_in_expr0` (textually replaces a recognized reducer subexpression in an `Expr0` with its agg name, for the `$⁚ltm⁚agg⁚{n} → target` link score), `resolve_link_score_name_for_loop` (picks the Bare-or-FixedIndex link-score name a loop-score reference should target). Module link score formulas (black-box delta-ratio and composite-ref) are inlined directly into `module_link_score_equation` in `db.rs` (called by the per-shape `link_score_equation_text_shaped`). `subscript_idents_at_element` pins a target's arrayed deps for a per-element scalar partial, and it pins each one over the dimensions THAT DEP declares rather than over the target's element tuple (GH #974): a bare arrayed reference in an apply-to-all body reads its own axes' coordinates matched by dimension NAME, so a subset-dims dep (`w[Age]` under `growth[Region,Age]`) got an over-arity subscript whose fragment failed to compile, and a REORDERED one (`w[Age,Region]`) got a subscript that compiled and silently read the transposed element. The projection is `post_transform::dep_element_pins`/`dep_row_for_target`, reused by `pin_bare_source_ref` for a bare reference to the LIVE SOURCE (which is why a positionally-MAPPED bare source reference resolves through `mapped_element_correspondence` instead of being left bare and frozen into an uncompilable multi-slot `PREVIOUS`). The partial-equation builders (`build_partial_equation_shaped`/`_with_live_ref`, `subscript_idents_at_element`) and every link-score equation generator return `Result<_, PartialEquationError>`: a parse failure (genuine `Err`, or an empty `Ok(None)` equation) has no AST to PREVIOUS-wrap, so emitting the unwrapped input would silently produce a non-ceteris-paribus "partial" identical to the full equation (link score magnitude constant |Δz/Δz| = 1) -- a hidden attribution error that compiles cleanly (GH #311). The db-bearing callers (`link_score_equation_text_shaped` and the `src/db/ltm/link_scores.rs` emitters) convert the error into a `Warning` (`emit_ltm_partial_equation_warning`, naming the variable + offending equation text) and skip the variable -- distinct from `model_ltm_fragment_diagnostics`, which only catches *compile* failures. The failure is effectively unreachable in production (the text is always a `print_eqn` re-print; an empty equation is rejected as an `EmptyEquation` Error upstream), so this is defense-in-depth. The `#[cfg(test)]` tests live in the sibling `src/ltm_augment_tests.rs` (split out for the per-file line cap). Four more siblings are `#[path]`-mounted into `ltm_augment` purely for that cap, so every caller still names their items `crate::ltm_augment::*`: **`ltm_augment_occurrence.rs`** (the wrap's read side of the occurrence IR -- `SlotOccurrences` groups a target's stream by slot ONCE and is the only way to obtain an `OccurrenceLookup`, so the borrow forces callers to hoist it out of their per-element loop), **`ltm_augment_post_transform.rs`** (the concrete-form lowerings: the agg-name substitution, and the `PerElement` row pinning the wrap calls AS IT DESCENDS -- the wrap is the only place that knows both the occurrence and whether it is about to FREEZE the reference, which is what picks the bare row for the live occurrence over the qualified row for every other one), **`ltm_augment_with_lookup.rs`** (the GH #910 implicit-WITH-LOOKUP rules), and **`ltm_augment_wrap_test_support.rs`** (the `#[cfg(test)]` occurrence reconstruction + the Expr0 classifier mirror described above). +- **`src/ltm_augment.rs`** - Equation generators for LTM synthetic variables: `generate_link_score_equation_for_link` (ceteris-paribus link scores; takes `RefShape` and source dimension elements to drive per-shape PREVIOUS wrapping), `generate_loop_score_variables` (emits one `loop_score` per loop as a dimension-shaped `datamodel::Equation`: `Scalar` for scalar loops, `ApplyToAll` for dimensioned loops whose links resolve through Bare A2A names, and per-slot `Equation::Arrayed` for dimensioned loops backed by per-element circuits via `Loop.slot_links` -- GH #653; relative loop scores are computed post-simulation in `ltm_post.rs`), `build_partial_equation_shaped` (the `#[cfg(test)]` TEXT entry point for the ceteris-paribus wrap; arrayed-per-element-equation (`Ast::Arrayed`) targets get one partial per element assembled into an `Equation::Arrayed`). **Production never parses a target equation**: `wrap_changed_first_ast` takes an `Expr0` lowered straight from the target's `Expr2` by `patch::expr2_to_expr0` (which is what `expr2_to_string` prints, so the former print->reparse was a parse of our own output), and every per-occurrence decision -- access shape, the GH #526 other-dep verdict, the literal-element index guard, and the `PerElement` row pinning -- is a lookup into the `db::ltm_ir` occurrence IR by the structural child-index path the wrap tracks, which equals the occurrence's `SiteId` BY CONSTRUCTION now that both walk the same tree. Every `PREVIOUS` the wrap SYNTHESIZES goes through `freeze_at_previous`, which chooses the call's first-DT initial value from the position being frozen: a VALUE position keeps the unary spelling (desugared to `0`, the XMILE-documented default, and unobservable behind the guard form's `TIME = INITIAL_TIME` arm), while a SUBSCRIPT INDEX names its own un-lagged operand -- `PREVIOUS(idx, idx)` -- because `0` is out of range for every 1-based dimension, so the frozen read yielded NaN at t=0 and `make_temp_arg`'s capture helper served that NaN as the score's FIRST LIVE step (GH #975). Only the two walkers that descend into indices need it (`wrap_non_matching_in_previous` via `wrap_index_non_matching_in_previous`, and `wrap_matching_in_previous`); `wrap_live_shaped_in_previous` and `freeze_pinned_body` document that they never do. A subscripted source reference the IR did NOT record -- reachable under a `LOOKUP` TABLE argument, which the walker skips as static data ("not a causal edge", which is right about ATTRIBUTION) -- still has to COMPILE, so the pin-only descent discharges it by NAME (`pin_dimension_name_indices`: an index naming one of the TARGET's iterated dimensions becomes the source element this target element reads on that axis, the same structural substitution `pin_bare_source_ref` performs for a bare `Var`). That is a lowering-completeness rule, not a second classifier: it consults no occurrence, infers no shape, and never makes the reference live-selectable -- it asks the SHARED row derivation `per_element_row_for_target` (hence `DimensionsContext::mapped_element_correspondence`) which element an axis reads, so the identity axis and a positionally-MAPPED one (`effect[State, old]` over an `effect[Region, Age]` source, either declaration direction) are one arm and it accepts exactly the mapped pairs `ltm_agg::classify_axis_access` accepts. An index the source's axis DECLARES as an element is resolved BEFORE that dimension-name reading, and that precedence is not the pin's own: it is the shared `dimensions::resolve_axis_index_name`, which `ltm_agg::classify_axis_access` reads too (GH #986 closed the divergence -- the classifier had the opposite order, so a mapped collision described the axis as `Iterated` over a dimension the compiler never iterates there). Element-first is what `compiler::subscript`'s `normalize_subscripts3` does ("First check if it's a named dimension element (takes priority)"), and the simulation is the authority: the two readings collide when a dimension declares an element whose name is also a dimension name (`Bucket = [old, region]` beside a `Region` dimension), and a describer that breaks the tie the other way names rows the simulation never reads (`a_colliding_index_name_reads_the_axis_element_in_the_simulation` is the numeric oracle; the XMILE spec's footnote [9] settles the adjacent VARIABLE-vs-element pair outright and sections 2.1/3.7.1 argue this pair from the namespace rule -- `resolve_axis_index_name`'s rustdoc says which is which). Everything that is NOT a bare identifier is left verbatim, needing no pin: a numeric literal, arithmetic over literals, an `@N` position (which `compiler::context` resolves to a concrete element offset in scalar context -- spelling it out here would be a second implementation of position syntax), and a compound expression selecting the element at RUNTIME. That last one used to be a conditional REFUSAL, and deleting it is GH #984: the wrap now freezes a `LOOKUP` table argument's index reads itself (`freeze_lookup_table_indices`), so a runtime index arrives here already lagged and the rule keeps it. That freeze WIDENS its own descent's dep set with the index idents, and without that it would not fire at all -- `variable::classify_dependencies`' `BuiltinContents::LookupTable` arm records the table's ident and never walks the table expression, so an index variable referenced only there is not a dependency and the wrap's `other_deps` freeze could never reach it. The widened set is scoped to that argument's indices, and the element / dimension-name guards run before the dep check, so it cannot make a selector wrap. (Leaving the dep set itself alone is deliberate: an index dropped from a variable's dependencies is a runlist-ordering question, not an LTM one.) What the SHARED derivation declines -- an unmapped or element-mapped pair (GH #756), a transposition, a dimension the target does not iterate -- is declined LOUDLY (`WrapOutcome::missing_occurrence` -> warned skip) rather than emitted with its dimension-name subscript intact, which would not resolve in a scalar fragment. The verdict space (`Pinned`, `Keep`, and the one loud `Unspellable`) is ENUMERATED cell by cell in `ltm_augment_pin_tests.rs`'s three verdict enumerations rather than sampled. There is ONE access-shape classifier family, on `Expr2`; the Expr0 mirror is test-support only and lives in `ltm_augment_wrap_test_support.rs` (kept because three wrap unit tests -- unparseable text, empty text, and the Fig. 2 Q4 `SUM(w[from]) + from` shape the engine REJECTS as a model -- cannot be db-backed fixtures), with `ltm_classifier_agreement_tests.rs` proving it matches the IR field for field (`SiteId` path, `shape`, `axes`, `in_reducer`) corpus-wide, `link_score_var_name` (synthetic name helper: Bare gets the canonical `{from}\u{2192}{to}` form, FixedIndex prepends `[elem]` to from; the obsolete per-shape `\u{205A}wildcard`/`\u{205A}dynamic` Wildcard/DynamicIndex suffixes were retired -- those shapes now collapse onto the Bare name, since *every statically-describable* inlined reducer (whole-extent or sliced) is hoisted into a `$⁚ltm⁚agg⁚{n}` node and only a `DynamicIndex` reference -- `arr[i+1]`, a range, or the not-hoistable dynamic-index reducer carve-out `SUM(pop[idx,*])` -- a whole-RHS variable-backed reducer's `Wildcard` argument, or a de-hoisted array-valued reducer's `Wildcard` arg (`RANK(pop[*], 1)`, GH #771) reach this function), `quote_ident` (identifier quoting for equations). Array support: `classify_reducer` (walks target Expr2 AST to identify reducing builtins -- Linear for SUM/MEAN, Nonlinear for MIN/MAX/STDDEV/RANK, Constant for SIZE -- a thin reader of `ltm_agg::reducer_kind`; it also hands back the reducer's array-argument AST as `ClassifiedReducer::body`, lowered from `Expr2` rather than printed, so the body-aware row partials never re-parse it), `generate_element_to_scalar_equation` (per-element link score equations for arrayed-to-scalar edges, used by both the variable-backed-reducer path and the `source[d] → $⁚ltm⁚agg⁚{n}` half) which dispatches on `ReducerKind` -- `generate_linear_partial` (SUM/MEAN algebraic shortcut), `generate_nonlinear_partial` (MIN/MAX nested binary calls; STDDEV the unrolled population-variance `sqrt` ceteris-paribus partial -- divisor `N`, matching `vm.rs::Opcode::ArrayStddev`, with the mean string-inlined; RANK the documented delta-ratio stand-in pinned by `test_generate_rank_keeps_delta_ratio` -- an order statistic, non-differentiable and unreachable via a real model RHS), `generate_scalar_to_element_equation` (per-element link score for the `$⁚ltm⁚agg⁚{n} → target[e]` half; takes a `source_ref_override: Option<&str>` so a multi-slot arrayed agg's `Δsource` denominator carries the projected `agg[]` subscript instead of the bare agg name, which wouldn't compile as a scalar), `substitute_reducers_in_expr0` (textually replaces a recognized reducer subexpression in an `Expr0` with its agg name, for the `$⁚ltm⁚agg⁚{n} → target` link score), `resolve_link_score_name_for_loop` (picks the Bare-or-FixedIndex link-score name a loop-score reference should target). Module link score formulas (black-box delta-ratio and composite-ref) are inlined directly into `module_link_score_equation` in `db.rs` (called by the per-shape `link_score_equation_text_shaped`). `subscript_idents_at_element` pins a target's arrayed deps for a per-element scalar partial, and it pins each one over the dimensions THAT DEP declares rather than over the target's element tuple (GH #974): a bare arrayed reference in an apply-to-all body reads its own axes' coordinates matched by dimension NAME, so a subset-dims dep (`w[Age]` under `growth[Region,Age]`) got an over-arity subscript whose fragment failed to compile, and a REORDERED one (`w[Age,Region]`) got a subscript that compiled and silently read the transposed element. The projection is `post_transform::dep_element_pins`/`dep_row_for_target`, reused by `pin_bare_source_ref` for a bare reference to the LIVE SOURCE (which is why a positionally-MAPPED bare source reference resolves through `mapped_element_correspondence` instead of being left bare and frozen into an uncompilable multi-slot `PREVIOUS`). The partial-equation builders (`build_partial_equation_shaped`/`_with_live_ref`, `subscript_idents_at_element`) and every link-score equation generator return `Result<_, PartialEquationError>`: a parse failure (genuine `Err`, or an empty `Ok(None)` equation) has no AST to PREVIOUS-wrap, so emitting the unwrapped input would silently produce a non-ceteris-paribus "partial" identical to the full equation (link score magnitude constant |Δz/Δz| = 1) -- a hidden attribution error that compiles cleanly (GH #311). The db-bearing callers (`link_score_equation_text_shaped` and the `src/db/ltm/link_scores.rs` emitters) convert the error into a `Warning` (`emit_ltm_partial_equation_warning`, naming the variable + offending equation text) and skip the variable -- distinct from `model_ltm_fragment_diagnostics`, which only catches *compile* failures. The failure is effectively unreachable in production (the text is always a `print_eqn` re-print; an empty equation is rejected as an `EmptyEquation` Error upstream), so this is defense-in-depth. The `#[cfg(test)]` tests live in the sibling `src/ltm_augment_tests.rs` (split out for the per-file line cap). Five more siblings are `#[path]`-mounted into `ltm_augment` purely for that cap, so every caller still names their items `crate::ltm_augment::*`: **`ltm_augment_occurrence.rs`** (the wrap's read side of the occurrence IR -- `SlotOccurrences` groups a target's stream by slot ONCE and is the only way to obtain an `OccurrenceLookup`, so the borrow forces callers to hoist it out of their per-element loop), **`ltm_augment_post_transform.rs`** (the concrete-form lowerings: the agg-name substitution, and the `PerElement` row pinning the wrap calls AS IT DESCENDS -- the wrap is the only place that knows both the occurrence and whether it is about to FREEZE the reference, which is what picks the bare row for the live occurrence over the qualified row for every other one), **`ltm_augment_with_lookup.rs`** (the GH #910 implicit-WITH-LOOKUP rules), **`ltm_augment_wrap_test_support.rs`** (the `#[cfg(test)]` occurrence reconstruction + the Expr0 classifier mirror described above), and **`ltm_augment_freeze.rs`** (the GH #975 first-DT initial value of every synthesized `PREVIOUS`). - **`src/ltm_augment_with_lookup.rs`** - The implicit-WITH-LOOKUP rules for LTM (GH #910), re-exported from `ltm_augment` so every `crate::ltm_augment::*` path is unchanged. A `v = WITH LOOKUP(input, table)` variable is lowered by the compiler to `LOOKUP(v, input)` (`compiler::apply_implicit_with_lookup`), so a link-score partial that RE-EVALUATES such a target's equation is in gf-INPUT units while the guard form's `PREVIOUS(target)` anchor is in gf-OUTPUT units. `is_implicit_with_lookup` carries the coverage doc: every partial is either a **full re-evaluation** (class 1 -- must be wrapped in the gf application) or a **delta-ratio stand-in** (class 2 -- the RANK arm and the nested-arithmetic arm, already in output units, must NEVER be wrapped or a gf output is fed back through the gf). `WithLookupSlotRefs` resolves the target's table reference ONCE per target (`NoGf` / `Shared` / `PerElement`), so a per-element-gf target costs one row-major `SubscriptIterator` walk rather than one per element; an arrayed target's shared table is pinned as `to[1,...]` because a bare arrayed reference resolves each iterated element's own table offset, which the VM reads as NaN past `table_count`. `compose_with_lookup_polarity` (in `src/ltm/polarity.rs`) is the polarity twin, mirroring `apply_implicit_with_lookup`'s placeholder-Positive and zero-point-table rules. The polarity tests live in `src/ltm/with_lookup_tests.rs`, a child module of `ltm::tests`. - **`src/ltm_post.rs`** - Post-simulation relative loop *and link* score computation. `compute_rel_loop_scores(results, loop_partitions)` normalizes each loop's `loop_score` series against the sum of absolute scores within its cycle partition, using SAFEDIV-0 semantics (zero denominator -> zero result). Called after simulation rather than emitted as synthetic equations to avoid O(P^2) equation-text growth on models with dense partitions. `loop_partitions` is an `IndexMap` (re-exported `engine::indexmap`). `compute_rel_loop_scores*` walk its **emission** order rather than re-sorting the loop ids: the partition-sum denominator accumulates `|loop_score|` in that order, and emission order keeps the IEEE-754 (non-associative) sum bit-for-bit identical to the pre-#461 compile-time emitter (GH #468). Emission order is itself deterministic across salsa cache invalidations / processes because `assign_loop_ids` orders loops by a content-derived key (`ltm::graph::loop_id_sort_key`), so it never flaps even though `IndexMap`'s `PartialEq` (salsa cache equality) is order-insensitive. `compute_rel_link_scores(links, step_count)` is the link-level analogue (GH #652): raw link scores divide by the change in the *target*, so they are not comparable across targets and ranking by raw magnitude surfaces numerically-degenerate links (near-constant targets blow up the score). It groups the input `RelLinkInput { to, score }` links by their `to` target and normalizes each link's score by the per-target, per-timestep sum of `|score|` over all that target's *scored* inputs -- a **signed** value in `[-1, 1]` (sign kept like `compute_rel_loop_scores`), with the same `denom_summand` NaN-exclusion / Inf-retention / SAFEDIV-0 semantics. Denominator scope is the scored inputs only: complete in discovery mode (every causal edge scored) but covering just the in-loop subset in exhaustive mode (documented caveat on the fn). libsimlin's `analyze_links_core` calls it over the final (post-synthetic-collapse) link set so the per-target denominator matches the links the caller receives. - **LTM open work**: known LTM bugs and improvements are tracked on GitHub under the `ltm` label; issue #488 is the pinned epic that organises them by area (core algorithm, discovery/post-sim, augmentation, module/array umbrellas). Each open `ltm`-labelled issue carries file:line references and a suggested fix, so a new session can pick a bite-sized piece without re-investigating the subsystem. @@ -196,7 +196,7 @@ The unit subsystem is partial-result throughout: a single bad declaration or one ## Utilities -- **`src/float.rs`** - Floating-point helpers. `approx_eq` is the ULP-based f64 equality used throughout. `crate::float::NA` is Vensim's `:NA:` ("missing data") sentinel: the *finite* value `-2^109` (exactly representable in f64), **NOT** IEEE NaN. It is finite by design so the Vensim existence idiom `IF THEN ELSE(x = :NA:, ...)` works (`approx_eq` matches the sentinel against itself, never against a contaminated value) and `:NA:` arithmetic stays finite rather than being poisoned by an absorbing NaN. Both `:NA:` entry points route here: the expression literal via the MDL→XMILE formatter (`src/mdl/xmile_compat.rs`) and the data-list literal via the MDL number-list parser (`src/mdl/parser.rs`, `NA_VALUE`). This is distinct from the genuine NaN the engine still produces for out-of-bounds vector reads (`vm_vector_elm_map.rs`) and empty array reducers (`vm.rs` ArrayMax/Min/Mean/Stddev). **The module docs are the landing page for "why does this codebase treat NaN the way it does"**, and the two conventions belong together: `NA` is finite and comparable ON PURPOSE, while a NaN is a propagating failure signal. What a practitioner sees is a line on a graph that stops, and their next task is provenance -- searching BACKWARD through the dependency graph, because NaN is absorbing and every variable downstream of the origin shows the identical symptom. Two engine consequences follow: attributing a NaN to its origin is high-value (wherever we know STRUCTURALLY that a variable must be NaN -- an unfilled equation being the clearest case -- a warning naming the variable replaces the whole hand search), and a NaN the ENGINE manufactures is noise in a channel practitioners already debug by hand (indistinguishable on the graph from the user's own division by zero, so it costs a debugging session that ends at our bug -- which is what makes GH #975 a real defect, not a cosmetic one). The technical footnote follows rather than motivates: a practitioner cannot author a NaN payload (one `nan` keyword, one canonical `f64::NAN`), so the engine needs no machinery treating NaN as a structured value, and bit-pattern comparison distinguishes nothing that exists. +- **`src/float.rs`** - Floating-point helpers. `approx_eq` is the ULP-based f64 equality used throughout. `crate::float::NA` is Vensim's `:NA:` ("missing data") sentinel: the *finite* value `-2^109` (exactly representable in f64), **NOT** IEEE NaN. It is finite by design so the Vensim existence idiom `IF THEN ELSE(x = :NA:, ...)` works (`approx_eq` matches the sentinel against itself, never against a contaminated value) and `:NA:` arithmetic stays finite rather than being poisoned by an absorbing NaN. Both `:NA:` entry points route here: the expression literal via the MDL→XMILE formatter (`src/mdl/xmile_compat.rs`) and the data-list literal via the MDL number-list parser (`src/mdl/parser.rs`, `NA_VALUE`). This is distinct from the genuine NaN the engine still produces for out-of-bounds vector reads (`vm_vector_elm_map.rs`) and empty array reducers (`vm.rs` ArrayMax/Min/Mean/Stddev). **The module docs are the landing page for "why does this codebase treat NaN the way it does"**, and the two conventions belong together: `NA` is finite and comparable ON PURPOSE, while a NaN is a propagating failure signal. What a practitioner sees is a line on a graph that stops, and their next task is provenance -- searching BACKWARD through the dependency graph, because NaN is absorbing and every variable downstream of the origin shows the identical symptom. Two engine consequences follow: attributing a NaN to its origin is high-value (wherever we know STRUCTURALLY that a variable must be NaN -- an unfilled equation being the clearest case -- a warning naming the variable replaces the whole hand search), and a NaN the ENGINE manufactures is noise in a channel practitioners already debug by hand (indistinguishable on the graph from the user's own division by zero, so it costs a debugging session that ends at our bug -- which is what makes GH #975 a real defect, not a cosmetic one; its root cause is the shape to watch for, an equation the ENGINE wrote whose first-DT value was the language default `0` in a position -- a subscript index -- where no dimension admits it, fixed by naming the un-lagged index as that freeze's initial value in `ltm_augment::freeze_at_previous`). The technical footnote follows rather than motivates: a practitioner cannot author a NaN payload (one `nan` keyword, one canonical `f64::NAN`), so the engine needs no machinery treating NaN as a structured value, and bit-pattern comparison distinguishes nothing that exists. - **`src/io.rs`** - `atomic_write(path, contents)`: writes bytes to a sibling `.new` temp file, fsyncs it, then renames over the target. Cleans up the temp file on error. Best-effort parent-directory fsync after rename for durability on power loss. Used by MCP and CLI tools that need crash-safe file output. ## Cargo features diff --git a/src/simlin-engine/src/db/ltm_char_golden/per_element_dynamic_index.txt b/src/simlin-engine/src/db/ltm_char_golden/per_element_dynamic_index.txt index 175d7a9eb..12e5a56a0 100644 --- a/src/simlin-engine/src/db/ltm_char_golden/per_element_dynamic_index.txt +++ b/src/simlin-engine/src/db/ltm_char_golden/per_element_dynamic_index.txt @@ -1,6 +1,6 @@ $⁚ltm⁚link_score⁚pop[boston,young]→growth[boston] -scalar: if (TIME = INITIAL_TIME) then 0 else if ((growth[region·boston] - PREVIOUS(growth[region·boston])) = 0) OR ((pop[region·boston,age·young] - PREVIOUS(pop[region·boston,age·young])) = 0) then 0 else SAFEDIV(((pop[boston, young] * PREVIOUS(pop[region·boston, PREVIOUS(idx)]) * 0.001) - PREVIOUS(growth[region·boston])), ABS((growth[region·boston] - PREVIOUS(growth[region·boston]))), 0) * SIGN((pop[region·boston,age·young] - PREVIOUS(pop[region·boston,age·young]))) +scalar: if (TIME = INITIAL_TIME) then 0 else if ((growth[region·boston] - PREVIOUS(growth[region·boston])) = 0) OR ((pop[region·boston,age·young] - PREVIOUS(pop[region·boston,age·young])) = 0) then 0 else SAFEDIV(((pop[boston, young] * PREVIOUS(pop[region·boston, PREVIOUS(idx, idx)]) * 0.001) - PREVIOUS(growth[region·boston])), ABS((growth[region·boston] - PREVIOUS(growth[region·boston]))), 0) * SIGN((pop[region·boston,age·young] - PREVIOUS(pop[region·boston,age·young]))) $⁚ltm⁚link_score⁚pop[nyc,young]→growth[nyc] -scalar: if (TIME = INITIAL_TIME) then 0 else if ((growth[region·nyc] - PREVIOUS(growth[region·nyc])) = 0) OR ((pop[region·nyc,age·young] - PREVIOUS(pop[region·nyc,age·young])) = 0) then 0 else SAFEDIV(((pop[nyc, young] * PREVIOUS(pop[region·nyc, PREVIOUS(idx)]) * 0.001) - PREVIOUS(growth[region·nyc])), ABS((growth[region·nyc] - PREVIOUS(growth[region·nyc]))), 0) * SIGN((pop[region·nyc,age·young] - PREVIOUS(pop[region·nyc,age·young]))) +scalar: if (TIME = INITIAL_TIME) then 0 else if ((growth[region·nyc] - PREVIOUS(growth[region·nyc])) = 0) OR ((pop[region·nyc,age·young] - PREVIOUS(pop[region·nyc,age·young])) = 0) then 0 else SAFEDIV(((pop[nyc, young] * PREVIOUS(pop[region·nyc, PREVIOUS(idx, idx)]) * 0.001) - PREVIOUS(growth[region·nyc])), ABS((growth[region·nyc] - PREVIOUS(growth[region·nyc]))), 0) * SIGN((pop[region·nyc,age·young] - PREVIOUS(pop[region·nyc,age·young]))) $⁚ltm⁚link_score⁚pop→growth dims=[Region] -a2a[Region]: if (TIME = INITIAL_TIME) then 0 else if ((growth - PREVIOUS(growth)) = 0) OR ((SUM(pop[region, PREVIOUS(idx)]) - PREVIOUS(SUM(pop[region, PREVIOUS(idx)]))) = 0) then 0 else SAFEDIV(((PREVIOUS(pop[region, age·young]) * pop[region, PREVIOUS(idx)] * 0.001) - PREVIOUS(growth)), ABS((growth - PREVIOUS(growth))), 0) * SIGN((SUM(pop[region, PREVIOUS(idx)]) - PREVIOUS(SUM(pop[region, PREVIOUS(idx)])))) +a2a[Region]: if (TIME = INITIAL_TIME) then 0 else if ((growth - PREVIOUS(growth)) = 0) OR ((SUM(pop[region, PREVIOUS(idx, idx)]) - PREVIOUS(SUM(pop[region, PREVIOUS(idx, idx)]))) = 0) then 0 else SAFEDIV(((PREVIOUS(pop[region, age·young]) * pop[region, PREVIOUS(idx, idx)] * 0.001) - PREVIOUS(growth)), ABS((growth - PREVIOUS(growth))), 0) * SIGN((SUM(pop[region, PREVIOUS(idx, idx)]) - PREVIOUS(SUM(pop[region, PREVIOUS(idx, idx)])))) diff --git a/src/simlin-engine/src/db/ltm_char_tests.rs b/src/simlin-engine/src/db/ltm_char_tests.rs index 96d25fd91..5416c08fe 100644 --- a/src/simlin-engine/src/db/ltm_char_tests.rs +++ b/src/simlin-engine/src/db/ltm_char_tests.rs @@ -1180,23 +1180,52 @@ fn char_per_element_dynamic_index() { } // Finding 1 materiality guard: the DYNAMIC-index sibling of the ambiguous-pin -// guard. The frozen `pop[Region, idx]` occurrence lowers to a `PREVIOUS(idx)` +// guard. The frozen `pop[Region, idx]` occurrence lowers to a `PREVIOUS(idx, idx)` // lag; the buggy blanket-skip would instead leave `idx` un-lagged, a // value-divergent change the constant-`pop` text golden alone would still catch -// (the golden diverges) but whose RUNTIME consequence this guard freezes. HEAD's -// series carries a NaN at the first live step (a pre-existing HEAD behavior: the -// synthesized `PREVIOUS(idx)` capture-helper aux reads an uninitialized value at -// the initial step; tracked separately as a latent bug). Reproducing that NaN is -// the byte-parity contract; the un-lagged regression makes the same step FINITE, -// so `results[1]` being NaN is the exact discriminator. This guard therefore -// pins (a) every fragment compiles (the `PREVIOUS(idx)` helper is well-formed), -// (b) the first-live-step score is NaN (the dynamic-index lag is present), and -// (c) a later step is finite and non-zero (the score is materially live). +// (the golden diverges) but whose RUNTIME consequence this guard freezes. +// +// GH #975 changed what the runtime discriminator can be. Before it, the lag's +// signature was a NaN at the first live step: the wrap emitted a UNARY +// `PREVIOUS(idx)`, which desugars to a `0` first-DT value, and `0` is out of +// range for a 1-based subscript -- so the synthesized capture-helper aux +// evaluated to NaN at t=0 and the outer `PREVIOUS` served that NaN as the first +// live step. The freeze now names the un-lagged index as its own first-DT value, +// so every step is finite and "finite at step 1" no longer discriminates +// anything. +// +// The model is therefore this guard's OWN rather than `per_element_dynamic_index_model`'s +// (which it used to share): there `pop[.,young]` and `pop[.,old]` hold the same +// value at every step, so the lagged and un-lagged index reads are numerically +// indistinguishable and the NaN was the only available signal. Here `skew` adds +// material to `old` and not to `young`, so the two Age rows diverge and the +// capture helper's series -- `pop[nyc, idx_{t-1}]` -- is different from the +// un-lagged `pop[nyc, idx_t]` at every step after the first. +// +// The guard pins (a) every fragment compiles (the `PREVIOUS(idx, idx)` helper is +// well-formed), (b) the capture helper is FINITE at t=0 and equals the LAGGED +// element read at every step -- checked against the model's own simulated `pop` +// series, with the un-lagged read asserted to differ so the check has teeth -- +// and (c) the score itself is finite at the first live step and materially live +// after it. fn per_element_dynamic_index_feedback_model() -> datamodel::Project { - // Same shape as `per_element_dynamic_index_model` (pop is already a - // growth-fed stock there), reused directly so the guard and the text pin - // exercise the identical model. - per_element_dynamic_index_model() + TestProject::new("dyn_index_feedback") + .with_sim_time(0.0, 3.0, 1.0) + .named_dimension("Region", &["nyc", "boston"]) + .named_dimension("Age", &["young", "old"]) + .scalar_aux("idx", "1 + (TIME MOD 2)") + // Per-Age drift with no dependency on `pop`: it separates the two Age + // rows without adding a feedback loop that would change the shape under + // test. + .array_with_ranges("agedrift[Age]", vec![("young", "0"), ("old", "5")]) + .array_flow("skew[Region,Age]", "agedrift[Age]", None) + .array_flow( + "growth[Region]", + "pop[Region, young] * pop[Region, idx] * 0.001", + None, + ) + .array_stock("pop[Region,Age]", "10", &["growth", "skew"], &[], None) + .build_datamodel() } #[test] @@ -1256,17 +1285,63 @@ fn per_element_dynamic_index_scores_preserve_head_lag() { .expect("simulation should run to completion"); let results = vm.into_results(); let rows: Vec<_> = results.iter().collect(); + let series = |name: &str| -> Vec { + let off = compiled.offsets[&Ident::::new(name)]; + rows.iter().map(|r| r[off]).collect() + }; + + // (b) The capture helper: `pop[nyc, PREVIOUS(idx, idx)]`, hoisted out of the + // frozen occurrence by `builtins_visitor::make_temp_arg`. It is the slot + // GH #975 was about -- the outer `PREVIOUS` serves its t=0 value as the + // score's first live step -- so it is asserted directly rather than through + // the score. + let helper = "$\u{205A}$\u{205A}ltm\u{205A}link_score\u{205A}pop[nyc,young]\u{2192}growth[nyc]\u{205A}0\u{205A}arg0"; + let helper_series = series(helper); + assert!( + helper_series[0].is_finite(), + "the synthesized PREVIOUS(idx) capture helper must have a well-defined \ + value at the initial step (GH #975); got {:?}", + helper_series + ); + // The lag itself, checked elementwise against the model's own `pop` series: + // `idx` is `1 + (TIME MOD 2)`, so the LAGGED read selects `young` at t=0/1 + // and alternates thereafter, while the UN-lagged read selects the other row. + // `young`/`old` diverge (see the fixture), so the two are distinguishable. + let idx_series = series("idx"); + let young = series("pop[nyc,young]"); + let old = series("pop[nyc,old]"); + let row_at = |i: usize, index: f64| if index == 1.0 { young[i] } else { old[i] }; + let mut lag_is_observable = false; + for t in 0..helper_series.len() { + // At t=0 the freeze's own first-DT initial value applies: the un-lagged + // index. Afterwards it is the index one step back. + let lagged_index = if t == 0 { + idx_series[0] + } else { + idx_series[t - 1] + }; + assert_eq!( + helper_series[t], + row_at(t, lagged_index), + "the capture helper must read pop at the LAGGED index at step {t}; \ + helper={helper_series:?} idx={idx_series:?} young={young:?} old={old:?}" + ); + lag_is_observable |= row_at(t, lagged_index) != row_at(t, idx_series[t]); + } + assert!( + lag_is_observable, + "the fixture must distinguish the lagged read from the un-lagged one, or \ + the assertion above cannot detect a dropped lag; young={young:?} old={old:?}" + ); + + // (c) The scores themselves: finite from the first live step on (the NaN + // GH #975 removed), and materially live afterwards. for off in score_offsets { - // Step 1 (the first live step) reproduces HEAD's NaN, the signature of - // the `PREVIOUS(idx)` lag reading its uninitialized capture helper. An - // un-lagged regression (`idx` not wrapped) makes this step FINITE. assert!( - rows[1][off].is_nan(), - "dynamic-index pop->growth score at offset {off} step 1 must be NaN \ - (the preserved PREVIOUS(idx) dynamic-index lag); a finite value means \ - the lag was dropped" + rows[1][off].is_finite(), + "dynamic-index pop->growth score at offset {off} step 1 must be finite: \ + the synthesized PREVIOUS(idx) capture helper is initialized (GH #975)" ); - // A later step is finite and non-zero: the score is materially live. let has_finite_nonzero = rows .iter() .skip(2) @@ -1986,9 +2061,11 @@ fn lookup_table_runtime_index_is_frozen_through_the_production_path() { &["link_score\u{205A}src\u{2192}y"], ); assert!( - text.contains("lookup(g[PREVIOUS(idx)], src)"), + text.contains("lookup(g[PREVIOUS(idx, idx)], src)"), "the table index must be frozen even though production does not classify \ - it as a dependency; got: {text}" + it as a dependency, and (GH #975) its freeze must name the un-lagged \ + index as its first-DT initial value rather than the desugared 0, which \ + is out of range for a 1-based subscript; got: {text}" ); assert!( !text.contains("lookup(PREVIOUS("), diff --git a/src/simlin-engine/src/db/ltm_tests.rs b/src/simlin-engine/src/db/ltm_tests.rs index d9511612e..cd5dfcc8c 100644 --- a/src/simlin-engine/src/db/ltm_tests.rs +++ b/src/simlin-engine/src/db/ltm_tests.rs @@ -1089,3 +1089,271 @@ fn collect_agg_petals_groups_single_agg_circuits() { // structurally impossible. The emitted (changed-LAST) text for that probe is // still pinned by the `scalar_feeder_bare_in_hoisted_reducer` characterization // golden (`db::ltm_char_tests`). + +// --------------------------------------------------------------------------- +// GH #986 (third consumer): the ceteris-paribus wrap resolves a bare-identifier +// subscript index against the AXIS IT INDEXES, not against the project's whole +// element namespace. +// +// `dimensions::resolve_axis_index_name` is the engine's single +// element-vs-dimension precedence rule, and it is what +// `compiler::subscript::normalize_subscripts3` implements. GH #986 unified +// `ltm_agg::classify_axis_access` and `post_transform::pin_dimension_name_indices` +// onto it and left the wrap on two PROJECT-WIDE predicates +// (`dimension_uniquely_containing_element`, `is_element_of_any_dimension`). +// +// The consequence was a wrong NUMBER with no diagnostic. A model variable whose +// canonical name happens to be an element of an UNRELATED dimension read as a +// runtime value to the simulation and as a static element selector to the wrap, +// so the "ceteris-paribus" partial left it LIVE and moved with it -- reporting +// real influence for an edge with no causal dependence at all. The qualification +// step made it worse: the index was rewritten to `otherdim·name`, naming an +// element of a dimension the subscripted variable is not even declared over, +// which still compiles and reads a different slot than the anchor did. +// --------------------------------------------------------------------------- + +/// `share[boston]` reads `q`/`gtab` at the runtime index `ctr`, and has no +/// causal dependence on `pop[nyc]` whatsoever. +/// +/// `declare_bucket` adds a dimension **no equation references**, whose first +/// element is named `ctr` -- the same canonical name as the model variable. It +/// changes nothing about the simulation; before the fix it changed the emitted +/// link score. +/// +/// `indexed_name` only varies the subscripted variable's NAME. Both iterations +/// exercise the SAME path -- an ordinary arrayed variable subscripted directly -- +/// and that is deliberate, because it is the only path the fix reaches. +/// +/// **The `LOOKUP` table-index path is NOT covered here, and not by anything +/// else, because the fix does not reach it.** A graphical-function table holder +/// is by construction absent from `IteratedDimCtx::dep_dims` (GH #606 keeps it +/// off the dependency graph), so `ltm_augment::axis_dim_at` can never resolve its +/// axis and `freeze_lookup_table_indices` passes a literal `None`. GH #984's +/// defect still reproduces there on a model that compiles with zero diagnostics; +/// the reproduction is recorded on GH #984. Writing a `LOOKUP` fixture here would +/// pin the CURRENT (wrong) behaviour or fail -- neither is what this test is for. +/// +/// An earlier revision of this rustdoc claimed the two iterations covered a +/// `LOOKUP` holder and an ordinary variable. They never did: the fixture has no +/// `LOOKUP`, no `GraphicalFunction`, and no lookup-only aux. That false claim is +/// exactly how the gap shipped -- the reviewer's fixture was a `LOOKUP`, the +/// reproduction was an ordinary subscript, a different guard fired, and nobody +/// noticed. +fn colliding_index_name_model(declare_bucket: bool, second_name: bool) -> datamodel::Project { + let mut p = TestProject::new("colliding_index") + .with_sim_time(0.0, 6.0, 1.0) + .named_dimension("Region", &["nyc", "boston", "la"]) + .named_dimension("Slot", &["s1", "s2"]); + if declare_bucket { + // Declared, never referenced. `ctr` collides with the model variable. + p = p.named_dimension("Bucket", &["ctr", "spare"]); + } + let indexed = if second_name { "gtab" } else { "q" }; + p.aux("tick", "1", None) + .stock("counter", "0", &["tick"], &[], None) + .aux("drive", "1 + counter", None) + // 1, 2, 1, 2, ... -- a genuine runtime index. + .aux("ctr", "1 + (INT(counter) MOD 2)", None) + .array_with_ranges( + &format!("{indexed}[Slot]"), + vec![("s1", "1 * drive"), ("s2", "10 * drive")], + ) + .array_flow_with_ranges( + "share[Region]", + vec![ + ("nyc", "pop[nyc] * 0.01"), + ("boston", &format!("{indexed}[ctr] * 0.002 + 0 * ctr")), + ("la", "pop[la] * 0.03"), + ], + ) + .array_flow_with_ranges( + "inflow[Region]", + vec![ + ("nyc", "share[nyc]"), + ("boston", "share[boston]"), + ("la", "share[la]"), + ], + ) + .array_stock("pop[Region]", "10", &["inflow"], &[], None) + .build_datamodel() +} + +/// The `boston` arm of the `pop[nyc] -> share` link score, plus the model's +/// diagnostics -- so a "the two agree" assertion can never be two compile +/// failures agreeing. +fn colliding_index_boston_arm(project: &datamodel::Project) -> (String, usize) { + let db = SimlinDb::default(); + let sync = sync_from_datamodel(&db, project); + let diags = crate::db::collect_all_diagnostics(&db, sync.project); + let ltm = crate::db::model_ltm_variables(&db, sync.models["main"].source, sync.project); + let arm = ltm + .vars + .iter() + .find(|v| { + v.name.contains("link_score") + && v.name.contains("pop[nyc]") + && v.name.ends_with("share") + }) + .map(|v| match &v.equation { + crate::db::LtmEquation::Arrayed { elements, .. } => elements + .iter() + .find(|(e, _)| e == "boston") + .map(|(_, arm)| arm.text.clone()) + .unwrap_or_else(|| panic!("no boston arm in {:?}", elements)), + other => panic!("expected an arrayed score, got {other:?}"), + }) + .expect("pop[nyc]->share link score"); + (arm, diags.len()) +} + +#[test] +fn a_colliding_index_name_is_resolved_against_the_axis_it_indexes() { + for second_name in [true, false] { + let (with_bucket, with_diags) = + colliding_index_boston_arm(&colliding_index_name_model(true, second_name)); + let (without_bucket, without_diags) = + colliding_index_boston_arm(&colliding_index_name_model(false, second_name)); + let label = if second_name { "gtab" } else { "q" }; + + // Both models compile cleanly: the equality below is two real scores + // agreeing, not two failures. + assert_eq!( + (with_diags, without_diags), + (0, 0), + "{label}: both models must compile with no diagnostics" + ); + + // The property, stated the way a modeller would hit it: declaring a + // dimension that no equation references cannot change a link score. + assert_eq!( + with_bucket, without_bucket, + "{label}: declaring an unreferenced dimension whose element name \ + collides with a model variable changed the emitted partial" + ); + + // `ctr` is a runtime read on a `Slot` axis that declares no element + // `ctr`, so the ceteris-paribus wrap freezes it. (The + // `PREVIOUS(ctr, ctr)` spelling is GH #975's index-position initial + // value.) + assert!( + with_bucket.contains("PREVIOUS(ctr, ctr)"), + "{label}: the index must be frozen, not left live; got: {with_bucket}" + ); + // The pre-fix failure mode, pinned by name so a regression cannot pass + // by merely differing: the index was rewritten to `bucket·ctr`, an + // element of a dimension `q`/`gtab` is not declared over -- which still + // compiles and reads a different slot than the anchor did. + assert!( + !with_bucket.contains("bucket\u{B7}ctr"), + "{label}: the index must not be qualified onto an unrelated \ + dimension; got: {with_bucket}" + ); + + // The same property on the SIMULATED series, which is what a + // practitioner sees. Text equality already implies it, but the emitted + // text is an intermediate: this is the assertion that would survive a + // rewrite of how the partial is spelled. + assert_eq!( + colliding_index_boston_series(&colliding_index_name_model(true, second_name)), + colliding_index_boston_series(&colliding_index_name_model(false, second_name)), + "{label}: declaring an unreferenced dimension changed the link score" + ); + } +} + +/// The simulated `boston` slot of the `pop[nyc] -> share` link score. +/// +/// NOTE what this deliberately does NOT assert: that the series is ZERO. +/// `share[boston]` has no causal dependence on `pop[nyc]`, so a fully +/// ceteris-paribus partial would be identically zero -- and it is not; it runs +/// -1.06 / +0.73 / -1.03 / +0.82 on this fixture. That residual is a SEPARATE +/// defect from the one above and predates this branch: an index frozen inside an +/// already-frozen head is DOUBLE-lagged (the partial reads `q` at `t-1` indexed +/// by `ctr` at `t-2`, where the anchor `PREVIOUS(share)` used `ctr` at `t-1`). +/// The current behaviour is PINNED but has never been ADJUDICATED, and the +/// distinction matters for whoever picks this up. +/// `db::ltm_char_tests::per_element_dynamic_index_scores_preserve_head_lag` and +/// three siblings do freeze the lag, but that pin was written to catch a blanket +/// skip of the whole index pass and merely uses the lag as its discriminator -- +/// it argues nowhere that reading the index at `t-2` against an anchor that read +/// it at `t-1` is right. The codebase's own position is the opposite: +/// `wrap_index_non_matching_in_previous`'s GH #759 comment calls reading an index +/// two steps back "semantically wrong for a genuinely-dynamic index". +/// +/// Deferred because it is a SEMANTICS question -- what ceteris paribus means for +/// an index read under a freeze -- and it interacts with GH #975's own head-lag +/// pin, not because the current answer has been settled. A crude probe +/// (`if frozen { return index; }`, which disables the entire index pass, not just +/// the re-freeze) takes this fixture to exactly 0 and reds 5 tests; that is an +/// UPPER BOUND on the cost of the narrow change, not a measurement of it. +fn colliding_index_boston_series(project: &datamodel::Project) -> Vec { + let mut db = SimlinDb::default(); + let sync = sync_from_datamodel(&db, project); + use salsa::Setter; + sync.project.set_ltm_enabled(&mut db).to(true); + let sync = sync_from_datamodel(&db, project); + sync.project.set_ltm_enabled(&mut db).to(true); + let compiled = crate::db::compile_project_incremental(&db, sync.project, "main") + .expect("the fixture must compile"); + let offsets = compiled.offsets.clone(); + let mut vm = crate::vm::Vm::new(compiled).expect("vm"); + vm.run_to_end().expect("run"); + let results = vm.into_results(); + let name = offsets + .keys() + .map(|k| k.as_str().to_string()) + .find(|k| k.contains("link_score") && k.contains("pop[nyc]") && k.ends_with("share")) + .expect("the pop[nyc]->share link score must be emitted"); + // `+ 1` is the `boston` slot: `Region = [nyc, boston, la]`, laid out in + // declaration order. + let base = offsets[&crate::common::Ident::new(&name)] + 1; + // Compared by BIT PATTERN, so a difference cannot hide in a rounding + // tolerance -- the claim is that the two models produce the same score, not + // a similar one. + (0..results.step_count) + .map(|s| results.data[s * results.step_size + base].to_bits()) + .collect() +} + +#[test] +fn an_index_naming_the_axis_own_element_stays_a_static_selector() { + // The control that keeps the fix from being "freeze every bare index": + // `s1` IS an element of `gtab`'s own `Slot` axis, so it is a selector and + // must stay unwrapped (and qualified onto its own dimension). + let project = TestProject::new("axis_element_index") + .named_dimension("Region", &["nyc", "boston", "la"]) + .named_dimension("Slot", &["s1", "s2"]) + .aux("tick", "1", None) + .stock("counter", "0", &["tick"], &[], None) + .aux("drive", "1 + counter", None) + .array_with_ranges("q[Slot]", vec![("s1", "1 * drive"), ("s2", "10 * drive")]) + .array_flow_with_ranges( + "share[Region]", + vec![ + ("nyc", "pop[nyc] * 0.01"), + ("boston", "q[s1] * 0.002"), + ("la", "pop[la] * 0.03"), + ], + ) + .array_flow_with_ranges( + "inflow[Region]", + vec![ + ("nyc", "share[nyc]"), + ("boston", "share[boston]"), + ("la", "share[la]"), + ], + ) + .array_stock("pop[Region]", "10", &["inflow"], &[], None) + .build_datamodel(); + let (arm, diags) = colliding_index_boston_arm(&project); + assert_eq!(diags, 0, "the control model must compile cleanly"); + assert!( + arm.contains("q[slot\u{B7}s1]"), + "an element of the indexed variable's OWN axis is a static selector, \ + qualified onto that axis; got: {arm}" + ); + assert!( + !arm.contains("PREVIOUS(s1"), + "a static element selector must never be frozen; got: {arm}" + ); +} diff --git a/src/simlin-engine/src/float.rs b/src/simlin-engine/src/float.rs index 40b2671ea..53f04bb67 100644 --- a/src/simlin-engine/src/float.rs +++ b/src/simlin-engine/src/float.rs @@ -39,7 +39,13 @@ //! it costs someone a debugging session that ends at our bug. That is what //! makes GH #975 (a spurious first-step NaN in a generated LTM link score) a //! real defect rather than a cosmetic one, and the reason to fix that class -//! properly rather than narrowly. +//! properly rather than narrowly. Its root cause is the shape to watch for: +//! the engine wrote an equation of its own -- the ceteris-paribus freeze of a +//! dynamic subscript index -- whose first-DT value was the language default +//! `0`, which is in range for no dimension. The fix names the un-lagged index +//! as that freeze's initial value instead +//! (`crate::ltm_augment::freeze_at_previous`), so nothing manufactures the +//! NaN in the first place. //! //! **The technical footnote**, which follows from the above rather than //! motivating it: the engine needs no machinery that treats a NaN as a value diff --git a/src/simlin-engine/src/ltm_augment.rs b/src/simlin-engine/src/ltm_augment.rs index 95ef55a29..8da4b7211 100644 --- a/src/simlin-engine/src/ltm_augment.rs +++ b/src/simlin-engine/src/ltm_augment.rs @@ -230,6 +230,10 @@ struct WrapCtx<'a> { /// occurrence's structural path (tracked as the wrap descends), not a /// re-derivation on the reparsed `Expr0`. occ: &'a OccurrenceLookup<'a>, + /// True while the descent is inside a subscript INDEX expression, which is + /// the one position where a frozen read's FIRST-DT value has to be chosen + /// rather than defaulted. See [`freeze_at_previous`] (GH #975). + in_subscript_index: bool, /// The `PerElement` row-pinning context (GH #525, T6), `Some` only for /// [`generate_per_element_link_equation`]. /// @@ -247,6 +251,14 @@ struct WrapCtx<'a> { pin: Option<&'a PerElementRefCtx<'a>>, } +/// The first-DT initial value of every `PREVIOUS` the LTM walkers SYNTHESIZE +/// ([`freeze_at_previous`], GH #975), in its own file only to keep this one +/// under the project line-count lint. +#[path = "ltm_augment_freeze.rs"] +mod freeze; + +use freeze::freeze_at_previous; + /// Append child index `i` to `path`, yielding the child node's structural path. /// The wrap's recursion mirrors `db::ltm_ir::walk_all_in_expr`'s child-index /// construction exactly, so the path at any node equals that occurrence's @@ -375,6 +387,7 @@ fn wrap_non_matching_in_previous( live_shape, live_reducer_text, other_deps, + in_subscript_index, .. } = ctx; // Track A stage 1: hold a designated hoisted reducer subexpression LIVE @@ -417,10 +430,10 @@ fn wrap_non_matching_in_previous( } expr } else { - Expr0::App(UntypedBuiltinFn("PREVIOUS".to_string(), vec![expr]), loc) + freeze_at_previous(expr, loc, in_subscript_index) } } else if other_deps.contains(&canonical) { - Expr0::App(UntypedBuiltinFn("PREVIOUS".to_string(), vec![expr]), loc) + freeze_at_previous(expr, loc, in_subscript_index) } else { expr } @@ -561,6 +574,7 @@ fn wrap_non_matching_in_previous( true, &child_path(path, i), frozen, + axis_dim_at(ctx, &canonical, i), ) }, ), @@ -580,6 +594,7 @@ fn wrap_non_matching_in_previous( false, &child_path(path, i), frozen, + axis_dim_at(ctx, &canonical, i), ) } }) @@ -627,6 +642,7 @@ fn wrap_non_matching_in_previous( skip_index_qualification, &child_path(path, i), child_frozen, + axis_dim_at(ctx, &canonical, i), ) }; let indices: Vec = match ctx.pin.filter(|_| &canonical == live_source) { @@ -646,10 +662,7 @@ fn wrap_non_matching_in_previous( }; let subscript = Expr0::Subscript(ident, indices, loc); if will_wrap { - Expr0::App( - UntypedBuiltinFn("PREVIOUS".to_string(), vec![subscript]), - loc, - ) + freeze_at_previous(subscript, loc, in_subscript_index) } else { subscript } @@ -791,7 +804,7 @@ fn wrap_non_matching_in_previous( ), None => reducer, }; - return Expr0::App(UntypedBuiltinFn("PREVIOUS".to_string(), vec![reducer]), loc); + return freeze_at_previous(reducer, loc, in_subscript_index); } let args = args .into_iter() @@ -958,6 +971,12 @@ fn freeze_lookup_table_indices( true, &child_path(path, i), frozen, + // A literal `None`, and a KNOWN GAP: a table holder is by + // construction absent from `dep_dims` (GH #606), so an + // `axis_dim_at` call here could only ever return `None` and + // would read as coverage for a path that is not fixed -- GH #984 + // still reproduces here. See `index_axis_verdict`. + None, ) }) .collect(); @@ -980,12 +999,26 @@ fn freeze_lookup_table_indices( /// unambiguous), and `PREVIOUS(dep[dim·elem])` compiles to a direct LoadPrev /// at the element's slot. /// -/// Qualification requires knowing *which* dimension the element belongs to. -/// The wrapper does not know the subscripted variable's declared dimensions, -/// so it only qualifies names that exactly one project dimension declares -/// (`dimension_uniquely_containing_element`); names shared by multiple -/// dimensions -- or shadowed cases the caller cannot distinguish -- keep the -/// conservative wrapping behavior. +/// Qualification requires knowing *which* dimension the element belongs to, and +/// this helper does not know the subscripted variable's declared dimensions: it +/// qualifies only names that exactly one PROJECT dimension declares +/// (`dimension_uniquely_containing_element`), and names shared by several +/// dimensions keep the conservative wrapping behavior. +/// +/// **That is why this is now a FALLBACK.** When the axis IS known, +/// [`index_axis_verdict`] answers the same question against the indexed +/// variable's own axis and this helper is not consulted at all. Ranging over +/// the project's dimensions instead of the axis's own elements is not merely +/// imprecise -- it disagrees with `compiler::subscript::normalize_subscripts3`, +/// so a variable colliding with an UNRELATED dimension's element name is +/// qualified onto that dimension and the frozen read lands on a slot the +/// `PREVIOUS(target)` anchor never touched. +/// +/// **It still runs on two PRODUCTION paths, not only at the test entry points** +/// -- every `LOOKUP` table index and everything under +/// `generate_per_element_link_equation` -- so the defect above is present tense +/// there. [`index_axis_verdict`]'s rustdoc enumerates the three shapes and says +/// why each reaches this. fn qualify_element_index( index: &IndexExpr0, dims_ctx: Option<&crate::dimensions::DimensionsContext>, @@ -1008,6 +1041,87 @@ fn qualify_element_index( ))) } +/// What a bare-identifier index NAMES on the axis it indexes -- the engine's own +/// precedence rule, applied to the wrap. +/// +/// `axis` is the declared `Dimension` at this index's position of the +/// subscripted variable, looked up in [`axis_dim_at`]. When it is known, +/// every "is this a selector or a runtime read?" question below is answered by +/// [`crate::dimensions::resolve_axis_index_name`] -- the SAME predicate +/// `compiler::subscript::normalize_subscripts3` implements and that GH #986 +/// unified `ltm_agg::classify_axis_access` and +/// `post_transform::pin_dimension_name_indices` onto. The wrap was the third +/// consumer and was left on two project-wide predicates +/// (`dimension_uniquely_containing_element` and `is_element_of_any_dimension`), +/// which range over EVERY dimension in the project while the compiler ranges +/// over the axis's own declared elements. +/// +/// That disagreement was a wrong NUMBER, not a missed optimization. A model +/// variable whose canonical name happens to be an element of some UNRELATED +/// dimension (a `Scenario = [base, high, low]` beside a variable named `base`) +/// read as a runtime value to the simulation and as a static selector to the +/// wrap, so the ceteris-paribus partial left it LIVE: the "frozen" partial moved +/// with it and the link score reported real influence for an edge with no causal +/// dependence at all. Worse, `qualify_element_index` then rewrote it to +/// `otherdim·name`, naming an element of a dimension the subscripted variable is +/// not declared over -- which still compiles, and reads a different slot than the +/// anchor did. +/// +/// Returning `None` means the axis is unknown and the caller keeps the +/// pre-existing project-wide behaviour. **That is NOT merely the test entry +/// points, and it is NOT conservative**: on a collision the fallback does not +/// decline to answer, it answers wrongly. Three shapes reach it, and two are +/// production: +/// +/// - a `LOOKUP` **table index** -- always, because a graphical-function holder is +/// by construction absent from `dep_dims` (GH #606 keeps it off the dependency +/// graph). See `freeze_lookup_table_indices`, which passes `None` explicitly. +/// - **`generate_per_element_link_equation`**, which threads `dep_dims: None`. +/// - the text-in/text-out test entry points, which have no table to thread. +/// +/// So the wrap is not one consumer of this precedence rule but three call-site +/// families with two different answers. GH #986's unification reaches the +/// arrayed / scalar / A2A / stock-flow emitters and no others; the remaining two +/// are named at their call sites and are their own change. +fn index_axis_verdict( + name: &str, + axis: Option<&crate::dimensions::Dimension>, + iter_ctx: Option<&IteratedDimCtx<'_>>, +) -> Option { + let axis = axis?; + Some(crate::dimensions::resolve_axis_index_name( + name, + axis, + |dim| { + iter_ctx.is_some_and(|ic| { + ic.target_iterated_dims + .iter() + .any(|d| d.eq_ignore_ascii_case(dim)) + }) + }, + )) +} + +/// The declared dimension at index position `pos` of `ident`. +/// +/// Reads `IteratedDimCtx::dep_dims`, the target's dep -> declared-dimensions +/// table the db layer already threads here for the GH #526 other-dep +/// correspondence check. Reusing it rather than adding a second table is what +/// keeps the two questions -- "does this dep's axis correspond positionally?" +/// and "what does this index NAME on that axis?" -- answered from one source; +/// a second table built by a second route is how the wrap and the compiler came +/// to disagree in the first place. +/// +/// A dep absent from the table (a scalar, an implicit/synthetic name, or a +/// caller that threads no `iter_ctx`) yields `None`, and the caller falls back. +fn axis_dim_at<'a>( + ctx: &WrapCtx<'a>, + ident: &Ident, + pos: usize, +) -> Option<&'a crate::dimensions::Dimension> { + ctx.iter_ctx?.dep_dims?.get(ident.as_str())?.get(pos) +} + fn wrap_index_non_matching_in_previous( index: IndexExpr0, ctx: &WrapCtx<'_>, @@ -1015,6 +1129,7 @@ fn wrap_index_non_matching_in_previous( skip_element_qualification: bool, path: &[u16], frozen: bool, + axis: Option<&crate::dimensions::Dimension>, ) -> IndexExpr0 { // Only `dims_ctx` (element/dimension recognition) and `iter_ctx` (the // iterated-dim-name guard) are read directly here; the full wrap context @@ -1022,6 +1137,49 @@ fn wrap_index_non_matching_in_previous( let &WrapCtx { dims_ctx, iter_ctx, .. } = ctx; + // The axis-resolved verdict, when the axis is known. It SUPERSEDES the two + // project-wide element guards below -- see [`index_axis_verdict`]. + let axis_verdict = match &index { + IndexExpr0::Expr(Expr0::Var(name, _)) => index_axis_verdict(name.as_str(), axis, iter_ctx), + _ => None, + }; + if let Some(verdict) = &axis_verdict { + use crate::dimensions::AxisIndexName; + match verdict { + // An element THIS axis declares: a static selector. Qualify it with + // the axis's own dimension -- the only qualification that is right + // for a name several dimensions declare, and the one the pre-GH #986 + // `dimension_uniquely_containing_element` could not produce. + AxisIndexName::Element(elem) => { + if skip_element_qualification { + return index; + } + let IndexExpr0::Expr(Expr0::Var(_, loc)) = &index else { + unreachable!("axis_verdict is Some only for a bare Var index") + }; + return IndexExpr0::Expr(Expr0::Var( + RawIdent::new_from_str(&format!( + "{}\u{B7}{}", + axis.expect("verdict implies an axis").name(), + elem.as_str() + )), + *loc, + )); + } + // A dimension the enclosing equation iterates: an iteration + // reference, left verbatim exactly as the guard below does. + AxisIndexName::IteratedDim => return index, + // Neither: a runtime read. Fall through to the recursive wrap, which + // freezes it if it is a dep. This is the arm the project-wide + // predicates got wrong. + AxisIndexName::Unresolved => {} + } + } + // The project-wide fallbacks, reached only when the axis is UNKNOWN (see + // [`index_axis_verdict`]). They answer the same questions over the project's + // whole element/dimension namespace instead of the axis's own, which is + // sound only when there is no axis to consult. + let axis_unknown = axis_verdict.is_none(); // An index that unambiguously names a dimension element is an element // selector, never a causal reference: qualify it and leave it unwrapped // (GH #587). This must be checked BEFORE the recursive wrap below, which @@ -1034,7 +1192,9 @@ fn wrap_index_non_matching_in_previous( // and the lowering qualifies it consistently. The recursive wrap still runs // for a genuinely-dynamic index below, so a frozen `pop[Region, idx]` keeps // its `PREVIOUS(idx)` lag. - if !skip_element_qualification && let Some(qualified) = qualify_element_index(&index, dims_ctx) + if axis_unknown + && !skip_element_qualification + && let Some(qualified) = qualify_element_index(&index, dims_ctx) { return qualified; } @@ -1055,6 +1215,8 @@ fn wrap_index_non_matching_in_previous( // An index that names a dimension element which *cannot* be qualified // (declared by multiple dimensions at different positions, e.g. C-LEARN's // region elements) is still left verbatim rather than PREVIOUS-wrapped. + // Like `qualify_element_index` above, this ranges over the whole project and + // so runs only when the axis is unknown -- see `index_axis_verdict`. // Wrapping it would make the subscript dynamic (`dep[PREVIOUS(elem)]`), // forcing a synthesized helper aux per call site -- the dominant residual // helper source on large arrayed models (GH #654) -- and is also @@ -1063,7 +1225,8 @@ fn wrap_index_non_matching_in_previous( // a non-shadowed element compiles to a static subscript (direct // LoadPrev), a genuinely-dynamic index still synthesizes its helper // there, with single-lag semantics. - if let IndexExpr0::Expr(Expr0::Var(name, _)) = &index + if axis_unknown + && let IndexExpr0::Expr(Expr0::Var(name, _)) = &index && let Some(ctx) = dims_ctx && ctx.is_element_of_any_dimension(&crate::common::CanonicalElementName::from_raw( &canonicalize(name.as_str()), @@ -1083,7 +1246,7 @@ fn wrap_index_non_matching_in_previous( // element downstream, exactly as in the target's own equation. The // `iter_ctx` leg covers callers without a project dims context (the // iterated/source dims are dimension names by construction). - if let IndexExpr0::Expr(Expr0::Var(name, _)) = &index { + if axis_unknown && let IndexExpr0::Expr(Expr0::Var(name, _)) = &index { let canonical = canonicalize(name.as_str()); let names_project_dim = dims_ctx.is_some_and(|ctx| ctx.is_dimension_name(canonical.as_ref())); @@ -1104,6 +1267,15 @@ fn wrap_index_non_matching_in_previous( // `other_dep_mismatch` doom DOES propagate: an index-nested mismatched // collapse dooms the changed-first partial just the same. let mut idx_out = WrapOutcome::default(); + // Everything below here contributes to an INDEX value, so every freeze + // performed under it takes the un-lagged operand as its first-DT initial + // value rather than the desugared `0`, which is out of range for a 1-based + // subscript (GH #975; see [`freeze_at_previous`]). + let index_ctx = WrapCtx { + in_subscript_index: true, + ..*ctx + }; + let ctx = &index_ctx; // The index expression is at `path` (the walk pushes the index position // before descending); a `Range`'s two operands are children 0 and 1 of // that, mirroring `walk_all_in_expr`. @@ -1414,6 +1586,8 @@ fn wrap_changed_first_ast( iter_ctx, dims_ctx, occ, + // The walk starts at the slot's root expression, never inside an index. + in_subscript_index: false, pin, }; let mut out = WrapOutcome::default(); @@ -1951,13 +2125,21 @@ fn shaped_guard_form_text( /// `arr[target + 1]` style index reference is frozen too; the outer /// subscripted variable itself is wrapped only when it names `target` /// (defensive -- the feeder this is used for is scalar and so is always a -/// bare `Var` reference). -fn wrap_matching_in_previous(expr: Expr0, target: &Ident) -> Expr0 { +/// bare `Var` reference). An index-position freeze takes the un-lagged operand +/// as its first-DT initial value, exactly as the changed-first walker does +/// ([`freeze_at_previous`], GH #975) -- `in_subscript_index` carries that +/// position down the descent. +fn wrap_matching_in_previous( + expr: Expr0, + target: &Ident, + in_subscript_index: bool, +) -> Expr0 { + let recurse = |e: Expr0| wrap_matching_in_previous(e, target, in_subscript_index); match expr { Expr0::Const(..) => expr, Expr0::Var(ref ident, loc) => { if &Ident::::new(ident.as_str()) == target { - Expr0::App(UntypedBuiltinFn("PREVIOUS".to_string(), vec![expr]), loc) + freeze_at_previous(expr, loc, in_subscript_index) } else { expr } @@ -1966,16 +2148,15 @@ fn wrap_matching_in_previous(expr: Expr0, target: &Ident) -> Expr0 { let indices: Vec = indices .into_iter() .map(|idx| match idx { - IndexExpr0::Expr(e) => IndexExpr0::Expr(wrap_matching_in_previous(e, target)), + IndexExpr0::Expr(e) => { + IndexExpr0::Expr(wrap_matching_in_previous(e, target, true)) + } other => other, }) .collect(); let subscript = Expr0::Subscript(ident.clone(), indices, loc); if &Ident::::new(ident.as_str()) == target { - Expr0::App( - UntypedBuiltinFn("PREVIOUS".to_string(), vec![subscript]), - loc, - ) + freeze_at_previous(subscript, loc, in_subscript_index) } else { subscript } @@ -1985,25 +2166,17 @@ fn wrap_matching_in_previous(expr: Expr0, target: &Ident) -> Expr0 { if name.eq_ignore_ascii_case("previous") || name.eq_ignore_ascii_case("init") { return Expr0::App(UntypedBuiltinFn(name, args), loc); } - let args = args - .into_iter() - .map(|a| wrap_matching_in_previous(a, target)) - .collect(); + let args = args.into_iter().map(recurse).collect(); Expr0::App(UntypedBuiltinFn(name, args), loc) } - Expr0::Op1(op, arg, loc) => { - Expr0::Op1(op, Box::new(wrap_matching_in_previous(*arg, target)), loc) + Expr0::Op1(op, arg, loc) => Expr0::Op1(op, Box::new(recurse(*arg)), loc), + Expr0::Op2(op, l, r, loc) => { + Expr0::Op2(op, Box::new(recurse(*l)), Box::new(recurse(*r)), loc) } - Expr0::Op2(op, l, r, loc) => Expr0::Op2( - op, - Box::new(wrap_matching_in_previous(*l, target)), - Box::new(wrap_matching_in_previous(*r, target)), - loc, - ), Expr0::If(c, t, f, loc) => Expr0::If( - Box::new(wrap_matching_in_previous(*c, target)), - Box::new(wrap_matching_in_previous(*t, target)), - Box::new(wrap_matching_in_previous(*f, target)), + Box::new(recurse(*c)), + Box::new(recurse(*t)), + Box::new(recurse(*f)), loc, ), } @@ -2068,7 +2241,7 @@ pub(crate) fn generate_scalar_feeder_to_agg_equation( return Err(PartialEquationError::new(agg_equation_text)); }; let feeder_ident = Ident::::new(feeder); - let frozen = print_eqn(&wrap_matching_in_previous(ast, &feeder_ident)); + let frozen = print_eqn(&wrap_matching_in_previous(ast, &feeder_ident, false)); let frozen = match gf_table_ref { Some(table_ref) => format!("LOOKUP({table_ref}, {frozen})"), None => frozen, @@ -2139,7 +2312,11 @@ pub(crate) fn generate_iterated_feeder_to_agg_equation( let pinned = pin_iterated_dim_indices(ast, iterated_dims, slot_parts_qualified) .ok_or_else(|| PartialEquationError::unfreezable(agg_equation_text))?; let feeder_ident = Ident::::new(feeder); - let frozen = print_eqn(&wrap_matching_in_previous(pinned.clone(), &feeder_ident)); + let frozen = print_eqn(&wrap_matching_in_previous( + pinned.clone(), + &feeder_ident, + false, + )); if frozen == print_eqn(&pinned) { // No feeder occurrence was frozen: the "frozen" evaluation would // equal the agg slot itself and the score a silent constant 0. @@ -2628,9 +2805,15 @@ pub(crate) fn generate_per_element_link_equation( let iter_ctx = IteratedDimCtx { source_dim_names: &source_dim_names, target_iterated_dims, - // The `PerElement` live shape suppresses the GH #526 other-dep collapse - // entirely (see `wrap_non_matching_in_previous`), so the verdict's - // `dep_dims` are never consulted; none to thread. + // KNOWN GAP, and the comment this replaces is why. It read "the + // `PerElement` live shape suppresses the GH #526 other-dep collapse + // entirely, so the verdict's `dep_dims` are never consulted; none to + // thread" -- true when `other_dep_verdict` was the ONLY consumer. + // `axis_dim_at` is a second one, so `None` here leaves every subscript + // index under this emitter on the project-wide fallbacks: a silent wrong + // number on a collision, reproduced on a compiling model. See + // `index_axis_verdict`; threading it is its own change, because the two + // consumers want different tables. dep_dims: None, }; let ref_ctx = PerElementRefCtx { diff --git a/src/simlin-engine/src/ltm_augment_freeze.rs b/src/simlin-engine/src/ltm_augment_freeze.rs new file mode 100644 index 000000000..c8fdbc08e --- /dev/null +++ b/src/simlin-engine/src/ltm_augment_freeze.rs @@ -0,0 +1,66 @@ +// Copyright 2026 The Simlin Authors. All rights reserved. +// Use of this source code is governed by the Apache License, +// Version 2.0, that can be found in the LICENSE file. + +//! The first-DT initial value of every `PREVIOUS` the LTM ceteris-paribus +//! walkers SYNTHESIZE (GH #975). +//! +//! One rule, consumed by the two walkers that descend into subscript indices +//! (`wrap_non_matching_in_previous` and `wrap_matching_in_previous`); the other +//! two document that they never do. In its own file only to keep +//! `ltm_augment.rs` under the project line-count lint; it is `#[path]`-mounted +//! as a child module, so callers still name it `crate::ltm_augment::*`. + +use crate::ast::Expr0; +use crate::builtins::UntypedBuiltinFn; + +/// Build the `PREVIOUS` call an LTM ceteris-paribus walker freezes `expr` with, +/// choosing the FIRST-DT initial value from the position `expr` occupies. +/// +/// The XMILE spec (`docs/reference/xmile-v1.0.html` §3.5.6) defines `PREVIOUS` +/// as taking a *variable and initial value expression*, returning "the value of +/// price in the last DT, **or zero in the first DT**" for `PREVIOUS(price, 0)`. +/// The unary spelling these walkers used to emit is desugared to `PREVIOUS(x, 0)` +/// by `builtins_visitor`, so every frozen read used `0` as its first-DT value. +/// +/// That is a sound default for a *value* position and a broken one for a +/// subscript INDEX: subscripts are 1-based, so `0` is out of range for every +/// dimension, and the read yields NaN. The NaN is not confined to the first DT. +/// A frozen dynamic index sits inside an outer freeze (`PREVIOUS(pop[r, +/// PREVIOUS(idx)])`), whose argument `builtins_visitor::make_temp_arg` hoists +/// into a capture-helper aux -- so the helper evaluates to NaN at t=0 and the +/// outer `PREVIOUS` serves that NaN as the score's FIRST LIVE step (GH #975, +/// observed as `0, NaN, 0.4974...`). A NaN the engine manufactures is +/// indistinguishable on a graph from the modeller's own division by zero, which +/// is what makes this a defect rather than a cosmetic wart (see +/// [`crate::float`]'s module docs). +/// +/// In index position the un-lagged index is the only well-defined answer -- at +/// the first DT "the index one step ago" IS the current index -- and it is in +/// range by construction, since it is the index the target's own equation reads +/// at that step. So the operand doubles as its own initial-value expression. +/// Passing it explicitly also *strengthens* the runlist: `classify_dependencies` +/// walks a `PREVIOUS` fallback with `in_init` (not `in_previous`) set, so the +/// index's identifiers leave `previous_only` and the same-step ordering edge the +/// fallback needs is created rather than filtered out of `dt_deps`. +/// +/// Value positions keep the bare unary spelling, and deliberately -- but the +/// reason is that `0` is a VALID value, not that it is unobservable. The guard +/// form's `if (TIME = INITIAL_TIME) then 0` arm does own the score's own first +/// step, but a value-position freeze that lands inside a `make_temp_arg` helper +/// is read at step 1 by exactly the route this issue is about. What makes it +/// benign there is that `0` is in range for a value where it is out of range for +/// a 1-based subscript, and that it is the spec's own answer for the +/// doubly-lagged read such a helper performs. +pub(super) fn freeze_at_previous( + expr: Expr0, + loc: crate::builtins::Loc, + in_subscript_index: bool, +) -> Expr0 { + let args = if in_subscript_index { + vec![expr.clone(), expr] + } else { + vec![expr] + }; + Expr0::App(UntypedBuiltinFn("PREVIOUS".to_string(), args), loc) +} diff --git a/src/simlin-engine/src/ltm_augment_pin_tests.rs b/src/simlin-engine/src/ltm_augment_pin_tests.rs index c13d36247..085a720b8 100644 --- a/src/simlin-engine/src/ltm_augment_pin_tests.rs +++ b/src/simlin-engine/src/ltm_augment_pin_tests.rs @@ -818,18 +818,19 @@ fn per_element_pin_freezes_a_runtime_table_arg_index() { ( "a variable index -- the simplest runtime element read", "pop[Region, idx]", - "lookup(pop[region\u{B7}boston, PREVIOUS(idx)]", + "lookup(pop[region\u{B7}boston, PREVIOUS(idx, idx)]", ), ( "a compound index: the freeze reaches the read, not the whole expression", "pop[Region, idx + 1]", - "lookup(pop[region\u{B7}boston, PREVIOUS(idx) + 1]", + "lookup(pop[region\u{B7}boston, PREVIOUS(idx, idx) + 1]", ), ( "a nested source read selecting the element at runtime", "pop[Region, pop[Region, old]]", "lookup(pop[region\u{B7}boston, \ - PREVIOUS(pop[region\u{B7}boston, age\u{B7}old])]", + PREVIOUS(pop[region\u{B7}boston, age\u{B7}old], \ + pop[region\u{B7}boston, age\u{B7}old])]", ), ]; @@ -1302,13 +1303,13 @@ fn per_element_pin_index_verdict_enumeration() { ( "an undeclared bare name -- a variable selecting the element", "pop[Region, idx]", - Some("pop[region\u{B7}boston, PREVIOUS(idx)]"), + Some("pop[region\u{B7}boston, PREVIOUS(idx, idx)]"), Some("pop[region\u{B7}boston, idx]"), ), ( "an arithmetic index expression", "pop[Region, idx + 1]", - Some("pop[region\u{B7}boston, PREVIOUS(idx) + 1]"), + Some("pop[region\u{B7}boston, PREVIOUS(idx, idx) + 1]"), Some("pop[region\u{B7}boston, idx + 1]"), ), ( @@ -1316,7 +1317,8 @@ fn per_element_pin_index_verdict_enumeration() { "pop[Region, pop[Region, young]]", Some( "pop[region\u{B7}boston, \ - PREVIOUS(pop[region\u{B7}boston, age\u{B7}young])]", + PREVIOUS(pop[region\u{B7}boston, age\u{B7}young], \ + pop[region\u{B7}boston, age\u{B7}young])]", ), Some("pop[region\u{B7}boston, pop[region\u{B7}boston, age\u{B7}young]]"), ), diff --git a/src/simlin-engine/src/ltm_augment_tests.rs b/src/simlin-engine/src/ltm_augment_tests.rs index 3d4aad2f2..c8c9c59d5 100644 --- a/src/simlin-engine/src/ltm_augment_tests.rs +++ b/src/simlin-engine/src/ltm_augment_tests.rs @@ -1997,9 +1997,11 @@ fn test_partial_equation_lookup_table_index_is_frozen() { "g[nyc]", ), ( + // The freeze is in INDEX position, so it names the un-lagged index + // as its first-DT initial value (GH #975). "a runtime index read -- frozen, head still bare", "g[idx]", - "g[PREVIOUS(idx)]", + "g[PREVIOUS(idx, idx)]", ), ] { for func in ["lookup", "lookup_forward", "lookup_backward"] { @@ -2795,12 +2797,15 @@ fn partial_equation_dynamic_index_wraps_inner_deps() { build_partial_equation_shaped("arr[idx + helper]", &deps, &live, &shape, &dims, None, None) .unwrap(); + // Both freezes sit in INDEX position, so each names its own un-lagged + // operand as its first-DT initial value (GH #975) rather than defaulting to + // the desugared `0`, which is out of range for a 1-based subscript. assert!( - partial.contains("PREVIOUS(idx)"), + partial.contains("PREVIOUS(idx, idx)"), "idx must be wrapped in PREVIOUS for ceteris-paribus; got: {partial}", ); assert!( - partial.contains("PREVIOUS(helper)"), + partial.contains("PREVIOUS(helper, helper)"), "helper must be wrapped in PREVIOUS for ceteris-paribus; got: {partial}", ); // The outer arr[...] reference must stay live (no PREVIOUS wrap @@ -4690,7 +4695,7 @@ fn test_wrap_matching_in_previous_skips_already_lagged() { }) .unwrap() .unwrap(); - let wrapped = wrap_matching_in_previous(ast, &Ident::::new("scale")); + let wrapped = wrap_matching_in_previous(ast, &Ident::::new("scale"), false); let text = print_eqn(&wrapped); // (The parse/print roundtrip lowercases the pre-existing `PREVIOUS` call // name; the newly-inserted wrappers keep the uppercase spelling. Both @@ -4701,6 +4706,27 @@ fn test_wrap_matching_in_previous_skips_already_lagged() { ); } +/// GH #975, `wrap_matching_in_previous`'s half of the index-position rule: a +/// freeze that lands in a subscript INDEX carries the un-lagged index as its +/// first-DT initial value, while a freeze in a VALUE position keeps the bare +/// unary spelling (desugared to a `0` initial, which is only sound for a value). +/// +/// One fixture reaches both positions -- `arr[scale]` (index) and `scale * 2` +/// (value) -- so the test cannot pass by treating every freeze alike. +#[test] +fn test_wrap_matching_in_previous_seeds_index_freezes_with_the_unlagged_index() { + let ast = Expr0::new("arr[scale] + scale * 2", crate::lexer::LexerType::Equation) + .unwrap() + .unwrap(); + let wrapped = wrap_matching_in_previous(ast, &Ident::::new("scale"), false); + assert_eq!( + print_eqn(&wrapped), + "arr[PREVIOUS(scale, scale)] + PREVIOUS(scale) * 2", + "an index-position freeze names its own initial value; a value-position \ + freeze keeps the unary spelling" + ); +} + /// GH #744 review I1: a FIXED-literal reference to the live source inside /// the body (`SUM(pop[*] * pop[nyc])` w.r.t. `pop`, row `nyc`) must NOT be /// row-pinned: the other co-reduced rows' bodies (`pop[i] * pop[nyc]`) also From a1c9031ab43669946210530da6d86313696c2e63 Mon Sep 17 00:00:00 2001 From: Bobby Powers Date: Tue, 28 Jul 2026 07:01:29 -0700 Subject: [PATCH 09/17] doc: a test that hand-builds its inputs proves nothing Batch 4 shipped a fix that could not fire: both its tests supplied a dependency set the extractor never generates, so they passed on an input that does not occur and the defect survived untouched. The rule sits beside the N-way-decision rule because they are the same failure wearing different clothes -- a test that reads as coverage without being coverage. --- CLAUDE.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CLAUDE.md b/CLAUDE.md index 7d1da4de3..284383c09 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -96,6 +96,8 @@ IMPORTANT: If feedback seems non-actionable, it means you need comments explaini **CRITICAL**: A test that pins one arm of an N-way decision reads exactly like a test that pins the decision. Derive the rows from the enumeration -- the variant list, the call-site list, the axis list -- and cover every arm, or state in the test which arms it does not cover and where they are covered instead. "It passes" and "it constrains the code" are different claims. +**CRITICAL**: A test that hand-builds its inputs proves nothing about the inputs production supplies. If a fixture constructs a dependency set, a scope, or a context by hand, either derive it through the same function production calls or justify in the test why the hand-built value is the one production produces. This has shipped a fix that could not fire: the fixture supplied a dependency set the extractor never generates, so the test passed on an input that does not occur and the defect it claimed to cover survived untouched. + ## Claims About Other Tools Vensim, Stella, and the XMILE specification are external systems. What one of them does is a **fact to be checked, not a premise to reason from** -- and the sources are cheap: From b283d2b771849925679867e3ffde280ad1152547 Mon Sep 17 00:00:00 2001 From: Bobby Powers Date: Tue, 28 Jul 2026 08:34:36 -0700 Subject: [PATCH 10/17] engine: warn when a variable has no usable equation A Vensim model built with the Sketch tool carries `A FUNCTION OF(...)` for every variable the modeller has drawn but not yet written a formula for. Per Vensim's documentation -- the quote already recorded in keyword_ident_tests.rs -- that construct "precludes simulation", and Vensim refuses to run the model; our MDL importer stores it as the equation text `NAN`, which we compile, simulate, and hand back as NaN. So we returned garbage where the source tool declined to answer at all, and the cost lands on the practitioner: per src/float.rs a NaN is absorbing, so what they see is a line on a graph that stops -- identically for every variable downstream of the origin -- and their next task is a hand search backward through the dependency graph. This is the case where the engine knows the origin structurally before the model runs, so a warning naming the variable replaces that whole search. Emitted engine-side rather than in the MDL reader: open_vensim has no warnings channel, while db::diagnostic's accumulator already reaches the CLI, MCP, the FFI and pysimlin -- and it covers a hand-authored XMILE NAN, which is the same claim however the variable got that way. Warning, not Error, so FormattedErrors::push cannot raise the failure flags a passing CLI run or get_errors check depends on. Two shapes are deliberately NOT reported, because for them the warning would assert something the simulation contradicts. A module input port's own equation is dead once a caller binds the port -- the compiler substitutes the binding, so the port is not NaN and neither is anything downstream -- recognized by equation_is_a_module_input_fallback. That predicate needs two clauses and the second is not redundant: can_be_module_input covers XMILE access="input", while our five SMOOTH/DELAY/TREND templates declare initial_value = NAN yet carry Compat::default(), because they detect binding with the isModuleInput() builtin instead. Since db::sync splices those templates into every project, dropping either clause misreports. Likewise an EXCEPT default is read only when has_except_default marks it live, since the MDL converter keeps a dead default as round-trip metadata and the compiler never evaluates one. The accepted cost is an under-report: an input port that no caller binds falls back to its own equation, and a NaN one now goes unwarned. The atom is decided by lexing rather than by comparing against the string "nan", so the literal's spelling stays in lexer::KEYWORDS alone -- which also makes a quoted reference to a variable named `nan` an ordinary equation rather than a missing one. An arrayed variable with only some unfilled arms is reported too, and named per arm: its elements are separate series, so an unfilled x[b] stops x[b]'s line exactly the way an unfilled scalar stops its own. It stays at most one finding per variable. Four checked-in Vensim sketch fixtures newly warn, 94 findings in total; each file's count equals its A FUNCTION OF count exactly. --- src/libsimlin/src/lib.rs | 4 + src/simlin-engine/CLAUDE.md | 4 +- src/simlin-engine/src/common.rs | 20 + src/simlin-engine/src/db/diagnostic.rs | 137 +++ src/simlin-engine/src/float.rs | 8 + src/simlin-engine/src/lib.rs | 2 + .../src/unfilled_equation_tests.rs | 784 ++++++++++++++++++ src/simlin-engine/src/variable.rs | 93 +++ 8 files changed, 1050 insertions(+), 2 deletions(-) create mode 100644 src/simlin-engine/src/unfilled_equation_tests.rs diff --git a/src/libsimlin/src/lib.rs b/src/libsimlin/src/lib.rs index 7d2dc1f6f..bc5c83d9b 100644 --- a/src/libsimlin/src/lib.rs +++ b/src/libsimlin/src/lib.rs @@ -311,6 +311,10 @@ impl From for SimlinErrorCode { // A macro body instantiating a module likewise collapses to the wire // Generic code; the engine code and the message carry the detail. engine::ErrorCode::MacroContainsModule => SimlinErrorCode::Generic, + // An unfilled equation (a variable whose equation is nothing but the + // NaN literal) likewise collapses to the wire Generic code; the + // message names the variable, which is the whole point of it. + engine::ErrorCode::UnfilledEquation => SimlinErrorCode::Generic, } } } diff --git a/src/simlin-engine/CLAUDE.md b/src/simlin-engine/CLAUDE.md index ede89369c..907dd3858 100644 --- a/src/simlin-engine/CLAUDE.md +++ b/src/simlin-engine/CLAUDE.md @@ -107,7 +107,7 @@ The primary compilation path uses salsa tracked functions for fine-grained incre - **`src/db/stages.rs`** (a `db` submodule) - The two name-keyed, PRE-LAYOUT model-compilation stages as salsa-cached `returns(ref)` queries. This is the ONLY place in the crate they are built from salsa inputs. `Project::from_salsa` used to carry a second inline copy, silently disagreeing on three fields; this body took `from_salsa`'s behaviour in all three, so retiring that copy was a deletion rather than a merge, and `from_salsa` now clones these memos. Three other `ModelStage0`/`ModelStage1` constructions remain, none of them a whole-model salsa build and each deliberate: the per-variable MINI Stage0 in `db::var_fragment::lower_var_fragment` and `db::ltm::compile` (scoped to one variable's dependencies -- pointing them here would add a project-wide dependency edge to every fragment compile), and `db::units::check_conveyor_param_units`' throwaway augmented `ModelStage1::new` over a CLONE of the cached Stage0 (see that module's header). `model_stage0(db, model, project) -> ModelStage0` parses a model's variables (through the per-variable `parse_source_variable_with_module_context` memos) plus the implicit SMOOTH/DELAY/TREND helpers those parses synthesize; `model_stage1(db, model, project) -> ModelStage1` lowers that Stage0's equations to `Expr2` via `ModelStage1::new` against a `ScopeStage0` holding the project's dimension context and the Stage0 of every model in `model_scope_models(db, model, project)` -- this model plus the transitive closure of the models its module variables instantiate. Both were previously rebuilt from scratch by each consumer -- `db::units::check_model_units` is tracked PER MODEL yet rebuilt both stages for EVERY project model on each call, making whole-project unit diagnostics quadratic in the model count (GH #966); the unit pass now READS these. Two decisions live here rather than at a call site, both currently inert in the stage VALUE and therefore pinned directly by `stages_tests`: `model_is_stdlib` marks a model implicit only when the `stdlib⁚` prefix is followed by a name in `stdlib::MODEL_NAMES` (a bare-prefix model an import produced is a user model, not a template), and `source_model_is_stdlib` is the canonicalizing handle-level wrapper `db::units`' unit-check skip gate now shares (GH #988) -- that gate used to carry its own looser bare-prefix spelling, so an imported `stdlib⁚` model silently lost unit checking while the stage query staged it as an ordinary user model, and `extra_module_idents` adds EVERY variable name of a stdlib model to its `ModuleIdentContext` so `PREVIOUS(module_input)` inside a stdlib body would rewrite through a scalar helper aux (no shipped stdlib body calls PREVIOUS/INIT, so this changes no parse today -- it exists so every path reaching a stdlib model's parse hits the same interned context and shares one set of parse memos instead of minting a second). Stage0 records duplicate-canonical-ident model errors from the memoized `db::model_duplicate_variables` groups (GH #885/#891); nothing on the unit path reads that field. `model_scope_models` is that closure, as an iterative worklist rather than a recursive tracked query -- a module cycle is a project a user can draw, and recursing a tracked query on one is salsa's unrecoverable dependency-graph panic (GH #806). Its edges come from each model's Stage0 `variables`, so they include the IMPLICIT modules builtin and macro expansion synthesized: a macro call targets the macro's own model, an ordinary user model that can be arrayed, which is why `db::project_module_graph` (explicit edges only, deliberately, to stay parse-free) is the wrong source here. A module's own IDENT is an edge too when it names a project model, because `resolve_relative` predates a module ident differing from its target's name and keys each dotted `src` component as a MODEL name; dropping that edge turns a project that lowered cleanly into one reporting `BadModuleInputSrc`. `check_model_units` reads the SAME closure for its inference map (`units_infer` consults its models map only for module targets, recursing along the same edges), so the two halves cannot disagree about what is reachable. The narrowing is what makes an unrelated model's edit invalidate neither a model's lowered stage nor its unit check, pinned by the execution counters in both directions; the whole-project map cost that, and cost the reads too. `model_scope_stage0` resolves the closure to Stage0s and REPAIRS a missing self entry (a handle that outlived its place in the project's model map), because without it `get_dimensions` returns `None` for every reference in the model and every arrayed equation lowers as though it were scalar; the Stage1 map in `check_model_units` deliberately does NOT repair it, since that absence is its signal that the handle is not the project's model. Three fixtures in `stages_tests` are what can see a narrowing go too far, each catching a different mistake: `arrayed_module_project` (`main` reduces over an ARRAYED output of its DIRECT target `sub_a` -- catches a self-only scope; before it existed, emptying the scope map entirely left every test in the crate green), `chain_project` (`main -> sub_a -> sub_c` with `main` reducing over `sub_a.sub_c.out_by_region`, where `sub_c` is reachable only transitively -- the only thing that catches a "self + direct targets" scope, so the closure must be TRANSITIVE), and `omitting_stdlib_models_from_the_lowering_scope_is_inert_today` (dropping the `stdlib⁚*` models wholesale is harmless for LOWERING, and that test asserts the precise reason -- no stdlib template is arrayed or instantiates a module -- so it fails the moment that stops being true; its module half doubles as the tripwire for the stdlib-is-a-SINK premise of `MacroRegistry::build`'s Pass 4 closure argument, so if it ever reds, the abort-class consequence is that `project_module_graph` no longer sees every cycle; the closure does not take that shortcut, and since the inference map shares it, a `stdlib⁚*`-skipping closure is no longer inert end to end -- it reds `unit_checking_test::test_smth1_unit_mismatch_initial`). Test-only thread-local execution counters (`reset_query_executions`/`query_executions`, covering both stage queries AND `check_model_units`, so one reset covers every counter a test in this area reads) let `stages_tests` count query-body entries: pointer equality of a `returns(ref)` memo cannot prove a body did not run, since salsa backdates a re-executed query whose value compares equal without moving the memo. - **`src/db/stages_tests.rs`** - Tests for the two stage queries: value oracles against the salsa-free `ModelStage0::new_in_project` (Stage0) and `ModelStage1::new` over a whole-project scope of those (Stage1), including `every_shape_project`, the combined fixture carrying every shape the deleted `Project::from_salsa` copy handled -- an implicit stdlib `SMTH1` expansion, an explicit user sub-model, a macro-marked model AND a caller of it, and duplicate canonical idents; the three Stage0 semantics decisions above; the GH #966 cost claim (a whole-project `collect_all_diagnostics` BUILDS each model's Stage0/Stage1 at most once, and only for the models something reaches -- in a fixture instantiating no stdlib template, the spliced stdlib set is never staged at all, which is the scope narrowing measured; re-reading every model's stages once per model, the pre-#966 access pattern, then rebuilds none) plus its companion `project_from_salsa_reads_the_cached_stages` (`from_salsa` demands each model's stages once through the queries and the unit pass afterwards rebuilds none -- the only evidence that distinguishes reading the memo from building an equal second copy, since the deleted build produced EQUAL values by design); and diagnostic reachability across the move (a unit warning still reaches both `check_model_units::accumulated` and `collect_all_diagnostics`, as does `project_macro_registry`'s `DuplicateMacroName`, which `check_model_units` now reaches ONLY two tracked queries deeper); and the scope narrowing itself -- the closure's shape (transitive, implicit/macro edges included, module-ident edge, finite on a cycle) and its INVALIDATION consequences in both directions, since a scope that is too wide only costs incrementality while one that is too narrow answers from a stale memo. Two of those tests demonstrate a CONSEQUENCE rather than a shape, because a name-list assertion cannot show that losing an edge would mis-lower or mis-diagnose anything. `dropping_a_macro_models_scope_edge_changes_the_lowered_value` is the macro-side twin of `arrayed_module_project`: `main` reduces over an ARRAYED macro output, so removing the macro model from the scope changes the lowered VALUE of exactly that caller, with a SCALAR-output control that changes nothing -- which pins the mechanism as dimension resolution through the macro edge rather than sensitivity to the map's contents. Its fixture guard reads the model's STAGE0 (not its scope) on purpose, so that a macro-edge-dropping narrowing reds it on the value rather than on a structural check. `a_two_hop_cross_module_unit_mismatch_is_still_reported` is the unit guard for the TRANSITIVE row of the matrix (the stdlib row already had `unit_checking_test::test_smth1_unit_mismatch_initial`): a widget/gadget contradiction that closes only after `units_infer` walks BOTH module hops, so a direct-targets-only closure drops the grandchild's constraints and the diagnostic silently disappears. Note what a scope mis-lowering can and cannot corrupt: `model_stage1`'s consumers are unit checking and the test-only `Project::from_salsa`, while simulation compiles from the per-variable fragment path's own mini Stage0s and never reads this stage. The two invalidation tests drive `sync_from_datamodel_incremental`, which REUSES the `SourceProject` handle: a value read through an older `SyncResult` after an edit is the POST-edit value, so every read is made against the sync current at that moment, and each has a control step (re-syncing the identical project must re-execute nothing) so a count is attributable to the edit. Every equality oracle here ranges over ALL the sync's models, stdlib templates included, and the one stdlib-only oracle uses the representative `smth1`. Both were restricted before GH #987/#981: `ModelStage0` derives `PartialEq` and compares parsed float constants, and every SMOOTH/DELAY/TREND template declares `initial_value = NAN`, so with a bare `f64` a stdlib stage did not compare equal even to ITSELF -- which is why the oracle covered user models only and the stdlib oracle had to use `npv`, the single template with no NaN literal. `Expr2::Const` now holds an `ast::Literal` compared by bit pattern, so those stages are ordinary values. - **`src/db/sync.rs`** (a `db` submodule) - Datamodel -> salsa-input sync: the `SyncResult`/`SyncedModel`/`SyncedVariable` handle maps, the `Clone`-able `PersistentSyncState`/`PersistentModelState`/`PersistentVariableState` snapshots threaded between sync calls (with `to_sync_result`/`to_synced_model`/`from_sync_result`), the one-shot stdlib-input builder (`build_stdlib_models`, feeding the root's `StdlibModels` cache), the fresh (`sync_from_datamodel`) and incremental (`sync_from_datamodel_incremental`) entry points + their per-variable helpers (`source_variable_from_datamodel`, `update_source_variable`), the `macro_declarations_from_datamodel` extractor, `pinned_loops_from_datamodel` (resolves a model's `loop_metadata` UIDs to canonical variable names for `SourceModel.pinned_loops`), and `expand_maps_to_chains` (the `maps_to`/`mappings` reachability closure the parser uses to size a variable's dimension dependency). -- **`src/db/diagnostic.rs`** (a `db` submodule) - Compilation diagnostics: the `CompilationDiagnostic` salsa accumulator, the typed `Diagnostic` value (`severity` field + `DiagnosticError` variants `Equation`/`Model`/`Unit`/`Assembly`), the per-model `model_all_diagnostics` query that triggers every diagnostic source (`compile_var_fragment` per variable, the unit-check pass, and -- when `ltm_enabled` -- `model_ltm_fragment_diagnostics`), the `model_duplicate_variables` query + `emit_duplicate_variable_diagnostics` (GH #885: an Error-severity `DuplicateVariable` diagnostic per group of variables whose names canonicalize to the same ident, derived from the raw `declared_variable_idents` input; `compile_project_incremental` fails hard on the same query, and `queue_compile::build_compiled` runs the identical datamodel-level check before expansion), and the drain helpers `collect_model_diagnostics` (one model) / `collect_all_diagnostics` (whole synced project). Three diagnostics deliberately bypass the accumulator: `collect_all_diagnostics` emits each directly from a memoized derivation, and their row counts DIFFER. The **macro-registry build error** (`project_macro_registry`) is at most one row per project, naming no model or variable. The **unit-definition errors** (`project_units_context_result`) are one row per declaration that failed to parse -- several are normal -- naming no model but carrying the unit's name as `variable`. The **module-reference cycle** (`project_module_graph`) is one row per MODEL that can reach a cycle, carrying that model's name and skipping its per-model passes; that one is deliberately per-model and must stay so, since a model reaching no cycle is still processed normally and an unrelated draft cycle cannot hide a valid model's diagnostics (GH #806) -- collapsing it to a single project-level row would revert that. It is emitted by `module_cycle_diagnostic`, which `collect_model_diagnostics` consults and `collect_all_diagnostics` inherits by delegating to it. That gate used to be inline in the whole-project loop only, so the per-model entry point -- equally `pub` -- drove a cyclic model's passes into `model_module_map` and salsa's dependency-graph cycle panic; one function for the gate is what keeps the two entry points from disagreeing about whether the same project panics. What the first two share is their ORIGIN, not their count: each is derived once per project and each used to be accumulated from inside its deriving query's body, which both over-reported (every model's subtree reaches the query, so the DFS found the one value once per model) and lost the diagnostic entirely once an unrelated revision bump let the DFS prune the subtree. +- **`src/db/diagnostic.rs`** (a `db` submodule) - Compilation diagnostics: the `CompilationDiagnostic` salsa accumulator, the typed `Diagnostic` value (`severity` field + `DiagnosticError` variants `Equation`/`Model`/`Unit`/`Assembly`), the per-model `model_all_diagnostics` query that triggers every diagnostic source (`compile_var_fragment` per variable, the unit-check pass, and -- when `ltm_enabled` -- `model_ltm_fragment_diagnostics`), the `model_duplicate_variables` query + `emit_duplicate_variable_diagnostics` (GH #885: an Error-severity `DuplicateVariable` diagnostic per group of variables whose names canonicalize to the same ident, derived from the raw `declared_variable_idents` input; `compile_project_incremental` fails hard on the same query, and `queue_compile::build_compiled` runs the identical datamodel-level check before expansion), `emit_unfilled_equation_warnings` (a Warning-severity `UnfilledEquation` per variable whose whole equation is the NaN literal -- the decision itself is the pure `variable::unfilled_arms`, one exhaustive match over `datamodel::Equation`'s three variants; this is where Vensim's `A FUNCTION OF(...)` sketch placeholder lands, and Vensim refuses to simulate a model containing one while we compile it and return NaN, so `src/float.rs`'s "attributing a NaN to its origin replaces the backward hunt" is the whole argument for it. The default is read only when `has_except_default` says it is LIVE, since the MDL converter keeps a dead one as round-trip metadata and the compiler skips it. MODULE INPUT PORTS are skipped -- `equation_is_a_module_input_fallback` -- because a bound port's own equation is dead (the compiler substitutes the caller's binding), so warning about it is false in every clause: the port is not NaN and nothing downstream of it is. That predicate has TWO clauses and the second is NOT redundant, checked by running with only the first: `can_be_module_input` is XMILE `access="input"`, the general rule a modeller writes; `db::stages::source_model_is_stdlib` (the shared predicate, GH #988) additionally covers our five SMOOTH/DELAY/TREND templates, which declare `initial_value = NAN` but carry `Compat::default()` because they detect binding with the `isModuleInput()` builtin instead of `access="input"` -- and `db::sync` splices them into every project, so without that clause every model ever compiled reports five findings it did not earn. Collapsing the two would mean marking the templates' ports `access="input"`, which moves module instantiation itself. Disclosed cost: an UNBOUND input port with a NaN equation really is NaN and now goes unwarned), and the drain helpers `collect_model_diagnostics` (one model) / `collect_all_diagnostics` (whole synced project). Three diagnostics deliberately bypass the accumulator: `collect_all_diagnostics` emits each directly from a memoized derivation, and their row counts DIFFER. The **macro-registry build error** (`project_macro_registry`) is at most one row per project, naming no model or variable. The **unit-definition errors** (`project_units_context_result`) are one row per declaration that failed to parse -- several are normal -- naming no model but carrying the unit's name as `variable`. The **module-reference cycle** (`project_module_graph`) is one row per MODEL that can reach a cycle, carrying that model's name and skipping its per-model passes; that one is deliberately per-model and must stay so, since a model reaching no cycle is still processed normally and an unrelated draft cycle cannot hide a valid model's diagnostics (GH #806) -- collapsing it to a single project-level row would revert that. It is emitted by `module_cycle_diagnostic`, which `collect_model_diagnostics` consults and `collect_all_diagnostics` inherits by delegating to it. That gate used to be inline in the whole-project loop only, so the per-model entry point -- equally `pub` -- drove a cyclic model's passes into `model_module_map` and salsa's dependency-graph cycle panic; one function for the gate is what keeps the two entry points from disagreeing about whether the same project panics. What the first two share is their ORIGIN, not their count: each is derived once per project and each used to be accumulated from inside its deriving query's body, which both over-reported (every model's subtree reaches the query, so the DFS found the one value once per model) and lost the diagnostic entirely once an unrelated revision bump let the DFS prune the subtree. - **`src/db/layout.rs`** (a `db` submodule) - The salsa-tracked per-model *body* layout query `compute_layout` (offsets from 0; the root's `IMPLICIT_VAR_COUNT` shift is applied later at assembly via `VariableLayout::root_shifted`). Lays out explicit variables, implicit (SMOOTH/DELAY/TREND) helpers, and -- when `ltm_enabled` -- the LTM synthetic variables and their implicit helpers. - **`src/db/fragment_compile.rs`** (a `db` submodule) - The *emission half* of per-variable compilation (the *lowering half* is the sibling `src/db/var_fragment.rs`). `compile_var_fragment` is the salsa-tracked per-variable fragment compiler; `compile_implicit_var_fragment`/`compile_implicit_var_phase_bytecodes` do the same for the implicit (SMOOTH/DELAY/TREND) helpers, sharing the `lower_implicit_var` parent->implicit->parse->lower prefix. The emission tail (`compile_phase_to_per_var_bytecodes`) lives in `src/db/assemble.rs` and is the **single fragment emission entry point** every emitter -- explicit, implicit, and both LTM ones -- goes through (GH #964). Neither half assigns offsets any more: lowering hands codegen a `Vec` whose variable references are names, so the private per-fragment layout and the reverse offset map that used to undo it are gone. Runlist membership is read through the `var_runlist_membership` projection, not the whole `ModelDepGraphResult`, so an unrelated variable add/delete/rename no longer invalidates every fragment in the model. - **`src/db/assemble.rs`** (a `db` submodule) - Module/simulation assembly: the table/metadata extraction helpers (`extract_tables_from_source_var`, `build_module_inputs`, `build_stub_variable`, `build_submodel_metadata`), the per-variable emission tail (`compile_phase_to_per_var_bytecodes`, the SINGLE fragment emission entry point since GH #964's stage 2; `fragment_emit_ctx`, which assembles the phase-invariant `compiler::ModuleCtx` its callers borrow into it; the `VarFragmentResult`/`PerVarSizes` values; and `temp_sizes_by_id`, the temp-id-ORDERED flattening of the temp-size map), the production element-graph source `var_phase_symbolic_fragment_prod` (the layout-independent `SymVarRef` substrate the cycle gate's element-order probe consumes), the resolved recurrence-SCC interleaver (`segment_member_by_element` / `combine_scc_fragment`, the per-element-granular generalization of `concatenate_fragments`), the salsa-tracked `assemble_module` / `assemble_simulation`, module-instance enumeration (`enumerate_module_instances`), and the flattened-offset map builder (`calc_flattened_offsets_incremental`). Until stage 2 there were THREE hand-copied emitters (this one plus two inline copies in `src/db/ltm/compile.rs`), each building a stand-in one-variable `compiler::Module` by struct literal and deep-cloning `offsets`/`tables`/`dimensions`/`dimensions_ctx`/`module_refs` per phase; the temp-size ordering fix below had to be applied to all three at once, which is what a mirror family costs. `module_refs` and `runlist_order` turned out to be read by no line of codegen at all, so collapsing the copies also deleted `build_caller_module_refs` -- a whole second per-variable dependency walk whose only consumer was those dead fields. `temp_sizes_by_id`'s ordering is load-bearing because `PerVarBytecodes` is a salsa-cached value with a derived `PartialEq`, so a `HashMap`-iteration-order flip made an identical fragment compare unequal, defeating backdating and making the compiled artifact irreproducible run to run, GH #595's class; `FragmentMerger::absorb` folds the entries order-independently, which is why the defect had no bytecode consequence and survived undetected. `assemble_module` injects each resolved SCC's combined per-element fragment at the first member's runlist slot (the dt fragment into flows, the init fragment as one synthetic-ident `SymbolicCompiledInitial`), preserving per-write `SymVarRef` identity so layout offsets are unchanged. `build_submodel_metadata` registers not only a sub-model's explicit/implicit source variables but -- when `ltm_enabled` -- its LTM synthetic vars (and their implicit helpers) at their `compute_layout` offsets, so a parent fragment's cross-module reference to a sub-model LTM var resolves the same way the full flattened-offset assembly does. This is what lets the exhaustive-mode input→macro link score (the composite-reference form `"{module}·$⁚ltm⁚composite⁚{port}"`) compile in the standalone fragment compiler (`compile_ltm_equation_fragment`) instead of stubbing to a constant 0 (GH #548): the composite link score through a SMOOTH/DELAY/TREND/NPV macro is the max-magnitude path score through the macro (LTM ref 6.3/6.4), computed by the sub-model's `$⁚ltm⁚composite⁚{port}` aux. @@ -196,7 +196,7 @@ The unit subsystem is partial-result throughout: a single bad declaration or one ## Utilities -- **`src/float.rs`** - Floating-point helpers. `approx_eq` is the ULP-based f64 equality used throughout. `crate::float::NA` is Vensim's `:NA:` ("missing data") sentinel: the *finite* value `-2^109` (exactly representable in f64), **NOT** IEEE NaN. It is finite by design so the Vensim existence idiom `IF THEN ELSE(x = :NA:, ...)` works (`approx_eq` matches the sentinel against itself, never against a contaminated value) and `:NA:` arithmetic stays finite rather than being poisoned by an absorbing NaN. Both `:NA:` entry points route here: the expression literal via the MDL→XMILE formatter (`src/mdl/xmile_compat.rs`) and the data-list literal via the MDL number-list parser (`src/mdl/parser.rs`, `NA_VALUE`). This is distinct from the genuine NaN the engine still produces for out-of-bounds vector reads (`vm_vector_elm_map.rs`) and empty array reducers (`vm.rs` ArrayMax/Min/Mean/Stddev). **The module docs are the landing page for "why does this codebase treat NaN the way it does"**, and the two conventions belong together: `NA` is finite and comparable ON PURPOSE, while a NaN is a propagating failure signal. What a practitioner sees is a line on a graph that stops, and their next task is provenance -- searching BACKWARD through the dependency graph, because NaN is absorbing and every variable downstream of the origin shows the identical symptom. Two engine consequences follow: attributing a NaN to its origin is high-value (wherever we know STRUCTURALLY that a variable must be NaN -- an unfilled equation being the clearest case -- a warning naming the variable replaces the whole hand search), and a NaN the ENGINE manufactures is noise in a channel practitioners already debug by hand (indistinguishable on the graph from the user's own division by zero, so it costs a debugging session that ends at our bug -- which is what makes GH #975 a real defect, not a cosmetic one; its root cause is the shape to watch for, an equation the ENGINE wrote whose first-DT value was the language default `0` in a position -- a subscript index -- where no dimension admits it, fixed by naming the un-lagged index as that freeze's initial value in `ltm_augment::freeze_at_previous`). The technical footnote follows rather than motivates: a practitioner cannot author a NaN payload (one `nan` keyword, one canonical `f64::NAN`), so the engine needs no machinery treating NaN as a structured value, and bit-pattern comparison distinguishes nothing that exists. +- **`src/float.rs`** - Floating-point helpers. `approx_eq` is the ULP-based f64 equality used throughout. `crate::float::NA` is Vensim's `:NA:` ("missing data") sentinel: the *finite* value `-2^109` (exactly representable in f64), **NOT** IEEE NaN. It is finite by design so the Vensim existence idiom `IF THEN ELSE(x = :NA:, ...)` works (`approx_eq` matches the sentinel against itself, never against a contaminated value) and `:NA:` arithmetic stays finite rather than being poisoned by an absorbing NaN. Both `:NA:` entry points route here: the expression literal via the MDL→XMILE formatter (`src/mdl/xmile_compat.rs`) and the data-list literal via the MDL number-list parser (`src/mdl/parser.rs`, `NA_VALUE`). This is distinct from the genuine NaN the engine still produces for out-of-bounds vector reads (`vm_vector_elm_map.rs`) and empty array reducers (`vm.rs` ArrayMax/Min/Mean/Stddev). **The module docs are the landing page for "why does this codebase treat NaN the way it does"**, and the two conventions belong together: `NA` is finite and comparable ON PURPOSE, while a NaN is a propagating failure signal. What a practitioner sees is a line on a graph that stops, and their next task is provenance -- searching BACKWARD through the dependency graph, because NaN is absorbing and every variable downstream of the origin shows the identical symptom. Two engine consequences follow: attributing a NaN to its origin is high-value (wherever we know STRUCTURALLY that a variable must be NaN -- an unfilled equation being the clearest case, now diagnosed as `ErrorCode::UnfilledEquation` by `db::diagnostic`'s `emit_unfilled_equation_warnings` -- a warning naming the variable replaces the whole hand search), and a NaN the ENGINE manufactures is noise in a channel practitioners already debug by hand (indistinguishable on the graph from the user's own division by zero, so it costs a debugging session that ends at our bug -- which is what makes GH #975 a real defect, not a cosmetic one; its root cause is the shape to watch for, an equation the ENGINE wrote whose first-DT value was the language default `0` in a position -- a subscript index -- where no dimension admits it, fixed by naming the un-lagged index as that freeze's initial value in `ltm_augment::freeze_at_previous`). The technical footnote follows rather than motivates: a practitioner cannot author a NaN payload (one `nan` keyword, one canonical `f64::NAN`), so the engine needs no machinery treating NaN as a structured value, and bit-pattern comparison distinguishes nothing that exists. - **`src/io.rs`** - `atomic_write(path, contents)`: writes bytes to a sibling `.new` temp file, fsyncs it, then renames over the target. Cleans up the temp file on error. Best-effort parent-directory fsync after rename for durability on power loss. Used by MCP and CLI tools that need crash-safe file output. ## Cargo features diff --git a/src/simlin-engine/src/common.rs b/src/simlin-engine/src/common.rs index cfff90128..9047b5b79 100644 --- a/src/simlin-engine/src/common.rs +++ b/src/simlin-engine/src/common.rs @@ -622,6 +622,25 @@ pub enum ErrorCode { /// module *targeting* a macro model is unaffected; only a module *inside* one /// is rejected. See `MacroRegistry::build`'s Pass 4 for the full argument. MacroContainsModule, + /// A variable's equation is nothing but the NaN literal, so the variable has + /// no usable equation and every value it produces is NaN. + /// + /// This is where Vensim's `A FUNCTION OF(...)` sketch placeholder lands: the + /// modeller drew the variable and its inputs but has not written the formula + /// yet, and our MDL importer stores that as the equation text `NAN`. Vensim's + /// own documentation says the construct "precludes simulation" -- Vensim + /// refuses to run such a model -- while we compile it, simulate it, and hand + /// back NaN. A hand-authored XMILE `NAN` reaches the same place and + /// means the same thing. + /// + /// Warning-level, not Error: the rest of the model is worth simulating, and + /// `FormattedErrors::push` counts `Error` severity only, so this must not flip + /// the failure-shaped flags. Its value is ATTRIBUTION -- see `crate::float`'s + /// module docs for why. A NaN is absorbing, so every variable downstream shows + /// the identical stopped line, and the modeller's next task is a backward hunt + /// through the dependency graph for the origin. Naming the one variable the + /// engine knows STRUCTURALLY must be NaN replaces that entire hunt. + UnfilledEquation, } impl fmt::Display for ErrorCode { @@ -709,6 +728,7 @@ impl fmt::Display for ErrorCode { ConveyorInitListUnsupported => "conveyor_init_list_unsupported", UnknownElementSubscript => "unknown_element_subscript", MacroContainsModule => "macro_contains_module", + UnfilledEquation => "unfilled_equation", }; write!(f, "{name}") diff --git a/src/simlin-engine/src/db/diagnostic.rs b/src/simlin-engine/src/db/diagnostic.rs index 46ee87cb8..c531f0abb 100644 --- a/src/simlin-engine/src/db/diagnostic.rs +++ b/src/simlin-engine/src/db/diagnostic.rs @@ -113,6 +113,10 @@ pub enum DiagnosticError { /// variable's `` entry whose subscript names no declared /// element combination is silently dropped everywhere downstream, so a /// typo'd subscript simulates plausibly-but-wrong with no signal (GH #905). +/// 4c. `emit_unfilled_equation_warnings` -- the unconditional +/// `UnfilledEquation` advisory: a variable whose equation is nothing but +/// the NaN literal has no usable equation, so it and everything downstream +/// of it simulate as NaN. /// 5. When LTM is enabled, `emit_conveyor_ltm_degraded_warnings` and /// `emit_queue_ltm_degraded_warnings` -- one `Warning` per conveyor stock /// and per queue stock in THIS model, because LTM's flow-to-stock link-score @@ -212,6 +216,12 @@ pub fn model_all_diagnostics(db: &dyn Db, model: SourceModel, project: SourcePro // the conveyor spec advisories above. emit_unknown_element_subscript_warnings(db, model, project); + // Unfilled equations: a variable whose equation is nothing but the NaN + // literal, which is where Vensim's `A FUNCTION OF(...)` placeholder lands. + // Unconditional, for the same reason as the two advisories above -- it + // describes the simulation, not an analysis overlay. + emit_unfilled_equation_warnings(db, model); + // When LTM is enabled, also trigger the LTM diagnostic pass so that // diagnostics accumulated by the LTM pipeline surface through // `collect_all_diagnostics`: the auto-flip-to-discovery warning from @@ -765,6 +775,133 @@ fn emit_unknown_element_subscript_warnings( } } +/// Is `var`'s own equation only a FALLBACK -- a stand-in for a value a calling +/// model normally supplies -- rather than the formula that produces its value? +/// +/// A module input port's equation is dead whenever a caller binds the port: the +/// compiler replaces it with the caller's expression, so the port simulates as +/// the caller's value. Warning about a NaN there is a false positive in every +/// clause -- the port is not NaN, nothing downstream of it is NaN, and nobody +/// forgot to write anything (verified on a running two-instance model: the port +/// reads 5 and its consumer 10, with no NaN in the results). +/// +/// The engine has TWO spellings of "input port" and this needs both. The second +/// is NOT subsumed by the first -- checked by running the suite with only the +/// first, which brings all five stdlib findings back: +/// +/// 1. `can_be_module_input` -- XMILE `access="input"`, the declaration a modeller +/// writes on a reusable sub-model of their own. This is the general rule, and +/// the one that matters: the `NAN`-port idiom is one we author and ship in +/// clause 2's templates, so it is the shape a modeller copies from us. +/// 2. [`source_model_is_stdlib`] -- our five SMOOTH/DELAY/TREND templates declare +/// `initial_value` as `NAN` but carry `Compat::default()`, because +/// they detect binding with the `isModuleInput(initial_value)` builtin in the +/// consuming equation instead of with `access="input"`. `db::sync` splices +/// every template into every project, so without this clause every model ever +/// compiled reports five findings it did not earn. It is the shared predicate +/// (GH #988), not a third spelling of the stdlib test. +/// +/// Collapsing the two would mean marking the templates' ports `access="input"`, +/// which changes `can_be_module_input` on shipped stdlib variables and therefore +/// module instantiation itself -- far past what a diagnostic should move. +/// +/// The cost, accepted: an input port that NO caller binds does fall back to its +/// own equation, and a NaN one really is NaN. That case now goes unwarned. +/// Recovering it needs per-instantiation binding analysis, and the rule this +/// diagnostic states is about a formula nobody wrote -- an input port's equation +/// is not where its value comes from. +fn equation_is_a_module_input_fallback( + db: &dyn Db, + model: SourceModel, + var: SourceVariable, +) -> bool { + var.can_be_module_input(db) || source_model_is_stdlib(db, model) +} + +/// Emit one Warning-severity [`crate::common::ErrorCode::UnfilledEquation`] +/// diagnostic per variable in `model` whose equation is nothing but the NaN +/// literal (`crate::variable::unfilled_arms`). +/// +/// What the shape means and why it earns a diagnostic is on the `ErrorCode` +/// variant and, one level down, on `crate::float`'s module docs. What follows is +/// only what is specific to emitting it HERE. +/// +/// Engine-side rather than reader-side on purpose: `compat::open_vensim` returns +/// a bare `Result` with no warnings channel, while this accumulator +/// already reaches every consumer (the CLI, MCP, the FFI, pysimlin). Emitting +/// here also covers a hand-authored XMILE `NAN`, which is correct -- +/// the variable has no usable equation however it got that way. +/// +/// Module input ports are skipped -- see [`equation_is_a_module_input_fallback`], +/// which is load-bearing rather than tidy. +/// +/// Warning, not Error: the model still compiles and the rest of it is worth +/// simulating. `FormattedErrors::push` counts `Error` severity only, so this +/// cannot raise `has_model_errors` / `has_variable_errors` and cannot turn a +/// passing CLI run or `get_errors` check red. +/// +/// Variables are visited in sorted-name order so accumulation is deterministic. +fn emit_unfilled_equation_warnings(db: &dyn Db, model: SourceModel) { + use crate::common::{Error, ErrorCode, ErrorKind}; + use crate::variable::{UnfilledArms, unfilled_arms}; + use salsa::Accumulator; + + let source_vars = model.variables(db); + let mut var_names: Vec<&String> = source_vars.keys().collect(); + var_names.sort_unstable(); + + let model_name = model.name(db); + for var_name in var_names { + let svar = source_vars[var_name]; + if equation_is_a_module_input_fallback(db, model, svar) { + continue; + } + let Some(unfilled) = unfilled_arms(svar.equation(db)) else { + continue; + }; + // A stock's `equation` is its INITIAL VALUE, not a formula re-evaluated + // each step, so "has no equation" reads oddly for a stock that has + // flows. The NaN claim below is unchanged either way -- a NaN initial + // poisons the integration for the whole run. + let (noun, missing) = if svar.kind(db) == SourceVariableKind::Stock { + ("stock", "no initial value") + } else { + ("variable", "no equation") + }; + let subject = match &unfilled { + UnfilledArms::Whole => format!("{noun} '{var_name}' has {missing}"), + UnfilledArms::Partial { elements, default } => { + let mut arms: Vec = elements.iter().map(|e| format!("'{e}'")).collect(); + if *default { + arms.push("every element with no equation of its own".to_string()); + } + format!( + "array {noun} '{var_name}' has {missing} for {}", + arms.join(", ") + ) + } + }; + let msg = format!( + "{subject}: a NaN literal stands where the formula belongs, so it simulates as NaN \ + -- and because NaN is absorbing, so does every variable downstream of it. Vensim's \ + Sketch tool writes 'A FUNCTION OF(...)' for an equation that has not been written \ + yet and refuses to simulate a model containing one; our importer stores that \ + placeholder as this NaN literal." + ); + CompilationDiagnostic(Diagnostic { + model: model_name.clone(), + variable: Some(var_name.clone()), + error: DiagnosticError::Model(Error::new( + ErrorKind::Model, + ErrorCode::UnfilledEquation, + Some(msg), + )), + severity: DiagnosticSeverity::Warning, + }) + .accumulate(db); + } +} + /// Format an f64 for an advisory message: up to 4 decimal places with /// trailing zeros (and a bare trailing dot) trimmed, so an accumulated-sum /// artifact like `0.7 + 0.2 + 0.4 = 1.2999999999999998` reads as "1.3" diff --git a/src/simlin-engine/src/float.rs b/src/simlin-engine/src/float.rs index 53f04bb67..c2d5cb94f 100644 --- a/src/simlin-engine/src/float.rs +++ b/src/simlin-engine/src/float.rs @@ -34,6 +34,14 @@ //! variable must evaluate to NaN -- an unfilled equation is the clearest case //! -- a warning naming the variable replaces the entire backward hunt. That //! is the argument for such diagnostics, and it is worth more than it looks. +//! The clearest case is now diagnosed: a variable whose whole equation is the +//! NaN literal gets a `Warning` naming it +//! (`crate::common::ErrorCode::UnfilledEquation`, decided by +//! `crate::variable::unfilled_arms` and emitted per model by +//! `crate::db::diagnostic`). That shape is where Vensim's `A FUNCTION OF(...)` +//! sketch placeholder lands, and Vensim itself refuses to simulate a model +//! containing one -- so without the warning we returned NaN where the source +//! tool declined to answer at all. //! 2. *A NaN the engine manufactures is noise in a channel debugged by hand.* On //! the graph it is indistinguishable from the user's own division by zero, so //! it costs someone a debugging session that ends at our bug. That is what diff --git a/src/simlin-engine/src/lib.rs b/src/simlin-engine/src/lib.rs index 9e5da3d07..cf7e3caa6 100644 --- a/src/simlin-engine/src/lib.rs +++ b/src/simlin-engine/src/lib.rs @@ -95,6 +95,8 @@ mod test_sir_xmile; #[cfg(test)] mod testutils; #[cfg(test)] +mod unfilled_equation_tests; +#[cfg(test)] mod unit_checking_test; mod units; mod units_check; diff --git a/src/simlin-engine/src/unfilled_equation_tests.rs b/src/simlin-engine/src/unfilled_equation_tests.rs new file mode 100644 index 000000000..f06834911 --- /dev/null +++ b/src/simlin-engine/src/unfilled_equation_tests.rs @@ -0,0 +1,784 @@ +// Copyright 2026 The Simlin Authors. All rights reserved. +// Use of this source code is governed by the Apache License, +// Version 2.0, that can be found in the LICENSE file. + +//! Contract for the `UnfilledEquation` advisory: a variable whose equation is +//! nothing but the NaN literal has no usable equation, and the engine says so +//! before the model is ever run. What the shape means and why it earns a +//! diagnostic is on `common::ErrorCode::UnfilledEquation` and `crate::float`. +//! +//! Two levels are pinned here, because neither implies the other: the pure +//! per-shape decision (`variable::unfilled_arms` over every `datamodel::Equation` +//! variant) and the end-to-end path from a real `.mdl`/`.xmile` file to the +//! accumulated diagnostic. The end-to-end half is the one that matters -- the +//! shape's whole reason for existing is what the MDL importer produces, and a +//! fixture that hand-built a `datamodel::Equation::Scalar("NAN")` would prove +//! nothing about that. + +use std::collections::BTreeSet; + +use crate::common::ErrorCode; +use crate::datamodel; +use crate::db::{ + Diagnostic, DiagnosticError, DiagnosticSeverity, SimlinDb, collect_all_diagnostics, + compile_project_incremental, sync_from_datamodel_incremental, +}; +use crate::variable::{UnfilledArms, is_nan_literal, unfilled_arms}; + +// --------------------------------------------------------------------------- +// The atom: is this equation text nothing but the NaN literal? +// --------------------------------------------------------------------------- + +/// Every spelling the LEXER accepts for a lone `nan` token must be recognized. +/// Case and surrounding whitespace are not the modeller's statement of intent, +/// and both spellings occur in the corpus: the MDL importer writes `NAN` for +/// `A FUNCTION OF(...)` while the MDL writer prints a `Const` NaN back as `NaN`, +/// so a round-tripped placeholder changes case (`keyword_ident_tests`). +#[test] +fn the_atom_accepts_every_lone_nan_spelling() { + for text in ["nan", "NAN", "NaN", "nAn", " nan ", "\t NAN\n"] { + assert!( + is_nan_literal(text), + "{text:?} is a lone NaN literal and must be recognized" + ); + } +} + +/// "Nothing but the NaN literal" is exactly one token. A NaN *inside* a larger +/// expression is a different claim with a different remedy -- a modeller +/// deliberately using NaN as an out-of-range sentinel (`crate::float`) -- and is +/// none of this diagnostic's business. A name that merely starts with `nan` +/// lexes as an identifier and is an ordinary reference. +/// +/// Two rows are dividends of deciding this by LEXING that a string comparison +/// would have got wrong, so they are the ones to keep if the list is ever +/// trimmed. `"nan"` QUOTED is a reference to a variable legally named `nan` +/// (`keyword_ident_tests`): the lexer yields an `Ident`, never `Token::Nan`, so +/// a real equation is not mistaken for a missing one. And the Stella +/// input-port idiom -- 31 instances in this repo -- is a brace COMMENT, which +/// lexes to zero tokens, so the shape that actually occurs in the wild produces +/// no false positive. +#[test] +fn the_atom_rejects_everything_that_is_not_a_lone_nan() { + for text in [ + "", + " ", + "3", + "nan + 0", + "0 * nan", + "(nan)", + "nancy", + "0/0", + "IF x > 0 THEN 1 ELSE nan", + "\"nan\"", + "{Enter equation for use when not hooked up to other models}", + ] { + assert!( + !is_nan_literal(text), + "{text:?} is not a lone NaN literal and must not be recognized" + ); + } +} + +// --------------------------------------------------------------------------- +// The per-shape decision, enumerated from `datamodel::Equation`'s variants +// --------------------------------------------------------------------------- + +/// The `datamodel::Equation` variant `eqn` is, as an EXHAUSTIVE match with no +/// catch-all arm: a new equation shape added to the datamodel is a compile error +/// right here, which is what makes the decision table's coverage checkable +/// rather than asserted. +fn equation_shape(eqn: &datamodel::Equation) -> &'static str { + match eqn { + datamodel::Equation::Scalar(_) => "Scalar", + datamodel::Equation::ApplyToAll(..) => "ApplyToAll", + datamodel::Equation::Arrayed(..) => "Arrayed", + } +} + +/// Counted from `datamodel::Equation`: three variants, three shapes. +const EQUATION_SHAPES: [&str; 3] = ["Scalar", "ApplyToAll", "Arrayed"]; + +fn dims() -> Vec { + vec!["items".to_string()] +} + +/// One per-element arm of an `Equation::Arrayed`: +/// `(subscript, equation, initial equation, graphical function)`. +type Arm = ( + String, + String, + Option, + Option, +); + +fn arm(subscript: &str, eqn: &str) -> Arm { + (subscript.to_string(), eqn.to_string(), None, None) +} + +/// An `Equation::Arrayed` with `default` as its EXCEPT default. `live` is the +/// `has_except_default` flag, which is what makes the default apply to elements +/// with no entry of their own -- a `Some` default with the flag clear is dead. +fn arrayed(arms: Vec, default: Option<&str>, live: bool) -> datamodel::Equation { + datamodel::Equation::Arrayed(dims(), arms, default.map(str::to_string), live) +} + +// The two axes an `Arrayed` verdict turns on, each enumerated from what +// `unfilled_arms`'s `Arrayed` body actually branches on. The decision table is +// their full product, and `the_decision_table_covers_every_cell` checks that +// mechanically -- so the row count below is DERIVED, not asserted. + +/// How many of the per-element arms are unfilled. Four states: `unfilled_arms` +/// tests `unfilled.is_empty()` and `unfilled.len() == elements.len()`, and those +/// two comparisons distinguish exactly these -- with the empty-`elements` case +/// separate because it satisfies BOTH (`0 == 0`). +const ARM_STATES: [&str; 4] = ["no-arms", "none-unfilled", "some-unfilled", "all-unfilled"]; + +/// The state of the EXCEPT default. Four states: it is dropped unless +/// `has_except_default` is set, and only then is its text read. +/// +/// A dead default whose text is FILLED is deliberately not a fifth state: the +/// flag drops the default before its text is ever looked at, so it is the same +/// state as `dead-default` by construction, not by coincidence. +const DEFAULT_STATES: [&str; 4] = [ + "no-default", + "dead-default", + "live-filled-default", + "live-unfilled-default", +]; + +/// The cell of the decision space an equation occupies, computed from the +/// equation ITSELF rather than from a row's label -- so a mislabelled row cannot +/// fake coverage. `Scalar` and `ApplyToAll` carry one arm and no default, so +/// their only axis is whether that arm is unfilled. +fn decision_cell(eqn: &datamodel::Equation) -> String { + match eqn { + datamodel::Equation::Scalar(s) | datamodel::Equation::ApplyToAll(_, s) => { + let filled = if is_nan_literal(s) { + "unfilled" + } else { + "filled" + }; + format!("{}/{filled}", equation_shape(eqn)) + } + datamodel::Equation::Arrayed(_, elements, default, live) => { + let unfilled = elements + .iter() + .filter(|(_, e, _, _)| is_nan_literal(e)) + .count(); + let arms = if elements.is_empty() { + "no-arms" + } else if unfilled == 0 { + "none-unfilled" + } else if unfilled == elements.len() { + "all-unfilled" + } else { + "some-unfilled" + }; + let default = match (default.as_deref(), live) { + (None, _) => "no-default", + (Some(_), false) => "dead-default", + (Some(d), true) if is_nan_literal(d) => "live-unfilled-default", + (Some(_), true) => "live-filled-default", + }; + format!("Arrayed/{arms}/{default}") + } + } +} + +/// Every cell the table must cover: the two single-arm shapes times their one +/// 2-state axis, plus `Arrayed`'s full `ARM_STATES` x `DEFAULT_STATES` product. +/// **2 + 2 + (4 x 4) = 20.** +fn expected_cells() -> BTreeSet { + let mut cells = BTreeSet::new(); + for shape in ["Scalar", "ApplyToAll"] { + for filled in ["unfilled", "filled"] { + cells.insert(format!("{shape}/{filled}")); + } + } + for arms in ARM_STATES { + for default in DEFAULT_STATES { + cells.insert(format!("Arrayed/{arms}/{default}")); + } + } + cells +} + +/// The decision table: one row per cell of the space enumerated above, 20 in +/// total (see [`expected_cells`] for the arithmetic). +/// +/// The previous version of this table claimed "6 reachable combinations" for +/// `Arrayed` and listed 6 rows, but it had silently split the default axis for +/// the all-arms case and collapsed it for the some-arms case. Three cells were +/// missing, including the only one that reaches the message branch joining +/// element names AND the default phrase. +#[allow(clippy::type_complexity)] +fn decision_table() -> Vec<(&'static str, datamodel::Equation, Option)> { + let whole = || Some(UnfilledArms::Whole); + let partial = |elements: &[&str], default: bool| { + Some(UnfilledArms::Partial { + elements: elements.iter().map(|e| e.to_string()).collect(), + default, + }) + }; + // Reused arm sets, so each row differs only in the axis it is exercising. + let none = || vec![arm("a", "1"), arm("b", "2")]; + let some = || vec![arm("a", "1"), arm("b", "NAN")]; + let all = || vec![arm("a", "NAN"), arm("b", "nan")]; + + vec![ + // -- Scalar / ApplyToAll: one arm, one axis. + ( + "scalar, unfilled", + datamodel::Equation::Scalar("NAN".to_string()), + whole(), + ), + ( + "scalar, filled", + datamodel::Equation::Scalar("3 * x".to_string()), + None, + ), + ( + "apply-to-all, unfilled -- every element of the array has no equation", + datamodel::Equation::ApplyToAll(dims(), "nan".to_string()), + whole(), + ), + ( + "apply-to-all, filled", + datamodel::Equation::ApplyToAll(dims(), "3 * x".to_string()), + None, + ), + // -- Arrayed, no arms at all: only a live unfilled default can make such + // a variable unfilled, and then it is unfilled everywhere. + ( + "arrayed, no arms, no default", + arrayed(vec![], None, false), + None, + ), + ( + "arrayed, no arms, dead default", + arrayed(vec![], Some("NAN"), false), + None, + ), + ( + "arrayed, no arms, live filled default", + arrayed(vec![], Some("7"), true), + None, + ), + ( + "arrayed, no arms, live unfilled default", + arrayed(vec![], Some("NAN"), true), + whole(), + ), + // -- Arrayed, no arm unfilled: only the default can be at fault. + ( + "arrayed, no arm unfilled, no default", + arrayed(none(), None, false), + None, + ), + ( + "arrayed, no arm unfilled, dead default", + arrayed(none(), Some("NAN"), false), + None, + ), + ( + "arrayed, no arm unfilled, live filled default", + arrayed(none(), Some("7"), true), + None, + ), + ( + "arrayed, no arm unfilled, live unfilled default", + arrayed(none(), Some("nan"), true), + partial(&[], true), + ), + // -- Arrayed, some arms unfilled: always Partial; the default only + // decides whether the default phrase joins the element list. + ( + "arrayed, some arms unfilled, no default", + arrayed(some(), None, false), + partial(&["b"], false), + ), + ( + "arrayed, some arms unfilled, dead default", + arrayed(some(), Some("NAN"), false), + partial(&["b"], false), + ), + ( + "arrayed, some arms unfilled, live filled default", + arrayed(some(), Some("7"), true), + partial(&["b"], false), + ), + ( + // The only cell producing a non-empty element list AND `default`. + "arrayed, some arms unfilled, live unfilled default", + arrayed(some(), Some("nan"), true), + partial(&["b"], true), + ), + // -- Arrayed, every arm unfilled: Whole unless a LIVE FILLED default + // still gives the unlisted elements a formula. + ( + "arrayed, every arm unfilled, no default", + arrayed(all(), None, false), + whole(), + ), + ( + // The dead default is not an arm, so this is the no-default cell. + "arrayed, every arm unfilled, dead default", + arrayed(all(), Some("NAN"), false), + whole(), + ), + ( + "arrayed, every arm unfilled, live filled default", + arrayed(all(), Some("7"), true), + partial(&["a", "b"], false), + ), + ( + "arrayed, every arm unfilled, live unfilled default", + arrayed(all(), Some("NAN"), true), + whole(), + ), + ] +} + +#[test] +fn the_decision_table_pins_every_row() { + for (label, eqn, expected) in decision_table() { + assert_eq!( + expected, + unfilled_arms(&eqn), + "decision-table row {label:?} ({}) disagrees", + equation_shape(&eqn) + ); + } +} + +/// The table must cover the decision space EXACTLY: every cell present, and no +/// row outside it. +/// +/// This is the check that makes the row count derived rather than asserted. +/// Set equality both ways is the point -- a subset test would let a shrinking +/// table pass, and the previous version of this file asserted only that the +/// three `Equation` variants appeared, which cannot see a missing combination +/// of the two `Arrayed` axes. Three cells were in fact missing. +/// +/// What the pieces guarantee together: `equation_shape` and `decision_cell` are +/// catch-all-free matches over `datamodel::Equation`, so a new variant is a +/// compile error in both; the author must then name its cells, and this +/// assertion fails until `expected_cells` and the table agree about them. +#[test] +fn the_decision_table_covers_every_cell() { + let covered: BTreeSet = decision_table() + .iter() + .map(|(_, eqn, _)| decision_cell(eqn)) + .collect(); + let expected = expected_cells(); + assert_eq!( + expected, covered, + "the decision table must be exactly the product of the enumerated axes" + ); + assert_eq!( + expected.len(), + decision_table().len(), + "one row per cell -- a duplicate row would hide a missing one" + ); + assert_eq!( + 20, + expected.len(), + "2 + 2 + (4 arm-states x 4 default-states)" + ); + + // The shape list is still worth pinning on its own: it is what the two + // catch-all-free matches above are enumerated FROM. + let shapes: BTreeSet<&str> = decision_table() + .iter() + .map(|(_, eqn, _)| equation_shape(eqn)) + .collect(); + assert_eq!(BTreeSet::from(EQUATION_SHAPES), shapes); +} + +// --------------------------------------------------------------------------- +// End to end: from a real file to the accumulated diagnostic +// --------------------------------------------------------------------------- + +fn diagnostics(project: &datamodel::Project) -> Vec { + let mut db = SimlinDb::default(); + let sync = sync_from_datamodel_incremental(&mut db, project, None); + collect_all_diagnostics(&db, sync.project) +} + +/// Every `UnfilledEquation` diagnostic the project reports, as +/// `(model, variable, message)`. +fn unfilled_findings(project: &datamodel::Project) -> Vec<(String, String, String)> { + diagnostics(project) + .into_iter() + .filter_map(|d| match &d.error { + DiagnosticError::Model(e) if e.code == ErrorCode::UnfilledEquation => { + assert_eq!( + DiagnosticSeverity::Warning, + d.severity, + "an unfilled equation must never be reported as an Error: the rest \ + of the model is worth simulating" + ); + Some(( + d.model.clone(), + d.variable.clone().unwrap_or_default(), + e.get_details().unwrap_or_default().to_string(), + )) + } + _ => None, + }) + .collect() +} + +fn xmile_doc(dimensions: &str, vars: &str, extra_models: &str) -> String { + format!( + r#" + +
ttt
+ + 02
1
+ + {dimensions} + {vars} + {extra_models} +"# + ) +} + +fn read_xmile(dimensions: &str, vars: &str) -> datamodel::Project { + read_xmile_with_models(dimensions, vars, "") +} + +fn read_xmile_with_models(dimensions: &str, vars: &str, extra_models: &str) -> datamodel::Project { + crate::compat::open_xmile(&mut xmile_doc(dimensions, vars, extra_models).as_bytes()) + .expect("XMILE must parse") +} + +/// Run `project`'s main model and return `var`'s final value. +fn final_value(project: &datamodel::Project, var: &str) -> f64 { + let mut vm = crate::queue_compile::build_vm(project, "main").expect("model must build"); + vm.run_to_end().expect("simulation must run"); + let results = vm.into_results(); + let offset = *results + .offsets + .get(var) + .unwrap_or_else(|| panic!("no results column for `{var}`")); + results + .iter() + .next_back() + .unwrap_or_else(|| panic!("`{var}` has an empty timeseries"))[offset] +} + +/// The MDL path, end to end: a Sketch-tool placeholder imports, compiles, and +/// produces exactly ONE warning naming the variable that has no equation. +/// +/// `marketing` is the unfilled one and `revenue` reads it, so `revenue` is NaN +/// too -- which is the entire point of attributing the origin. The count +/// assertion is what pins that: exactly one finding for two NaN series. +#[test] +fn an_mdl_a_function_of_placeholder_warns_naming_the_variable() { + let mdl = concat!( + "price = 3\n\t~\t\n\t~\t|\n\n", + "marketing = A FUNCTION OF( price )\n\t~\t\n\t~\t|\n\n", + "revenue = marketing * price\n\t~\t\n\t~\t|\n\n", + "\\\\\\---/// Sketch information - do not modify anything except names\n" + ); + let project = crate::compat::open_vensim(mdl).expect("MDL must parse"); + + // The premise: the importer really does store the placeholder as the NaN + // literal. If this stops holding, the rest of the test proves nothing. + let marketing = project.models[0] + .variables + .iter() + .find(|v| v.get_ident() == "marketing") + .expect("the placeholder variable must survive import"); + assert_eq!( + Some(&datamodel::Equation::Scalar("NAN".to_string())), + marketing.get_equation() + ); + + let findings = unfilled_findings(&project); + assert_eq!( + 1, + findings.len(), + "exactly one unfilled-equation finding, for the variable that HAS no \ + equation -- not for the variables downstream of it, which is the whole \ + reason to report it. Got: {findings:#?}" + ); + assert_eq!("marketing", findings[0].1); + assert!( + findings[0].2.contains("'marketing' has no equation"), + "the message must name the variable: {:?}", + findings[0].2 + ); + + // And the model still compiles: this is an advisory, not a rejection. + let mut db = SimlinDb::default(); + let sync = sync_from_datamodel_incremental(&mut db, &project, None); + compile_project_incremental(&db, sync.project, "main") + .expect("an unfilled equation must not block compilation"); +} + +/// The XMILE path: a hand-authored `NAN` is the same claim -- the +/// variable has no usable equation however it got that way -- so it gets the +/// same warning. This is why the diagnostic lives in the engine rather than in +/// the MDL reader. +#[test] +fn a_hand_authored_xmile_nan_equation_warns_naming_the_variable() { + let project = read_xmile( + "", + r#" + 3 + NAN + marketing * price"#, + ); + let findings = unfilled_findings(&project); + assert_eq!( + vec!["marketing".to_string()], + findings.iter().map(|f| f.1.clone()).collect::>(), + "got: {findings:#?}" + ); +} + +/// The control, carrying both things that must NOT be reported. +/// +/// `db::sync` splices all nine stdlib templates into EVERY project, so this is +/// simultaneously the tightest test of the stdlib exclusion: five of those +/// templates (`smth1`, `smth3`, `delay1`, `delay3`, `trend`) legitimately +/// declare their `initial_value` port as `NAN`, because the port's +/// value arrives from the caller. Without the exclusion this model -- and every +/// model ever compiled -- reports five findings it did not earn. +/// +/// `guard` and `poison` are the other boundary, at the level a modeller sees it: +/// a NaN inside a larger expression is a filled equation, and widening the atom +/// past a single token would report it. +/// +/// Both positions are present deliberately. A widening that drops the +/// single-token check catches whichever end it looks at: `guard` puts `NAN` +/// LAST, so it only reds a "contains a nan token anywhere" widening, while +/// `poison` puts it FIRST, which is what reds a widening that keeps only the +/// leading-token test. `poison` is contrived -- no modeller writes it -- and it +/// earns its place solely as that second tripwire. +#[test] +fn an_ordinary_model_reports_nothing() { + let project = read_xmile( + "", + r#" + 3 + price * 2 + IF revenue > 0 THEN revenue ELSE NAN + NAN + revenue"#, + ); + assert_eq!( + Vec::<(String, String, String)>::new(), + unfilled_findings(&project) + ); +} + +/// A BOUND module input port in a user sub-model must NOT be reported, and the +/// simulated numbers are why. +/// +/// The port's own equation is dead: the compiler replaces it with the caller's +/// binding, so `port` is 5 and `out` is 10 with no NaN anywhere. Every factual +/// clause of the warning would be false for this model -- the port is not NaN, +/// nothing downstream of it is NaN, and nobody forgot to write anything. The +/// value assertions are what make this a test of the CLAIM rather than of the +/// skip: if the port's equation ever did drive the result, they would fail here +/// before the finding count did. +/// +/// This is the general rule that the stdlib skip is one instance of. The idiom +/// is one WE author and ship (`stdlib/*.stmx`), so it is what a modeller +/// building a reusable sub-model copies from us. +#[test] +fn a_bound_module_input_port_is_not_reported() { + let project = read_xmile_with_models( + "", + r#" + 5 + + m.out"#, + r#" + NAN + port * 2 + "#, + ); + + assert_eq!( + Vec::<(String, String, String)>::new(), + unfilled_findings(&project), + "a bound input port's own equation is a fallback the caller overrides" + ); + assert_eq!(5.0, final_value(&project, "m·port")); + assert_eq!(10.0, final_value(&project, "m·out")); + assert_eq!(10.0, final_value(&project, "total")); +} + +/// The stdlib exclusion again, from the direction that would survive a +/// "skip models nothing instantiates" narrowing: a model that really does +/// INSTANTIATE `SMTH1` still reports nothing. The spliced template is now +/// reachable, its `initial_value = NAN` port is genuinely compiled, and it is +/// still not the user's unfilled equation. +#[test] +fn instantiating_a_stdlib_template_reports_nothing() { + let project = read_xmile( + "", + r#" + 3 + TIME + SMTH1(price, 2)"#, + ); + assert_eq!( + Vec::<(String, String, String)>::new(), + unfilled_findings(&project), + "a stdlib template's `initial_value = NAN` port is a legitimate \ + declaration, not a modeller's unfilled equation" + ); +} + +/// An arrayed variable with only SOME unfilled elements is reported, and the +/// message names them. +/// +/// The decision (`variable::unfilled_arms`): an arrayed variable's elements are +/// separate simulated series, so an unfilled `values[b]` stops `values[b]`'s +/// line exactly the way an unfilled scalar stops its own. Staying silent unless +/// EVERY arm is unfilled would go quiet precisely where the model is hardest to +/// read -- two lines fine, one stopping. It stays one finding per variable, so +/// a wholly-unfilled 500-element array is still a single message. +#[test] +fn a_partially_unfilled_array_names_its_unfilled_elements() { + let project = read_xmile( + r#""#, + r#" + + 1 + NAN + NAN + + "#, + ); + let findings = unfilled_findings(&project); + assert_eq!(1, findings.len(), "one finding per variable: {findings:#?}"); + assert_eq!("values", findings[0].1); + assert!( + findings[0].2.contains("no equation for 'b', 'c'"), + "the message must name the unfilled elements: {:?}", + findings[0].2 + ); +} + +/// The one decision-table cell that reaches the message branch joining element +/// names AND the default phrase -- `some arms unfilled` + a LIVE unfilled EXCEPT +/// default -- built from a real XMILE file rather than by hand. +/// +/// A top-level `` alongside `` entries is what sets +/// `has_except_default` in the reader (`xmile/variables.rs`), so this shape is +/// reachable and the join is production behaviour, not a formatting curiosity. +/// The review found this branch untested because the old table had collapsed +/// the default axis for the some-arms row. +#[test] +fn a_partial_array_with_an_unfilled_default_names_both() { + let project = read_xmile( + r#""#, + r#" + + NAN + 1 + NAN + + "#, + ); + let findings = unfilled_findings(&project); + assert_eq!(1, findings.len(), "one finding per variable: {findings:#?}"); + assert!( + findings[0] + .2 + .contains("no equation for 'b', every element with no equation of its own"), + "the message must name the unfilled element AND the default: {:?}", + findings[0].2 + ); +} + +/// A stock's `equation` is its INITIAL VALUE, so the message says so. "variable +/// 's' has no equation" reads as though the stock had no dynamics at all, when +/// what it is missing is where to start; the NaN claim is unchanged, because a +/// NaN initial poisons the integration for the whole run. +#[test] +fn an_unfilled_stock_is_described_as_missing_its_initial_value() { + let project = read_xmile( + "", + r#" + NANfill + 1"#, + ); + let findings = unfilled_findings(&project); + assert_eq!(1, findings.len(), "{findings:#?}"); + assert!( + findings[0] + .2 + .starts_with("stock 'level' has no initial value"), + "a stock is missing an initial value, not an equation: {:?}", + findings[0].2 + ); + assert!( + final_value(&project, "level").is_nan(), + "the NaN claim must still hold for a stock: a NaN initial poisons the \ + whole integration" + ); +} + +/// Accumulation order is by variable name. The emitter's rustdoc claims this and +/// nothing pinned it -- every other fixture here reports zero or one finding, so +/// they are all order-blind. +/// +/// Six variables, declared in reverse, deliberately: the source is a `HashMap` +/// whose iteration order is per-process random, so a three-name fixture comes +/// out sorted by chance often enough to be a useless detector (measured: it +/// caught a removed `sort_unstable` in 2 of 5 runs). Six names leave a 1-in-720 +/// accidental pass, the same reasoning `db::fragment_determinism_tests` uses for +/// its repeat count. +#[test] +fn findings_are_accumulated_in_sorted_name_order() { + let project = read_xmile( + "", + r#" + NAN + NAN + NAN + NAN + NAN + NAN"#, + ); + let names: Vec = unfilled_findings(&project) + .into_iter() + .map(|f| f.1) + .collect(); + assert_eq!( + vec!["alpha", "delta", "mike", "quebec", "victor", "zulu"], + names + ); +} + +/// The advisory must not make a project look failed. `FormattedErrors::push` +/// raises `has_model_errors` / `has_variable_errors` for `Error` severity only, +/// and those flags gate failure-shaped decisions downstream (the CLI's +/// `NotSimulatable` suppression, and through it its reporting). A new warning +/// that flipped them would turn passing workflows red on merit alone. +#[test] +fn the_warning_does_not_raise_the_failure_flags() { + use crate::errors::FormattedErrors; + + let project = read_xmile("", r#"NAN"#); + let mut formatted = FormattedErrors::default(); + for diag in &diagnostics(&project) { + formatted.push(crate::errors::format_diagnostic(diag)); + } + assert!( + !formatted.errors.is_empty(), + "the warning must reach the formatted surface" + ); + assert!( + !formatted.has_model_errors && !formatted.has_variable_errors, + "an unfilled-equation Warning must not raise the failure flags: \ + {formatted:#?}" + ); +} diff --git a/src/simlin-engine/src/variable.rs b/src/simlin-engine/src/variable.rs index cb551091c..dd80e8a2e 100644 --- a/src/simlin-engine/src/variable.rs +++ b/src/simlin-engine/src/variable.rs @@ -490,6 +490,99 @@ pub(crate) fn var_is_lookup_only(eqn: Option<&datamodel::Equation>, has_tables: } } +/// Does `equation` consist of nothing but the NaN literal? +/// +/// Decided by LEXING rather than by comparing against the string `"nan"`: the +/// spelling of the literal lives in `lexer::KEYWORDS`, and restating it here +/// would be a second copy of that table to keep in step (the same reason +/// `ast::needs_quoting` reads `lexer::is_reserved_word`, GH #976). Lexing also +/// settles case and surrounding whitespace for free -- both spellings occur in +/// practice, since the MDL importer writes `NAN` for `A FUNCTION OF(...)` while +/// the MDL writer prints a `Const` NaN back as `NaN`. +/// +/// "Nothing but" is exactly one token, so a NaN nested in a larger expression +/// (`nan + 0`, `IF x > 0 THEN 1 ELSE nan`) is NOT this -- that is a modeller +/// deliberately using NaN as a sentinel, a different claim with a different +/// remedy. A parenthesized `(nan)` is likewise outside the rule; nothing +/// produces it, and admitting it would mean deciding how far to unwrap. +pub(crate) fn is_nan_literal(equation: &str) -> bool { + use crate::lexer::{Lexer, LexerType, Token}; + let mut lexer = Lexer::new(equation, LexerType::Equation); + matches!(lexer.next(), Some(Ok((_, Token::Nan, _)))) && lexer.next().is_none() +} + +/// Which of a variable's equation arms are UNFILLED -- carry the NaN literal +/// where a formula belongs (see [`is_nan_literal`]). +/// +/// An "arm" is one whole equation: the single formula of a scalar or +/// apply-to-all variable, one `` entry of a per-element arrayed +/// variable, or an arrayed variable's EXCEPT default (the formula for every +/// element with no entry of its own). +#[derive(Debug, PartialEq, Eq)] +pub(crate) enum UnfilledArms { + /// The variable has NO equation anywhere: its scalar/apply-to-all formula is + /// the NaN literal, or every arm of its arrayed equation is (its EXCEPT + /// default included, when it has one). + Whole, + /// Only part of an arrayed variable is unfilled: these element subscripts, + /// in declaration order, plus the EXCEPT default when `default` is set. + Partial { + elements: Vec, + default: bool, + }, +} + +/// Classify `equation` by which of its arms are unfilled, or `None` when none +/// are. +/// +/// Arms rather than whole variables, deliberately. An arrayed variable's +/// elements are separate simulated series, so an unfilled `x[b]` stops `x[b]`'s +/// line on the graph exactly the way an unfilled scalar stops its own, and costs +/// the same backward hunt (`crate::float`). Reporting only the all-arms case +/// would go silent precisely where the model is most confusing -- some lines +/// fine, one stopping -- so a partially-unfilled variable is reported too, and +/// [`UnfilledArms::Partial`] names the arms so the message can point at them. +/// Either way it is at most ONE finding per variable, never one per element. +pub(crate) fn unfilled_arms(equation: &datamodel::Equation) -> Option { + use crate::datamodel::Equation; + match equation { + // One arm covering the whole variable: a scalar formula, or one formula + // applied to every element. + Equation::Scalar(s) | Equation::ApplyToAll(_, s) => { + is_nan_literal(s).then_some(UnfilledArms::Whole) + } + Equation::Arrayed(_, elements, default, has_except_default) => { + let unfilled: Vec = elements + .iter() + .filter(|(_, eqn, _, _)| is_nan_literal(eqn)) + .map(|(subscript, _, _, _)| subscript.clone()) + .collect(); + // `has_except_default` is what makes the default LIVE, so a `Some` + // default with the flag clear is not an arm at all. The pairing is + // reachable: the MDL converter keeps a default as round-trip + // metadata while setting the flag false when EXCEPT is the sole + // source of elements (`mdl::convert::variables`), and the compiler + // falls back to the default for a missing element only under the + // same flag (`compiler::expand_arrayed_with_hoisting`). Reading the + // text without the flag would report a NaN nothing ever evaluates. + let default = default.as_deref().filter(|_| *has_except_default); + let default_unfilled = default.is_some_and(is_nan_literal); + if unfilled.is_empty() && !default_unfilled { + None + } else if unfilled.len() == elements.len() && (default.is_none() || default_unfilled) { + // Nothing the variable can evaluate to is anything but NaN -- + // including the elements the default covers, if there is one. + Some(UnfilledArms::Whole) + } else { + Some(UnfilledArms::Partial { + elements: unfilled, + default: default_unfilled, + }) + } + } + } +} + #[cfg(test)] mod is_lookup_only_tests { use super::*; From ae71ed0df660480b1f25c9e26047deadeb2c293b Mon Sep 17 00:00:00 2001 From: Bobby Powers Date: Tue, 28 Jul 2026 09:10:54 -0700 Subject: [PATCH 11/17] engine: unfilled-equation verdict needs slot coverage Two corrections to the UnfilledEquation advisory added in b283d2b7, both found by review, both cases where the warning stated something the simulation contradicts. First, the verdict turned on (which arms are unfilled, what the EXCEPT default says) when it also needs to know whether the arms COVER the variable's dimension. An armless slot's value is not something the arms' text can answer, and both directions were wrong on models that run today. A NaN default that no slot can reach was reported: with arms a=1,b=2 over D=[a,b] and default NAN, expand_arrayed_with_hoisting never consults the default and the compiled model holds 1 and 2 with no NaN anywhere. And a sparse array whose listed arms were all NaN was reported as WHOLLY unfilled, though its omitted slots compile to a finite 0. unfilled_arms now takes a variable::ArmCoverage; db::diagnostic computes it, since resolving dimension names against the project is a database read and the classifier stays pure. The membership test is the compiler's own whole-string element lookup rather than the sibling advisory's union rule, because that lookup IS the fact being asked about. The two arrayed advisories now share one dimension resolver instead of two copies. Second, "NaN is absorbing, so every variable downstream is NaN too" is false, and b283d2b7 asserted it in the emitted message, in the ErrorCode doc, in src/float.rs's module documentation and in its own commit body. NaN is absorbing through ARITHMETIC; every IEEE comparison against a NaN is false, so `IF x > 0 THEN 1 ELSE 0` reading a NaN x returns a finite 0 and everything below it is finite. That is not a footnote -- it is exactly what bites during the backward search for a NaN's origin that float.rs argues is the real cost of a NaN, so the module doc now states the two consequences a modeller needs: a finite variable does not clear its inputs, and a NaN can vanish partway down a chain instead of reaching the output that would have revealed it. The message now scopes the spread to arithmetic. The decision table gains coverage as a third axis (2 + 2 + [4 x 4 x 2 - 4 impossible] = 32 cells). Its fixtures are now generated from the axis states and checked to land in the cell they are filed under, while the verdicts stay authored as flat literals -- a grouped expectation would be a second copy of the rule under test. Worth recording why neither the original 20-cell product nor the review's 9-cell analysis caught this: both enumerated the axes the code branched on, and the code branched on two because the third was never passed in. The corpus warning count is unchanged at 94 across 4 files. --- src/simlin-engine/CLAUDE.md | 2 +- src/simlin-engine/src/common.rs | 15 +- src/simlin-engine/src/db/diagnostic.rs | 113 +++-- src/simlin-engine/src/float.rs | 25 +- .../src/unfilled_equation_tests.rs | 421 ++++++++++++++---- src/simlin-engine/src/variable.rs | 72 ++- 6 files changed, 504 insertions(+), 144 deletions(-) diff --git a/src/simlin-engine/CLAUDE.md b/src/simlin-engine/CLAUDE.md index 907dd3858..7b0657aef 100644 --- a/src/simlin-engine/CLAUDE.md +++ b/src/simlin-engine/CLAUDE.md @@ -196,7 +196,7 @@ The unit subsystem is partial-result throughout: a single bad declaration or one ## Utilities -- **`src/float.rs`** - Floating-point helpers. `approx_eq` is the ULP-based f64 equality used throughout. `crate::float::NA` is Vensim's `:NA:` ("missing data") sentinel: the *finite* value `-2^109` (exactly representable in f64), **NOT** IEEE NaN. It is finite by design so the Vensim existence idiom `IF THEN ELSE(x = :NA:, ...)` works (`approx_eq` matches the sentinel against itself, never against a contaminated value) and `:NA:` arithmetic stays finite rather than being poisoned by an absorbing NaN. Both `:NA:` entry points route here: the expression literal via the MDL→XMILE formatter (`src/mdl/xmile_compat.rs`) and the data-list literal via the MDL number-list parser (`src/mdl/parser.rs`, `NA_VALUE`). This is distinct from the genuine NaN the engine still produces for out-of-bounds vector reads (`vm_vector_elm_map.rs`) and empty array reducers (`vm.rs` ArrayMax/Min/Mean/Stddev). **The module docs are the landing page for "why does this codebase treat NaN the way it does"**, and the two conventions belong together: `NA` is finite and comparable ON PURPOSE, while a NaN is a propagating failure signal. What a practitioner sees is a line on a graph that stops, and their next task is provenance -- searching BACKWARD through the dependency graph, because NaN is absorbing and every variable downstream of the origin shows the identical symptom. Two engine consequences follow: attributing a NaN to its origin is high-value (wherever we know STRUCTURALLY that a variable must be NaN -- an unfilled equation being the clearest case, now diagnosed as `ErrorCode::UnfilledEquation` by `db::diagnostic`'s `emit_unfilled_equation_warnings` -- a warning naming the variable replaces the whole hand search), and a NaN the ENGINE manufactures is noise in a channel practitioners already debug by hand (indistinguishable on the graph from the user's own division by zero, so it costs a debugging session that ends at our bug -- which is what makes GH #975 a real defect, not a cosmetic one; its root cause is the shape to watch for, an equation the ENGINE wrote whose first-DT value was the language default `0` in a position -- a subscript index -- where no dimension admits it, fixed by naming the un-lagged index as that freeze's initial value in `ltm_augment::freeze_at_previous`). The technical footnote follows rather than motivates: a practitioner cannot author a NaN payload (one `nan` keyword, one canonical `f64::NAN`), so the engine needs no machinery treating NaN as a structured value, and bit-pattern comparison distinguishes nothing that exists. +- **`src/float.rs`** - Floating-point helpers. `approx_eq` is the ULP-based f64 equality used throughout. `crate::float::NA` is Vensim's `:NA:` ("missing data") sentinel: the *finite* value `-2^109` (exactly representable in f64), **NOT** IEEE NaN. It is finite by design so the Vensim existence idiom `IF THEN ELSE(x = :NA:, ...)` works (`approx_eq` matches the sentinel against itself, never against a contaminated value) and `:NA:` arithmetic stays finite rather than being poisoned by an absorbing NaN. Both `:NA:` entry points route here: the expression literal via the MDL→XMILE formatter (`src/mdl/xmile_compat.rs`) and the data-list literal via the MDL number-list parser (`src/mdl/parser.rs`, `NA_VALUE`). This is distinct from the genuine NaN the engine still produces for out-of-bounds vector reads (`vm_vector_elm_map.rs`) and empty array reducers (`vm.rs` ArrayMax/Min/Mean/Stddev). **The module docs are the landing page for "why does this codebase treat NaN the way it does"**, and the two conventions belong together: `NA` is finite and comparable ON PURPOSE, while a NaN is a propagating failure signal. What a practitioner sees is a line on a graph that stops, and their next task is provenance -- searching BACKWARD through the dependency graph, because a NaN spreads through ARITHMETIC to whatever reads it, so a stopped line usually means the origin is upstream. The module doc states the exception on purpose, because it bites during exactly that search: NaN is absorbing in arithmetic but NOT through comparisons, every IEEE comparison against a NaN being false, so `IF x > 0 THEN 1 ELSE 0` reading a NaN returns a finite 0 and its whole downstream subtree is finite. A finite variable therefore does not clear its inputs, and no diagnostic here may claim that everything downstream of a NaN is NaN. Two engine consequences follow: attributing a NaN to its origin is high-value (wherever we know STRUCTURALLY that a variable must be NaN -- an unfilled equation being the clearest case, now diagnosed as `ErrorCode::UnfilledEquation` by `db::diagnostic`'s `emit_unfilled_equation_warnings` -- a warning naming the variable replaces the whole hand search), and a NaN the ENGINE manufactures is noise in a channel practitioners already debug by hand (indistinguishable on the graph from the user's own division by zero, so it costs a debugging session that ends at our bug -- which is what makes GH #975 a real defect, not a cosmetic one; its root cause is the shape to watch for, an equation the ENGINE wrote whose first-DT value was the language default `0` in a position -- a subscript index -- where no dimension admits it, fixed by naming the un-lagged index as that freeze's initial value in `ltm_augment::freeze_at_previous`). The technical footnote follows rather than motivates: a practitioner cannot author a NaN payload (one `nan` keyword, one canonical `f64::NAN`), so the engine needs no machinery treating NaN as a structured value, and bit-pattern comparison distinguishes nothing that exists. - **`src/io.rs`** - `atomic_write(path, contents)`: writes bytes to a sibling `.new` temp file, fsyncs it, then renames over the target. Cleans up the temp file on error. Best-effort parent-directory fsync after rename for durability on power loss. Used by MCP and CLI tools that need crash-safe file output. ## Cargo features diff --git a/src/simlin-engine/src/common.rs b/src/simlin-engine/src/common.rs index 9047b5b79..3e963c78a 100644 --- a/src/simlin-engine/src/common.rs +++ b/src/simlin-engine/src/common.rs @@ -636,10 +636,17 @@ pub enum ErrorCode { /// Warning-level, not Error: the rest of the model is worth simulating, and /// `FormattedErrors::push` counts `Error` severity only, so this must not flip /// the failure-shaped flags. Its value is ATTRIBUTION -- see `crate::float`'s - /// module docs for why. A NaN is absorbing, so every variable downstream shows - /// the identical stopped line, and the modeller's next task is a backward hunt - /// through the dependency graph for the origin. Naming the one variable the - /// engine knows STRUCTURALLY must be NaN replaces that entire hunt. + /// module docs for why. A NaN spreads through arithmetic to whatever reads it, + /// so the modeller's next task is a backward hunt through the dependency graph + /// for the origin. Naming the one variable the engine knows STRUCTURALLY must + /// be NaN replaces that entire hunt. + /// + /// The spread is through arithmetic only, which is why neither this doc nor + /// the emitted message claims every downstream variable is NaN: IEEE + /// comparisons against a NaN are false, so `IF x > 0 THEN 1 ELSE 0` reading a + /// NaN `x` returns a finite `0` and everything below it is finite too. A + /// diagnostic that asserted otherwise would send the modeller looking in the + /// wrong place. UnfilledEquation, } diff --git a/src/simlin-engine/src/db/diagnostic.rs b/src/simlin-engine/src/db/diagnostic.rs index c531f0abb..9de83302d 100644 --- a/src/simlin-engine/src/db/diagnostic.rs +++ b/src/simlin-engine/src/db/diagnostic.rs @@ -115,8 +115,8 @@ pub enum DiagnosticError { /// typo'd subscript simulates plausibly-but-wrong with no signal (GH #905). /// 4c. `emit_unfilled_equation_warnings` -- the unconditional /// `UnfilledEquation` advisory: a variable whose equation is nothing but -/// the NaN literal has no usable equation, so it and everything downstream -/// of it simulate as NaN. +/// the NaN literal has no usable equation, so it simulates as NaN and that +/// NaN spreads through arithmetic into whatever reads it. /// 5. When LTM is enabled, `emit_conveyor_ltm_degraded_warnings` and /// `emit_queue_ltm_degraded_warnings` -- one `Warning` per conveyor stock /// and per queue stock in THIS model, because LTM's flow-to-stock link-score @@ -220,7 +220,7 @@ pub fn model_all_diagnostics(db: &dyn Db, model: SourceModel, project: SourcePro // literal, which is where Vensim's `A FUNCTION OF(...)` placeholder lands. // Unconditional, for the same reason as the two advisories above -- it // describes the simulation, not an analysis overlay. - emit_unfilled_equation_warnings(db, model); + emit_unfilled_equation_warnings(db, model, project); // When LTM is enabled, also trigger the LTM diagnostic pass so that // diagnostics accumulated by the LTM pipeline surface through @@ -673,7 +673,6 @@ fn emit_unknown_element_subscript_warnings( project: SourceProject, ) { use crate::common::{CanonicalElementName, Error, ErrorCode, ErrorKind}; - use crate::dimensions::Dimension; use salsa::Accumulator; use std::collections::HashSet; @@ -694,21 +693,9 @@ fn emit_unknown_element_subscript_warnings( continue; } - // Resolve the variable's dimension names against the project's - // dimension definitions by canonical name (the same rule as - // `variable::get_dimensions`). Any unresolved name means the - // equation already carries BadDimensionName -- skip. - let dims: Option> = dim_names - .iter() - .map(|name| { - let canonical = crate::canonicalize(name); - project_dims - .iter() - .find(|d| crate::canonicalize(d.name()) == canonical) - .map(Dimension::from) - }) - .collect(); - let Some(dims) = dims else { + // Any unresolved dimension name means the equation already carries + // BadDimensionName -- skip rather than cascade on top of it. + let Some(dims) = resolve_equation_dimensions(project_dims, dim_names) else { continue; }; @@ -775,6 +762,70 @@ fn emit_unknown_element_subscript_warnings( } } +/// Resolve an arrayed equation's dimension NAMES against the project's +/// dimension declarations, by canonical name -- the same rule as +/// `variable::get_dimensions`. `None` if any name is unresolvable, which means +/// the equation already carries `BadDimensionName` from `compile_var_fragment`. +/// +/// Shared by the two arrayed-equation advisories so they cannot drift about what +/// a dimension name resolves to. +fn resolve_equation_dimensions( + project_dims: &[crate::datamodel::Dimension], + dim_names: &[String], +) -> Option> { + dim_names + .iter() + .map(|name| { + let canonical = crate::canonicalize(name); + project_dims + .iter() + .find(|d| crate::canonicalize(d.name()) == canonical) + .map(crate::dimensions::Dimension::from) + }) + .collect() +} + +/// Do `equation`'s explicit arms already name every slot its dimensions declare? +/// +/// `None` when the answer cannot be determined -- an unresolvable dimension name +/// (already reported as `BadDimensionName`), which the caller treats as "say +/// nothing about this variable" rather than guessing. +/// +/// The membership rule is deliberately the COMPILER's, not this file's sibling +/// advisory's. `compiler::expand_arrayed_with_hoisting` selects a slot's +/// equation with `elements.get(&CanonicalElementName::from_raw(combination.join(",")))` +/// and falls back to the EXCEPT default only on a miss, so that lookup IS the +/// fact being asked about; deriving it any other way would be re-deriving the +/// compiler's own decision by a second route. (`emit_unknown_element_subscript_warnings` +/// accepts the UNION of that rule and the conveyor init-list matcher's, because +/// it is asking a different question -- "does ANY consumer resolve this entry?" +/// -- and its rustdoc records that the two rules diverge.) +fn arm_coverage( + project_dims: &[crate::datamodel::Dimension], + equation: &crate::datamodel::Equation, +) -> Option { + use crate::common::CanonicalElementName; + use crate::variable::ArmCoverage; + + let crate::datamodel::Equation::Arrayed(dim_names, elements, _, _) = equation else { + // A scalar's single formula, and an apply-to-all's, IS every slot's. + return Some(ArmCoverage::CoversEverySlot); + }; + let dims = resolve_equation_dimensions(project_dims, dim_names)?; + let arm_keys: std::collections::HashSet = elements + .iter() + .map(|(subscript, _, _, _)| CanonicalElementName::from_raw(subscript)) + .collect(); + let covers_all = crate::dimensions::SubscriptIterator::new(&dims).all(|combination| { + arm_keys.contains(&CanonicalElementName::from_raw(&combination.join(","))) + }); + Some(if covers_all { + ArmCoverage::CoversEverySlot + } else { + ArmCoverage::LeavesSlotsUncovered + }) +} + /// Is `var`'s own equation only a FALLBACK -- a stand-in for a value a calling /// model normally supplies -- rather than the formula that produces its value? /// @@ -841,7 +892,7 @@ fn equation_is_a_module_input_fallback( /// passing CLI run or `get_errors` check red. /// /// Variables are visited in sorted-name order so accumulation is deterministic. -fn emit_unfilled_equation_warnings(db: &dyn Db, model: SourceModel) { +fn emit_unfilled_equation_warnings(db: &dyn Db, model: SourceModel, project: SourceProject) { use crate::common::{Error, ErrorCode, ErrorKind}; use crate::variable::{UnfilledArms, unfilled_arms}; use salsa::Accumulator; @@ -850,13 +901,18 @@ fn emit_unfilled_equation_warnings(db: &dyn Db, model: SourceModel) { let mut var_names: Vec<&String> = source_vars.keys().collect(); var_names.sort_unstable(); + let project_dims = project.dimensions(db); let model_name = model.name(db); for var_name in var_names { let svar = source_vars[var_name]; if equation_is_a_module_input_fallback(db, model, svar) { continue; } - let Some(unfilled) = unfilled_arms(svar.equation(db)) else { + let equation = svar.equation(db); + let Some(coverage) = arm_coverage(project_dims, equation) else { + continue; + }; + let Some(unfilled) = unfilled_arms(equation, coverage) else { continue; }; // A stock's `equation` is its INITIAL VALUE, not a formula re-evaluated @@ -881,12 +937,17 @@ fn emit_unfilled_equation_warnings(db: &dyn Db, model: SourceModel) { ) } }; + // The NaN claim is deliberately scoped to ARITHMETIC. NaN is not + // absorbing through comparisons -- `IF x > 0 THEN 1 ELSE 0` reading a + // NaN `x` returns a finite 0 -- so "every variable downstream is NaN" + // would be false guidance for exactly the backward search this + // diagnostic exists to shorten (see `crate::float`). let msg = format!( - "{subject}: a NaN literal stands where the formula belongs, so it simulates as NaN \ - -- and because NaN is absorbing, so does every variable downstream of it. Vensim's \ - Sketch tool writes 'A FUNCTION OF(...)' for an equation that has not been written \ - yet and refuses to simulate a model containing one; our importer stores that \ - placeholder as this NaN literal." + "{subject}: a NaN literal stands where the formula belongs, so it simulates as NaN, \ + and that NaN spreads through arithmetic into whatever reads it. Vensim's Sketch \ + tool writes 'A FUNCTION OF(...)' for an equation that has not been written yet and \ + refuses to simulate a model containing one; our importer stores that placeholder as \ + this NaN literal." ); CompilationDiagnostic(Diagnostic { model: model_name.clone(), diff --git a/src/simlin-engine/src/float.rs b/src/simlin-engine/src/float.rs index c2d5cb94f..854a0981e 100644 --- a/src/simlin-engine/src/float.rs +++ b/src/simlin-engine/src/float.rs @@ -14,10 +14,22 @@ //! //! **What a practitioner sees, and what it costs them.** A line on a graph that //! stops. The modeller's next task is *provenance*: search backward through the -//! dependency graph for where the NaN came from, because NaN is absorbing -- -//! every variable downstream of the origin shows the identical symptom, so the -//! graph says that something broke and not what. That backward hunt is the real -//! cost of a NaN, and it is paid by hand. +//! dependency graph for where the NaN came from, because a NaN spreads through +//! ARITHMETIC to whatever reads it, so a stopped line usually means the origin is +//! upstream rather than here. The graph says that something broke and not what. +//! That backward hunt is the real cost of a NaN, and it is paid by hand. +//! +//! **The exception matters during that hunt, so it is stated here rather than +//! discovered.** NaN is absorbing in arithmetic, NOT through comparisons and +//! conditionals. Every IEEE comparison against a NaN is false, so +//! `IF x > 0 THEN 1 ELSE 0` with a NaN `x` takes the ELSE branch and returns a +//! perfectly finite `0` -- and everything downstream of THAT is finite too. Two +//! consequences for a modeller tracing an origin: a finite variable does not +//! clear its inputs (it may be sitting on the NaN behind a conditional), and a +//! NaN can vanish partway down a chain rather than reaching the output that +//! would have revealed it. So "everything downstream is NaN" is the common case, +//! not the rule, and any diagnostic that claims otherwise is giving the modeller +//! false guidance. //! //! **What it usually is.** Most often the runtime result of a division by zero, //! which then propagates through everything reading it. Sometimes it is @@ -82,7 +94,10 @@ /// That existence test only works because `:NA:` is finite: ordinary `=` /// equality (`approx_eq`) matches the sentinel against itself, and arithmetic on /// `:NA:` computes a finite result rather than poisoning the expression the way -/// NaN (which is absorbing) would. Both `:NA:` paths in the engine -- the +/// NaN (which IS absorbing in arithmetic) would. Note the existence test itself +/// is the shape that would NOT have been rescued by a NaN: an `x = NAN` +/// comparison is false rather than poisoning, so the idiom would silently take +/// its else-branch instead of failing loudly. Both `:NA:` paths -- the /// expression literal (via the MDL->XMILE formatter) and the data-list literal /// (via the MDL number-list parser) -- route to this single constant so the /// representation is consistent and Vensim-faithful. diff --git a/src/simlin-engine/src/unfilled_equation_tests.rs b/src/simlin-engine/src/unfilled_equation_tests.rs index f06834911..d53efc73b 100644 --- a/src/simlin-engine/src/unfilled_equation_tests.rs +++ b/src/simlin-engine/src/unfilled_equation_tests.rs @@ -23,7 +23,7 @@ use crate::db::{ Diagnostic, DiagnosticError, DiagnosticSeverity, SimlinDb, collect_all_diagnostics, compile_project_incremental, sync_from_datamodel_incremental, }; -use crate::variable::{UnfilledArms, is_nan_literal, unfilled_arms}; +use crate::variable::{ArmCoverage, UnfilledArms, is_nan_literal, unfilled_arms}; // --------------------------------------------------------------------------- // The atom: is this equation text nothing but the NaN literal? @@ -123,10 +123,11 @@ fn arrayed(arms: Vec, default: Option<&str>, live: bool) -> datamodel::Equa datamodel::Equation::Arrayed(dims(), arms, default.map(str::to_string), live) } -// The two axes an `Arrayed` verdict turns on, each enumerated from what -// `unfilled_arms`'s `Arrayed` body actually branches on. The decision table is -// their full product, and `the_decision_table_covers_every_cell` checks that -// mechanically -- so the row count below is DERIVED, not asserted. +// THREE axes decide an `Arrayed` verdict. The first two are properties of the +// equation TEXT; the third is not derivable from it at all, which is how the +// first version of this table came to be wrong in both directions at once. +// The table is their product and `the_decision_table_covers_every_cell` checks +// that mechanically, so the row count is DERIVED rather than asserted. /// How many of the per-element arms are unfilled. Four states: `unfilled_arms` /// tests `unfilled.is_empty()` and `unfilled.len() == elements.len()`, and those @@ -147,11 +148,27 @@ const DEFAULT_STATES: [&str; 4] = [ "live-unfilled-default", ]; -/// The cell of the decision space an equation occupies, computed from the -/// equation ITSELF rather than from a row's label -- so a mislabelled row cannot -/// fake coverage. `Scalar` and `ApplyToAll` carry one arm and no default, so -/// their only axis is whether that arm is unfilled. -fn decision_cell(eqn: &datamodel::Equation) -> String { +/// Whether the explicit arms already name every slot the dimensions declare. +/// +/// This axis is the one no amount of staring at the equation's text can supply +/// -- it needs the dimensions RESOLVED -- and leaving it out made the verdict +/// wrong in both directions on models that run today: a NaN default no slot can +/// reach was reported, and a sparse array whose omitted slots compile to `0` was +/// reported as wholly unfilled. +const COVERAGE_STATES: [&str; 2] = ["covers-every-slot", "leaves-slots-uncovered"]; + +fn coverage_state(coverage: ArmCoverage) -> &'static str { + match coverage { + ArmCoverage::CoversEverySlot => "covers-every-slot", + ArmCoverage::LeavesSlotsUncovered => "leaves-slots-uncovered", + } +} + +/// The cell of the decision space a row occupies, computed from the row's +/// EQUATION and COVERAGE rather than from its label -- so a mislabelled row +/// cannot fake coverage. `Scalar` and `ApplyToAll` carry one arm and no default, +/// so their only axis is whether that arm is unfilled. +fn decision_cell(eqn: &datamodel::Equation, coverage: ArmCoverage) -> String { match eqn { datamodel::Equation::Scalar(s) | datamodel::Equation::ApplyToAll(_, s) => { let filled = if is_nan_literal(s) { @@ -181,14 +198,20 @@ fn decision_cell(eqn: &datamodel::Equation) -> String { (Some(d), true) if is_nan_literal(d) => "live-unfilled-default", (Some(_), true) => "live-filled-default", }; - format!("Arrayed/{arms}/{default}") + format!("Arrayed/{arms}/{default}/{}", coverage_state(coverage)) } } } -/// Every cell the table must cover: the two single-arm shapes times their one -/// 2-state axis, plus `Arrayed`'s full `ARM_STATES` x `DEFAULT_STATES` product. -/// **2 + 2 + (4 x 4) = 20.** +/// Every cell the table must cover. +/// +/// **2 (Scalar) + 2 (ApplyToAll) + [(4 arm-states x 4 default-states x 2 +/// coverage-states) - 4 impossible] = 32.** +/// +/// The four excluded cells are `no-arms` x `covers-every-slot`: zero arms cannot +/// name every slot of a dimension that has at least one. Feeding the classifier +/// that pair would pin behaviour for an input the emitter cannot construct, +/// which is worth less than saying why it is absent. fn expected_cells() -> BTreeSet { let mut cells = BTreeSet::new(); for shape in ["Scalar", "ApplyToAll"] { @@ -198,22 +221,48 @@ fn expected_cells() -> BTreeSet { } for arms in ARM_STATES { for default in DEFAULT_STATES { - cells.insert(format!("Arrayed/{arms}/{default}")); + for coverage in COVERAGE_STATES { + if arms == "no-arms" && coverage == "covers-every-slot" { + continue; + } + cells.insert(format!("Arrayed/{arms}/{default}/{coverage}")); + } } } cells } -/// The decision table: one row per cell of the space enumerated above, 20 in -/// total (see [`expected_cells`] for the arithmetic). +/// Build the arrayed fixture for one `(arms, default)` cell. Generated rather +/// than hand-written so the table cannot drift out of the axis product; the +/// EXPECTED VERDICTS in [`decision_table`] stay authored, which is the half that +/// must not be derived from the implementation. +fn arrayed_fixture(arms: &str, default: &str) -> datamodel::Equation { + let arms = match arms { + "no-arms" => vec![], + "none-unfilled" => vec![arm("a", "1"), arm("b", "2")], + "some-unfilled" => vec![arm("a", "1"), arm("b", "NAN")], + "all-unfilled" => vec![arm("a", "NAN"), arm("b", "nan")], + other => panic!("unknown arm state {other:?}"), + }; + let (default, live) = match default { + "no-default" => (None, false), + "dead-default" => (Some("NAN"), false), + "live-filled-default" => (Some("7"), true), + "live-unfilled-default" => (Some("nan"), true), + other => panic!("unknown default state {other:?}"), + }; + arrayed(arms, default, live) +} + +/// The verdict each cell must produce, authored cell by cell. /// -/// The previous version of this table claimed "6 reachable combinations" for -/// `Arrayed` and listed 6 rows, but it had silently split the default axis for -/// the all-arms case and collapsed it for the some-arms case. Three cells were -/// missing, including the only one that reaches the message branch joining -/// element names AND the default phrase. +/// Deliberately a flat list of literals rather than a `match` with grouped or +/// wildcard arms: a grouped expectation is a second copy of the rule under test, +/// and would agree with a wrong implementation for the same wrong reason. Every +/// entry here was worked out from what the compiled model actually evaluates, +/// slot by slot. #[allow(clippy::type_complexity)] -fn decision_table() -> Vec<(&'static str, datamodel::Equation, Option)> { +fn expected_verdicts() -> Vec<(&'static str, Option)> { let whole = || Some(UnfilledArms::Whole); let partial = |elements: &[&str], default: bool| { Some(UnfilledArms::Partial { @@ -221,133 +270,209 @@ fn decision_table() -> Vec<(&'static str, datamodel::Equation, Option Vec<( + String, + datamodel::Equation, + ArmCoverage, + Option, +)> { + let verdicts = expected_verdicts(); + let mut rows = Vec::new(); + let mut push = |cell: String, eqn: datamodel::Equation, coverage: ArmCoverage| { + let expected = verdicts + .iter() + .find(|(name, _)| *name == cell) + .unwrap_or_else(|| panic!("no authored verdict for cell {cell:?}")) + .1 + .clone(); + // The generated fixture must actually LAND in the cell it is filed + // under, or the coverage check below would be measuring labels. + assert_eq!(cell, decision_cell(&eqn, coverage), "fixture/cell mismatch"); + rows.push((cell, eqn, coverage, expected)); + }; + + push( + "Scalar/unfilled".to_string(), + datamodel::Equation::Scalar("NAN".to_string()), + ArmCoverage::CoversEverySlot, + ); + push( + "Scalar/filled".to_string(), + datamodel::Equation::Scalar("3 * x".to_string()), + ArmCoverage::CoversEverySlot, + ); + push( + "ApplyToAll/unfilled".to_string(), + datamodel::Equation::ApplyToAll(dims(), "nan".to_string()), + ArmCoverage::CoversEverySlot, + ); + push( + "ApplyToAll/filled".to_string(), + datamodel::Equation::ApplyToAll(dims(), "3 * x".to_string()), + ArmCoverage::CoversEverySlot, + ); + for arms in ARM_STATES { + for default in DEFAULT_STATES { + for coverage_name in COVERAGE_STATES { + if arms == "no-arms" && coverage_name == "covers-every-slot" { + continue; + } + let coverage = if coverage_name == "covers-every-slot" { + ArmCoverage::CoversEverySlot + } else { + ArmCoverage::LeavesSlotsUncovered + }; + push( + format!("Arrayed/{arms}/{default}/{coverage_name}"), + arrayed_fixture(arms, default), + coverage, + ); + } + } + } + rows +} + #[test] fn the_decision_table_pins_every_row() { - for (label, eqn, expected) in decision_table() { + for (cell, eqn, coverage, expected) in decision_table() { assert_eq!( expected, - unfilled_arms(&eqn), - "decision-table row {label:?} ({}) disagrees", - equation_shape(&eqn) + unfilled_arms(&eqn, coverage), + "decision-table cell {cell:?} disagrees" + ); + } +} + +/// Every authored verdict must belong to a real cell. Without this an entry left +/// behind by a renamed axis state would sit unused and unnoticed. +#[test] +fn every_authored_verdict_names_a_real_cell() { + let cells = expected_cells(); + for (name, _) in expected_verdicts() { + assert!( + cells.contains(name), + "authored verdict {name:?} names no cell of the decision space" ); } } @@ -369,7 +494,7 @@ fn the_decision_table_pins_every_row() { fn the_decision_table_covers_every_cell() { let covered: BTreeSet = decision_table() .iter() - .map(|(_, eqn, _)| decision_cell(eqn)) + .map(|(_, eqn, coverage, _)| decision_cell(eqn, *coverage)) .collect(); let expected = expected_cells(); assert_eq!( @@ -382,16 +507,16 @@ fn the_decision_table_covers_every_cell() { "one row per cell -- a duplicate row would hide a missing one" ); assert_eq!( - 20, + 32, expected.len(), - "2 + 2 + (4 arm-states x 4 default-states)" + "2 + 2 + (4 arm-states x 4 default-states x 2 coverage-states - 4 impossible)" ); // The shape list is still worth pinning on its own: it is what the two // catch-all-free matches above are enumerated FROM. let shapes: BTreeSet<&str> = decision_table() .iter() - .map(|(_, eqn, _)| equation_shape(eqn)) + .map(|(_, eqn, _, _)| equation_shape(eqn)) .collect(); assert_eq!(BTreeSet::from(EQUATION_SHAPES), shapes); } @@ -635,6 +760,116 @@ fn instantiating_a_stdlib_template_reports_nothing() { ); } +/// A NaN EXCEPT default that NO slot can reach is not an unfilled equation. +/// +/// The arms name every element of `D`, so `expand_arrayed_with_hoisting` never +/// consults the default; the compiled model holds 1/2/3 and contains no NaN +/// anywhere. Before the coverage axis this reported "no equation for every +/// element with no equation of its own" -- a finding about a slot that does not +/// exist. The value assertions are what make this a test of the claim: they fail +/// first if the default ever does drive a slot. +#[test] +fn an_unreachable_nan_default_is_not_reported() { + let project = read_xmile( + r#""#, + r#" + + NAN + 1 + 2 + 3 + + "#, + ); + assert_eq!( + Vec::<(String, String, String)>::new(), + unfilled_findings(&project), + "the arms cover D, so the NaN default is dead code" + ); + assert_eq!(1.0, final_value(&project, "full[a]")); + assert_eq!(2.0, final_value(&project, "full[b]")); + assert_eq!(3.0, final_value(&project, "full[c]")); +} + +/// A SPARSE array whose every listed arm is unfilled is not WHOLLY unfilled: +/// the slots with no arm compile to a finite `0`, not to NaN. +/// +/// `sparse[c]` has no entry and no default, so it is 0 while `[a]` and `[b]` are +/// NaN. Before the coverage axis this said "variable 'sparse' has no equation", +/// which reads as every slot being NaN. It now names the two arms that are. +/// +/// (That silent `0` for an armless slot is its own reportable shape and a +/// deliberately separate one -- GH #905 covers it -- so this test asserts the +/// value rather than expecting a second finding.) +#[test] +fn a_sparse_array_of_unfilled_arms_names_the_arms_not_the_variable() { + let project = read_xmile( + r#""#, + r#" + + NAN + NAN + + "#, + ); + let findings = unfilled_findings(&project); + assert_eq!(1, findings.len(), "one finding per variable: {findings:#?}"); + assert!( + findings[0] + .2 + .starts_with("array variable 'sparse' has no equation for 'a', 'b'"), + "the message must name the unfilled arms, not claim the whole variable \ + has no equation: {:?}", + findings[0].2 + ); + assert!(final_value(&project, "sparse[a]").is_nan()); + assert!(final_value(&project, "sparse[b]").is_nan()); + assert_eq!( + 0.0, + final_value(&project, "sparse[c]"), + "the armless slot compiles to a finite 0 -- which is why the whole \ + variable does not 'have no equation'" + ); +} + +/// The message may not claim that everything downstream of the unfilled variable +/// is NaN, because NaN is absorbing through ARITHMETIC only. +/// +/// `arith` reads `marketing` and is NaN; `guarded` reads it through a comparison +/// and is a finite 0, and `downstream` below that is a finite 5. IEEE says every +/// comparison against a NaN is false, so the conditional takes its ELSE branch +/// and the NaN stops there. A modeller tracing an origin backward needs that: a +/// finite variable does not clear its inputs. +#[test] +fn the_message_does_not_claim_every_downstream_variable_is_nan() { + let project = read_xmile( + "", + r#" + NAN + marketing * 2 + IF marketing > 0 THEN 1 ELSE 0 + guarded + 5"#, + ); + + // The premise: a conditional really does absorb the NaN here. + assert!(final_value(&project, "arith").is_nan()); + assert_eq!(0.0, final_value(&project, "guarded")); + assert_eq!(5.0, final_value(&project, "downstream")); + + let findings = unfilled_findings(&project); + assert_eq!(1, findings.len(), "{findings:#?}"); + let message = &findings[0].2; + assert!( + message.contains("spreads through arithmetic"), + "the message must scope the spread to arithmetic: {message:?}" + ); + assert!( + !message.contains("every variable downstream"), + "the message must not assert that every downstream variable is NaN -- \ + `guarded` is a finite 0: {message:?}" + ); +} + /// An arrayed variable with only SOME unfilled elements is reported, and the /// message names them. /// diff --git a/src/simlin-engine/src/variable.rs b/src/simlin-engine/src/variable.rs index dd80e8a2e..519f2510e 100644 --- a/src/simlin-engine/src/variable.rs +++ b/src/simlin-engine/src/variable.rs @@ -518,7 +518,7 @@ pub(crate) fn is_nan_literal(equation: &str) -> bool { /// apply-to-all variable, one `` entry of a per-element arrayed /// variable, or an arrayed variable's EXCEPT default (the formula for every /// element with no entry of its own). -#[derive(Debug, PartialEq, Eq)] +#[derive(Debug, Clone, PartialEq, Eq)] pub(crate) enum UnfilledArms { /// The variable has NO equation anywhere: its scalar/apply-to-all formula is /// the NaN literal, or every arm of its arrayed equation is (its EXCEPT @@ -532,6 +532,27 @@ pub(crate) enum UnfilledArms { }, } +/// Whether a variable's explicit arms already name every slot its dimensions +/// declare -- the fact that decides what the slots WITHOUT an arm evaluate to, +/// and therefore whether an EXCEPT default is reachable at all. +/// +/// This cannot be read off a `datamodel::Equation`: it needs the equation's +/// dimensions RESOLVED against the project's declarations, which is a database +/// read. So it is computed by the caller (`db::diagnostic::arm_coverage`) and +/// passed in, leaving [`unfilled_arms`] a pure classifier. The alternative -- +/// giving the classifier the resolved element set -- would move a salsa read +/// into a pure function for no gain, since a single bit is all the verdict uses. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum ArmCoverage { + /// Every declared slot has an equation of its own. A scalar or apply-to-all + /// variable is always this: its single formula IS every slot's. + CoversEverySlot, + /// At least one declared slot has no arm naming it, so something else + /// decides that slot's value -- the EXCEPT default if one is live, else the + /// compiler's silent `0`. + LeavesSlotsUncovered, +} + /// Classify `equation` by which of its arms are unfilled, or `None` when none /// are. /// @@ -543,7 +564,19 @@ pub(crate) enum UnfilledArms { /// fine, one stopping -- so a partially-unfilled variable is reported too, and /// [`UnfilledArms::Partial`] names the arms so the message can point at them. /// Either way it is at most ONE finding per variable, never one per element. -pub(crate) fn unfilled_arms(equation: &datamodel::Equation) -> Option { +/// +/// `coverage` is load-bearing for both directions of the arrayed verdict, and +/// both were wrong without it. A NaN EXCEPT default that no slot can reach is +/// not an unfilled equation (arms `a=1,b=2` over `D=[a,b]` with default `NAN` +/// warned, though the compiled model contains no NaN at all), and a sparse +/// array's omitted slots are not covered by its listed arms (arms `a=NAN,b=NAN` +/// over `D=[a,b,c]` claimed the WHOLE variable had no equation, though `[c]` +/// compiles to `0`). Both follow from the same rule: what an armless slot +/// evaluates to is not something the arms' text can answer. +pub(crate) fn unfilled_arms( + equation: &datamodel::Equation, + coverage: ArmCoverage, +) -> Option { use crate::datamodel::Equation; match equation { // One arm covering the whole variable: a scalar formula, or one formula @@ -557,21 +590,30 @@ pub(crate) fn unfilled_arms(equation: &datamodel::Equation) -> Option Date: Tue, 28 Jul 2026 09:53:40 -0700 Subject: [PATCH 12/17] engine: share the compiler's implicit-axis allocation The per-element pin projection decided which target axis supplied each axis of a bare arrayed dependency by searching the target's dimension NAMES, and a name is not an axis identity. Two shapes came out wrong, both compiling and both reading a row the simulation does not: a target that repeats a dimension collapsed to one map entry, so `target[D,D] = driver * w` over a `w[D]` pinned the second axis where the simulation reads the first; and two dependency axes that could each map to either target axis both claimed the first, because an independent per-axis search tracks no used set. Neither had a diagnostic. The compiler already resolves exactly this reference, so its allocation moves into `dimensions::allocate_implicit_axes_partial` and both callers read it -- positional, and one-to-one because each active axis is consumed once. The compiler takes the total projection of that answer and errors without it; LTM takes the partial one, because a subscripted reference spells some of its own axes. The explicit over-arity bail and the equal-arity reordering fast path are deleted rather than moved: the first is pigeonhole and the second can only fire where the general first pass agrees. What could not be shared is declined instead. The other per-element row derivations are asked their question as a dimension name -- that is what an AxisRead::Iterated carries -- so a repeated dimension is not expressible in their interface, and making it so means changing the reference-shape IR. That edge now gets a Warning and no score, which is the trade this area has taken every time a plausible wrong number was the alternative. Three checked-in models declare a repeated dimension, so this is a shape users write. Also fixes a test helper that built element indices 0-based where production is 1-based; it went unnoticed until the projection started calling get_offset on a fixture dimension and read the wrong mapped partner. --- src/simlin-engine/src/compiler/context.rs | 162 ++--------- src/simlin-engine/src/compiler/dimensions.rs | 150 ++++++++++ src/simlin-engine/src/db/ltm/link_scores.rs | 106 +++++++- src/simlin-engine/src/db/ltm_char_tests.rs | 257 ++++++++++++++++++ src/simlin-engine/src/ltm_augment.rs | 5 +- .../src/ltm_augment_pin_tests.rs | 103 ++++++- .../src/ltm_augment_post_transform.rs | 139 +++++----- src/simlin-engine/src/ltm_augment_tests.rs | 8 +- 8 files changed, 686 insertions(+), 244 deletions(-) diff --git a/src/simlin-engine/src/compiler/context.rs b/src/simlin-engine/src/compiler/context.rs index 77da0020c..8e31bf843 100644 --- a/src/simlin-engine/src/compiler/context.rs +++ b/src/simlin-engine/src/compiler/context.rs @@ -16,7 +16,9 @@ use crate::dimensions::{Dimension, DimensionsContext}; use crate::variable::Variable; use crate::{Error, sim_err}; -use super::dimensions::{UnaryOp, find_dimension_reordering, match_dimensions_with_mapping}; +use super::dimensions::{ + UnaryOp, allocate_implicit_axes, find_dimension_reordering, match_dimensions_with_mapping, +}; use super::expr::{BuiltinFn, Expr, SubscriptIndex, VarRef}; use super::subscript::{ IndexOp, Subscript3Config, ViewBuildConfig, ViewBuildResult, build_view_from_ops, @@ -253,6 +255,14 @@ impl Context<'_> { self.get_submodel_metadata(self.model_name, ident) } + /// The active subscript each of `dims` reads, for a BARE arrayed reference + /// inside an apply-to-all body. + /// + /// The axis ALLOCATION -- which active axis supplies which of `dims` -- is + /// `dimensions::allocate_implicit_axes`, shared with the LTM per-element + /// projection so a link-score pin cannot spell a row this reference does not + /// read. See that function for the two properties (positional, one-to-one) + /// that a name-keyed re-derivation gets wrong. fn get_implicit_subscripts(&self, dims: &[Dimension], ident: &str) -> Result> { if self.active_dimension.is_none() { return sim_err!(ArrayReferenceNeedsExplicitSubscripts, ident.to_owned()); @@ -261,151 +271,13 @@ impl Context<'_> { let active_subscripts = self.active_subscript.as_ref().unwrap(); assert_eq!(active_dims.len(), active_subscripts.len()); - // Check if dimensions can be reordered to match - if dims.len() == active_dims.len() { - // Get dimension names (all canonical at this point) - let source_dim_names: Vec = dims.iter().map(|d| d.name().to_string()).collect(); - let target_dim_names: Vec = - active_dims.iter().map(|d| d.name().to_string()).collect(); - - // Check if dimensions can be reordered - // Note: we're asking "how to reorder target to match source" - if let Some(_reordering) = - find_dimension_reordering(&target_dim_names, &source_dim_names) - { - // Build subscripts in the order needed by the source dims - // reordering[i] tells us which target dimension to use for source position i - let mut subscripts: Vec<&str> = Vec::with_capacity(dims.len()); - for source_dim in dims { - // Find which active dimension matches this source dimension - for (j, active_dim) in active_dims.iter().enumerate() { - if active_dim.name() == source_dim.name() { - subscripts.push(active_subscripts[j].as_str()); - break; - } - } - } - return Ok(subscripts); - } - } - - // Fall back to original logic for partial dimension matching - // if we need more dimensions than are implicit, that's an error - if dims.len() > active_dims.len() { - return sim_err!(MismatchedDimensions, ident.to_owned()); - } - - // goal: if this is a valid equation, dims will be a subset of active_dims (order preserving) - - let mut subscripts: Vec<&str> = Vec::with_capacity(dims.len()); - - // Track which active dimensions have been used - let mut used: Vec = vec![false; active_dims.len()]; - - for dim in dims.iter() { - // FIRST PASS: Try to find an exact name match anywhere in unused active dims. - // This prevents size-based fallback from grabbing the wrong dimension when - // the correct name match exists later in the list. - let name_match_idx = active_dims.iter().enumerate().find_map(|(i, candidate)| { - if !used[i] && candidate.name() == dim.name() { - Some(i) - } else { - None - } - }); - - if let Some(idx) = name_match_idx { - subscripts.push(active_subscripts[idx].as_str()); - used[idx] = true; - continue; - } - - // SECOND PASS: Check for dimension mapping matches in both directions. - // Forward: dim has any mapping to an active dimension - // Reverse: active_dim has any mapping to dim - let mapping_match_idx = { - // Forward: dim has mapping to active dim (or active is subdim of mapping target) - let mut found = active_dims.iter().enumerate().find_map(|(i, candidate)| { - if used[i] { - return None; - } - let candidate_name = candidate.canonical_name(); - if self - .dimensions_ctx - .has_mapping_to(dim.canonical_name(), candidate_name) - { - return Some(i); - } - if self - .dimensions_ctx - .has_mapping_to_parent_of(dim.canonical_name(), candidate_name) - { - return Some(i); - } - None - }); - // Reverse: active_dim has mapping to dim - if found.is_none() { - found = active_dims.iter().enumerate().find_map(|(i, candidate)| { - if used[i] { - return None; - } - if self - .dimensions_ctx - .has_mapping_to(candidate.canonical_name(), dim.canonical_name()) - { - return Some(i); - } - None - }); - } - found - }; - - if let Some(idx) = mapping_match_idx { - subscripts.push(active_subscripts[idx].as_str()); - used[idx] = true; - continue; - } - - // THIRD PASS: Only if no name or mapping match exists, try size-based matching - // for indexed dimensions. Find the first unused indexed dimension with - // the same size. - // - // IMPORTANT: Size-based fallback only applies when BOTH dimensions are - // indexed. Named dimensions must match by name (or subdimension relationship) - // because their elements have semantic meaning. For example, Cities=[Boston, - // Seattle] and Products=[Widgets,Gadgets] shouldn't match just because both - // have size 2 - that would be semantically incorrect. - // - // NOTE: The two-pass (name -> size) matching logic is shared with the VM via - // dimensions::match_dimensions_two_pass. This compiler version adds a mapping - // pass between name and size matching. - let size_match_idx = if let Dimension::Indexed(_, dim_size) = dim { - active_dims.iter().enumerate().find_map(|(i, candidate)| { - if !used[i] - && let Dimension::Indexed(_, candidate_size) = candidate - && dim_size == candidate_size - { - return Some(i); - } - None - }) - } else { - None - }; - - if let Some(idx) = size_match_idx { - subscripts.push(active_subscripts[idx].as_str()); - used[idx] = true; - continue; - } - - // No match found - return sim_err!(MismatchedDimensions, ident.to_owned()); + match allocate_implicit_axes(dims, active_dims, self.dimensions_ctx) { + Some(alloc) => Ok(alloc + .into_iter() + .map(|i| active_subscripts[i].as_str()) + .collect()), + None => sim_err!(MismatchedDimensions, ident.to_owned()), } - - Ok(subscripts) } fn get_implicit_subscript_off(&self, dims: &[Dimension], ident: &str) -> Result { diff --git a/src/simlin-engine/src/compiler/dimensions.rs b/src/simlin-engine/src/compiler/dimensions.rs index 23c7cbeb3..1a8b18099 100644 --- a/src/simlin-engine/src/compiler/dimensions.rs +++ b/src/simlin-engine/src/compiler/dimensions.rs @@ -6,6 +6,156 @@ use std::collections::HashMap; use crate::dimensions::{Dimension, DimensionsContext}; +/// For each dimension of `dims`, the POSITION in `active_dims` whose subscript +/// supplies it, or `None` for a dimension no active axis supplies -- the +/// engine's SINGLE implicit-subscript axis allocation, the rule behind a BARE +/// arrayed reference inside an apply-to-all body. +/// +/// Two properties matter to every caller and neither is re-derivable safely: +/// +/// - the answer is POSITIONAL, so a target that repeats a dimension +/// (`target[D,D]`) is handled by index rather than by name. A map keyed by +/// dimension name collapses the two occurrences and answers with whichever was +/// inserted last; +/// - the allocation is ONE-TO-ONE: each active axis is consumed at most once +/// (`used`), so two dependency axes that could each match the same active axis +/// are given different ones in declaration order. Searching per dependency axis +/// independently lets both claim the first match. +/// +/// Both were live silent-wrong-row defects in the LTM per-element projection +/// (P2-1 / P2-2 of the whole-branch review) precisely because that projection +/// re-derived this rule instead of asking for it. `crate::ltm_augment` now calls +/// this, so its pins and the executed reads agree by construction rather than by +/// parallel implementation. Do not add a second copy of this decision. +/// +/// The compiler wants the TOTAL answer and errors without it, so it calls +/// [`allocate_implicit_axes`]; the LTM projection wants the partial one, because +/// a SUBSCRIPTED reference spells some of its own axes and only needs the rest +/// resolved. One algorithm, two projections of its result. +pub(crate) fn allocate_implicit_axes_partial( + dims: &[Dimension], + active_dims: &[Dimension], + dimensions_ctx: &DimensionsContext, +) -> Vec> { + let mut alloc: Vec> = Vec::with_capacity(dims.len()); + + // Track which active dimensions have been used. + let mut used: Vec = vec![false; active_dims.len()]; + + for dim in dims.iter() { + // FIRST PASS: Try to find an exact name match anywhere in unused active dims. + // This prevents size-based fallback from grabbing the wrong dimension when + // the correct name match exists later in the list. + let name_match_idx = active_dims.iter().enumerate().find_map(|(i, candidate)| { + if !used[i] && candidate.name() == dim.name() { + Some(i) + } else { + None + } + }); + + if let Some(idx) = name_match_idx { + alloc.push(Some(idx)); + used[idx] = true; + continue; + } + + // SECOND PASS: Check for dimension mapping matches in both directions. + // Forward: dim has any mapping to an active dimension + // Reverse: active_dim has any mapping to dim + let mapping_match_idx = { + // Forward: dim has mapping to active dim (or active is subdim of mapping target) + let mut found = active_dims.iter().enumerate().find_map(|(i, candidate)| { + if used[i] { + return None; + } + let candidate_name = candidate.canonical_name(); + if dimensions_ctx.has_mapping_to(dim.canonical_name(), candidate_name) { + return Some(i); + } + if dimensions_ctx.has_mapping_to_parent_of(dim.canonical_name(), candidate_name) { + return Some(i); + } + None + }); + // Reverse: active_dim has mapping to dim + if found.is_none() { + found = active_dims.iter().enumerate().find_map(|(i, candidate)| { + if used[i] { + return None; + } + if dimensions_ctx + .has_mapping_to(candidate.canonical_name(), dim.canonical_name()) + { + return Some(i); + } + None + }); + } + found + }; + + if let Some(idx) = mapping_match_idx { + alloc.push(Some(idx)); + used[idx] = true; + continue; + } + + // THIRD PASS: Only if no name or mapping match exists, try size-based matching + // for indexed dimensions. Find the first unused indexed dimension with + // the same size. + // + // IMPORTANT: Size-based fallback only applies when BOTH dimensions are + // indexed. Named dimensions must match by name (or subdimension relationship) + // because their elements have semantic meaning. For example, Cities=[Boston, + // Seattle] and Products=[Widgets,Gadgets] shouldn't match just because both + // have size 2 - that would be semantically incorrect. + // + // NOTE: The two-pass (name -> size) matching logic is shared with the VM via + // dimensions::match_dimensions_two_pass. This compiler version adds a mapping + // pass between name and size matching. + let size_match_idx = if let Dimension::Indexed(_, dim_size) = dim { + active_dims.iter().enumerate().find_map(|(i, candidate)| { + if !used[i] + && let Dimension::Indexed(_, candidate_size) = candidate + && dim_size == candidate_size + { + return Some(i); + } + None + }) + } else { + None + }; + + alloc.push(size_match_idx); + if let Some(idx) = size_match_idx { + used[idx] = true; + } + } + + alloc +} + +/// [`allocate_implicit_axes_partial`] with every dimension resolved, or `None`. +/// +/// `None` is the compiler's `MismatchedDimensions`: no complete allocation +/// exists. That subsumes the old explicit `dims.len() > active_dims.len()` bail +/// by pigeonhole -- the allocation is one-to-one, so a longer `dims` always +/// leaves an axis unresolved -- and it subsumes the old equal-arity +/// [`find_dimension_reordering`] fast path, which could only fire on a +/// duplicate-free permutation, where the partial allocation's first pass finds +/// the same unique partner for every dimension. +pub(crate) fn allocate_implicit_axes( + dims: &[Dimension], + active_dims: &[Dimension], + dimensions_ctx: &DimensionsContext, +) -> Option> { + allocate_implicit_axes_partial(dims, active_dims, dimensions_ctx) + .into_iter() + .collect() +} + /// Three-pass dimension matching: name -> mapping -> size. /// /// For each source dimension, finds the target dimension by trying in order: diff --git a/src/simlin-engine/src/db/ltm/link_scores.rs b/src/simlin-engine/src/db/ltm/link_scores.rs index 20e0e0822..4038d688e 100644 --- a/src/simlin-engine/src/db/ltm/link_scores.rs +++ b/src/simlin-engine/src/db/ltm/link_scores.rs @@ -66,10 +66,33 @@ fn pinnable_arrayed_deps( pinnable } +/// The target element as a POSITIONAL tuple over `to_dims`: one bare element +/// name per axis, in axis order, duplicates and all. +/// +/// This -- not a map keyed by dimension name -- is what the pin projection +/// consumes, because a target may repeat a dimension (`target[D,D]`) and the two +/// axes then read different coordinates. Empty when the tuple's arity does not +/// match, which the projection reads as "no axis resolves". +fn target_element_parts(to_dims: &[crate::dimensions::Dimension], element: &str) -> Vec { + let parts: Vec = element.split(',').map(|p| p.trim().to_string()).collect(); + if parts.len() == to_dims.len() { + parts + } else { + Vec::new() + } +} + /// The projection data one target element supplies: target dimension /// (canonical) -> (that element's coordinate on the dimension, its index within /// the dimension's element list). /// +/// Keyed by NAME, so a target that repeats a dimension is not representable +/// here: the second axis overwrites the first. Its one remaining consumer +/// (`per_element_row_for_target`, whose `AxisRead::Iterated` interface is +/// name-keyed too) is guarded by `emit_per_element_link_scores` declining a +/// repeated-dimension target outright. The pin projection does NOT use this -- +/// see [`target_element_parts`]. +/// /// `element` is a comma-joined BARE element tuple over `to_dims`, and /// `dim_element_lists` is `to_dims`' element lists (already computed by every /// caller for `cartesian_subscripts`). A part that names no element of its @@ -84,10 +107,18 @@ fn target_elem_by_dim_for( if parts.len() != to_dims.len() { return HashMap::new(); } - let mut by_dim = HashMap::with_capacity(to_dims.len()); + let mut by_dim: HashMap = HashMap::with_capacity(to_dims.len()); for ((dim, elems), part) in to_dims.iter().zip(dim_element_lists).zip(&parts) { if let Some(idx) = elems.iter().position(|e| e == part) { - by_dim.insert(dim.name().to_string(), ((*part).to_string(), idx)); + // FIRST occurrence wins: a dimension-NAME index resolves to the first + // active axis with that name (`compiler::subscript`'s `ActiveDimRef` + // is `active_dims.position(..)`), so if this map is ever asked about + // a repeated dimension it should answer the way the compiler does. + // `emit_per_element_link_scores` declines such a target outright, so + // this is the fallback behind that guard rather than the guarantee. + by_dim + .entry(dim.name().to_string()) + .or_insert_with(|| ((*part).to_string(), idx)); } } by_dim @@ -1279,7 +1310,6 @@ pub(super) fn try_scalar_to_arrayed_link_scores( .iter() .map(crate::ltm_augment::dimension_element_names) .collect(); - let to_dim_names: Vec = to_dims.iter().map(|d| d.name().to_string()).collect(); let dim_ctx = project_dimensions_context(db, project); let elements = cartesian_subscripts(&dim_element_lists); // GH #910: resolved once for the whole target -- see `WithLookupSlotRefs`. @@ -1346,8 +1376,8 @@ pub(super) fn try_scalar_to_arrayed_link_scores( // projected off this target element (GH #974). let deps_to_sub = crate::ltm_augment::dep_element_pins( pinnable, - &to_dim_names, - &target_elem_by_dim_for(&to_dims, &dim_element_lists, element), + &to_dims, + &target_element_parts(&to_dims, element), dim_ctx, ); match crate::ltm_augment::generate_scalar_to_element_equation( @@ -1461,6 +1491,31 @@ pub(super) fn try_scalar_to_arrayed_link_scores( Some(cross_vars) } +/// Accumulate the `Warning` for a `PerElement` edge whose target REPEATS a +/// dimension (`target[D,D]`). +/// +/// The per-element row derivations address a target axis by its dimension NAME, +/// so two axes sharing one name are indistinguishable to them. Rather than emit +/// a partial that compiles and reads whichever axis the lookup resolved to, the +/// edge gets no link-score variable and loops through it are dropped -- the same +/// loud-skip contract the disjoint and dim-incompatible classes follow. +fn emit_repeated_target_dimension_warning(db: &dyn Db, model: SourceModel, from: &str, to: &str) { + use salsa::Accumulator; + let msg = format!( + "LTM link score for edge {from} -> {to} could not be computed: {to} repeats a \ + dimension in its subscripts, and the per-element row derivation addresses a \ + target axis by dimension name, so it cannot tell the two axes apart; this edge \ + will have no link-score variable and feedback loops through it will not be scored" + ); + CompilationDiagnostic(Diagnostic { + model: model.name(db).clone(), + variable: None, + error: DiagnosticError::Assembly(msg), + severity: DiagnosticSeverity::Warning, + }) + .accumulate(db); +} + /// Accumulate the AC3.4 `Warning` for a disjoint-dim arrayed -> arrayed /// edge that has no scoreable per-element derivation: the target's /// per-element equations reference the source via something other than @@ -2494,7 +2549,6 @@ fn emit_per_element_link_scores( .iter() .map(crate::ltm_augment::dimension_element_names) .collect(); - let to_dim_names: Vec = to_dims.iter().map(|d| d.name().to_string()).collect(); let elements = cartesian_subscripts(&to_dim_element_lists); // Per target element: the body text + dep sets (shared for A2A, @@ -2508,6 +2562,31 @@ fn emit_per_element_link_scores( None }; + // A target that REPEATS a dimension (`target[D,D]`) is not projectable by + // this emitter, so decline the edge rather than emit a wrong row. Every row + // derivation it reaches -- `per_element_row_for_target` and + // `pin_dimension_name_indices` -- is asked its question as a dimension NAME + // (that is what an `AxisRead::Iterated` and a dimension-name index carry), and + // two axes sharing one name have two different coordinates. The simulation + // resolves such an index to the FIRST matching axis; a partial that guesses + // otherwise compiles and reads the other row, which is the failure this whole + // area keeps producing (P2-1, and GH #986 before it). The SCALAR-source and + // AGG emitters do not go through those derivations -- they project + // positionally via `dep_element_pins` -- and are unaffected + // (`repeated_target_dimension_reads_the_first_axis`). + if to_dims.len() + != to_dims + .iter() + .map(|d| d.canonical_name()) + .collect::>() + .len() + { + if unscoreable_edges.insert((from.to_string(), to.to_string())) { + emit_repeated_target_dimension_warning(db, model, from, to); + } + return true; + } + // Name-level dedup: two sites can in principle derive the same // (row, e) name; first emission wins (matching the per-shape pass's // name-dedup convention). Vars collect locally and commit only if no @@ -2560,8 +2639,8 @@ fn emit_per_element_link_scores( // off this target element (GH #974). let to_sub = crate::ltm_augment::dep_element_pins( pinnable, - &to_dim_names, - &target_elem_by_dim, + &to_dims, + &target_element_parts(&to_dims, element), dim_ctx, ); @@ -2616,6 +2695,8 @@ fn emit_per_element_link_scores( &to_sub, from_dims, &target_elem_by_dim, + &to_dims, + &target_element_parts(&to_dims, element), &target_iterated_dims, dim_ctx, gf_table_ref.as_deref(), @@ -3296,11 +3377,6 @@ pub(super) fn emit_agg_to_target_link_scores( // computed by `source_pins_for_target` below. let dim_ctx = project_dimensions_context(db, project); - let to_dim_names: Vec = to_dims.iter().map(|d| d.name().to_string()).collect(); - let dim_element_lists_for_target: Vec> = to_dims - .iter() - .map(crate::ltm_augment::dimension_element_names) - .collect(); #[derive(Clone)] struct AggTargetProjectionAxis { target_pos: usize, @@ -3492,8 +3568,8 @@ pub(super) fn emit_agg_to_target_link_scores( let deps_to_subscript = |element: &str| { crate::ltm_augment::dep_element_pins( &pinnable_deps, - &to_dim_names, - &target_elem_by_dim_for(&to_dims, &dim_element_lists_for_target, element), + &to_dims, + &target_element_parts(&to_dims, element), dim_ctx, ) }; diff --git a/src/simlin-engine/src/db/ltm_char_tests.rs b/src/simlin-engine/src/db/ltm_char_tests.rs index 5416c08fe..91ad6b601 100644 --- a/src/simlin-engine/src/db/ltm_char_tests.rs +++ b/src/simlin-engine/src/db/ltm_char_tests.rs @@ -2110,3 +2110,260 @@ fn lookup_table_head_and_static_index_survive_the_wrap() { "a PREVIOUS of a table has no value slot; got: {text}" ); } + +// --------------------------------------------------------------------------- +// P2-1 / P2-2 of the whole-branch review: the per-element pin projection keyed +// its axis lookup by dimension NAME, which is not an axis identity. Both shapes +// below COMPILE AND RUN, so each was a silent wrong row rather than a latent +// one, and each is measured against the VM before its pin is asserted. +// +// This is the second time in this area a name-keyed table produced a wrong row +// (GH #986 was the first), which is why the projection now asks +// `compiler::dimensions::allocate_implicit_axes_partial` instead of restating +// the rule. +// --------------------------------------------------------------------------- + +/// Read one variable's final-step value out of a compiled+run model. +fn final_value(project: &datamodel::Project, name: &str) -> f64 { + let db = SimlinDb::default(); + let sp = sync_from_datamodel(&db, project).project; + let compiled = compile_project_incremental(&db, sp, "main").expect("the fixture compiles"); + let off = *compiled + .offsets + .iter() + .find(|(k, _)| k.as_str() == name) + .unwrap_or_else(|| panic!("no slot named {name}")) + .1; + let mut vm = crate::vm::Vm::new(compiled).expect("VM creation should succeed"); + vm.run_to_end().expect("simulation should run"); + vm.into_results().iter().next_back().expect("a saved step")[off] +} + +fn repeated_dimension_model() -> datamodel::Project { + TestProject::new("repeated_target_dim") + .with_sim_time(0.0, 1.0, 1.0) + .named_dimension("Region", &["nyc", "boston"]) + .array_with_ranges_direct( + "w", + vec!["Region".to_string()], + vec![("nyc", "1"), ("boston", "2")], + None, + ) + // A stock, so the model is stateful and LTM scores it at all. + .flow("inflow", "1", None) + .stock("driver", "3", &["inflow"], &[], None) + // `target` repeats `Region`: two axes, one name. + .array_aux_direct( + "target", + vec!["Region".to_string(), "Region".to_string()], + "driver * w", + None, + ) + .build_datamodel() +} + +/// A target that REPEATS a dimension gives a bare dep the FIRST axis, and the +/// pin must say so (P2-1). +/// +/// `target[Region,Region] = driver * w` with `w[Region]`: the simulation reads +/// `w` at the FIRST `Region` coordinate, so `target[nyc,boston]` reads `w[nyc]`. +/// The pin projection keyed its lookup by dimension name, so the second axis +/// overwrote the first and it emitted `PREVIOUS(w[region·boston])` -- a fragment +/// that compiles and reads the other row. +#[test] +fn repeated_target_dimension_reads_the_first_axis() { + let project = repeated_dimension_model(); + + // What the simulation reads, measured. `w[nyc]` and `w[boston]` differ, so + // the two candidate reads are distinguishable. + assert_ne!( + final_value(&project, "w[nyc]"), + final_value(&project, "w[boston]") + ); + assert_eq!( + final_value(&project, "target[nyc,boston]"), + final_value(&project, "driver") * final_value(&project, "w[nyc]"), + "a bare dep under a repeated-dimension target reads the FIRST axis" + ); + + // ...and the pin spells that row. + let (db, model, source_project) = char_fixture_db(&project); + let text = link_score_text( + &db, + model, + source_project, + &["link_score\u{205A}driver\u{2192}target[nyc,boston]"], + ); + assert!( + text.contains("previous(w[region\u{B7}nyc])"), + "the pin must read the FIRST Region axis, as the simulation does; got: {text}" + ); + assert!( + !text.contains("previous(w[region\u{B7}boston])"), + "a name-keyed lookup keeps only the LAST axis and spells the wrong row; \ + got: {text}" + ); +} + +fn doubly_mapped_model() -> datamodel::Project { + let mut project = TestProject::new("doubly_mapped") + .with_sim_time(0.0, 1.0, 1.0) + .named_dimension("T", &["t1", "t2"]) + .named_dimension("U", &["u1", "u2"]) + .array_with_ranges_direct( + "aw", + vec!["A".to_string()], + vec![("a1", "1"), ("a2", "2")], + None, + ) + .array_with_ranges_direct( + "bw", + vec!["B".to_string()], + vec![("b1", "10"), ("b2", "20")], + None, + ) + .flow("inflow", "1", None) + .stock("driver", "1", &["inflow"], &[], None) + .array_aux_direct( + "dep", + vec!["A".to_string(), "B".to_string()], + "aw[A] + bw[B]", + None, + ) + .array_aux_direct( + "target", + vec!["T".to_string(), "U".to_string()], + "driver * dep", + None, + ) + .build_datamodel(); + // `A` and `B` each map to BOTH target dimensions, so an independent per-axis + // search hands both of them `T`. + let both = || { + vec![ + datamodel::DimensionMapping { + target: "T".to_string(), + element_map: vec![], + }, + datamodel::DimensionMapping { + target: "U".to_string(), + element_map: vec![], + }, + ] + }; + let mut a = + datamodel::Dimension::named("A".to_string(), vec!["a1".to_string(), "a2".to_string()]); + a.mappings = both(); + let mut b = + datamodel::Dimension::named("B".to_string(), vec!["b1".to_string(), "b2".to_string()]); + b.mappings = both(); + project.dimensions.push(a); + project.dimensions.push(b); + project +} + +/// Two dependency axes that can each map to either target axis are allocated +/// ONE-TO-ONE, in declaration order (P2-2). +/// +/// `target[T,U] = driver * dep` with `dep[A,B]`, where `A` and `B` both carry +/// positional mappings to both `T` and `U`. The simulation consumes each target +/// axis once, so `target[t1,u2]` reads `dep[a1,b2]`. The pin projection searched +/// per dependency axis independently with no `used` set, so both claimed `T` and +/// it emitted `PREVIOUS(dep[a1,b1])` -- again compilable and again the wrong row. +#[test] +fn doubly_mapped_dep_axes_are_allocated_one_to_one() { + let project = doubly_mapped_model(); + + // What the simulation reads, measured; the two candidate rows differ. + assert_ne!( + final_value(&project, "dep[a1,b1]"), + final_value(&project, "dep[a1,b2]") + ); + assert_eq!( + final_value(&project, "target[t1,u2]"), + final_value(&project, "driver") * final_value(&project, "dep[a1,b2]"), + "each target axis is consumed once: A takes T, B takes U" + ); + + // ...and the pin spells that row. + let (db, model, source_project) = char_fixture_db(&project); + let text = link_score_text( + &db, + model, + source_project, + &["link_score\u{205A}driver\u{2192}target[t1,u2]"], + ); + assert!( + text.contains("previous(dep[a\u{B7}a1, b\u{B7}b2])"), + "the pin must allocate one-to-one, as the simulation does; got: {text}" + ); + assert!( + !text.contains("previous(dep[a\u{B7}a1, b\u{B7}b1])"), + "an independent per-axis search gives both dep axes the FIRST target \ + axis and spells the wrong row; got: {text}" + ); +} + +/// A `PerElement` edge whose target REPEATS a dimension is declined loudly. +/// +/// The scalar-source and agg emitters project positionally and handle a repeated +/// dimension correctly (`repeated_target_dimension_reads_the_first_axis`), but the +/// per-element emitter's row derivations address a target axis by its dimension +/// NAME -- that is what an `AxisRead::Iterated` carries -- and two axes with one +/// name are indistinguishable to them. Emitting anyway produces a partial that +/// compiles and reads whichever axis the lookup resolved to, so the edge is +/// skipped with a `Warning` instead: loud beats a plausible wrong number, which +/// is the trade this area has taken every time. +/// +/// The fixture's own equation compiles and runs, so the decline is LTM's and not +/// a consequence of an unsupported model. +#[test] +fn per_element_edge_declines_a_repeated_dimension_target() { + use crate::db::{DiagnosticError, DiagnosticSeverity, collect_model_diagnostics}; + + let project = TestProject::new("per_element_repeated_dim") + .with_sim_time(0.0, 1.0, 1.0) + .named_dimension("Region", &["nyc", "boston"]) + .named_dimension("Age", &["young", "old"]) + .array_flow("growth[Region,Age]", "1", None) + .array_stock("pop[Region,Age]", "10", &["growth"], &[], None) + // `pop[Region, young]` is a PerElement site; the target repeats `Region`. + .array_aux_direct( + "target", + vec!["Region".to_string(), "Region".to_string()], + "pop[Region, young]", + None, + ) + .build_datamodel(); + + // The model itself is fine -- the decline below is LTM's. + let plain_db = SimlinDb::default(); + let plain = sync_from_datamodel(&plain_db, &project).project; + compile_project_incremental(&plain_db, plain, "main") + .expect("the repeated-dimension model compiles"); + + let (db, model, source_project) = char_fixture_db(&project); + let diags = collect_model_diagnostics(&db, model, source_project); + assert!( + diags + .iter() + .any(|d| d.severity == DiagnosticSeverity::Warning + && matches!(&d.error, DiagnosticError::Assembly(m) + if m.contains("repeats a dimension") && m.contains("pop") && m.contains("target"))), + "the edge must be declined with a Warning naming it; got {:?}", + diags.iter().map(|d| &d.error).collect::>() + ); + + // ...and no per-element score is emitted for it, rather than a wrong one. + let ltm = model_ltm_variables(&db, model, source_project); + let per_element: Vec<&String> = ltm + .vars + .iter() + .map(|v| &v.name) + .filter(|n| n.contains("link_score\u{205A}pop[") && n.contains("\u{2192}target[")) + .collect(); + assert!( + per_element.is_empty(), + "a declined edge must emit no per-element score; got {per_element:?}" + ); +} diff --git a/src/simlin-engine/src/ltm_augment.rs b/src/simlin-engine/src/ltm_augment.rs index 8da4b7211..260e144fc 100644 --- a/src/simlin-engine/src/ltm_augment.rs +++ b/src/simlin-engine/src/ltm_augment.rs @@ -2795,6 +2795,8 @@ pub(crate) fn generate_per_element_link_equation( to_deps_to_subscript: &HashMap, DepElementPin>, from_dims: &[crate::dimensions::Dimension], target_elem_by_dim: &HashMap, + target_dims: &[crate::dimensions::Dimension], + target_elements: &[String], target_iterated_dims: &[String], dims_ctx: &crate::dimensions::DimensionsContext, gf_table_ref: Option<&str>, @@ -2821,8 +2823,9 @@ pub(crate) fn generate_per_element_link_equation( site_axes, row_parts_bare, from_dims, + target_dims, + target_elements, target_elem_by_dim, - target_iterated_dims, dim_ctx: dims_ctx, }; // The ceteris-paribus wrap runs on the target element's OWN equation, diff --git a/src/simlin-engine/src/ltm_augment_pin_tests.rs b/src/simlin-engine/src/ltm_augment_pin_tests.rs index 085a720b8..c12d9ce58 100644 --- a/src/simlin-engine/src/ltm_augment_pin_tests.rs +++ b/src/simlin-engine/src/ltm_augment_pin_tests.rs @@ -72,13 +72,16 @@ fn per_element_pin_descends_into_range_endpoints() { let mut target_elem_by_dim = HashMap::new(); target_elem_by_dim.insert("region".to_string(), ("boston".to_string(), 1usize)); + let target_dims = vec![make_named_dimension("region", &["nyc", "boston"])]; + let target_elements = vec!["boston".to_string()]; let ctx = super::post_transform::PerElementRefCtx { from: &from, site_axes: &site_axes, row_parts_bare: &row_parts_bare, from_dims: &from_dims, + target_dims: &target_dims, + target_elements: &target_elements, target_elem_by_dim: &target_elem_by_dim, - target_iterated_dims: &target_iterated_dims, dim_ctx: &dim_ctx, }; @@ -172,6 +175,22 @@ fn dep_element_pins_projection_enumeration() { target: "region".to_string(), element_map, }]; + // `dblx`/`dbly` each map to BOTH target axes, so a per-axis search that + // tracks no `used` set hands them the same one (P2-2). + let dbl = |name: &str, elems: Vec| { + let mut d = datamodel::Dimension::named(name.to_string(), elems); + d.mappings = vec![ + datamodel::DimensionMapping { + target: "region".to_string(), + element_map: vec![], + }, + datamodel::DimensionMapping { + target: "age".to_string(), + element_map: vec![], + }, + ]; + d + }; DimensionsContext::from( [ datamodel::Dimension::named( @@ -183,6 +202,8 @@ fn dep_element_pins_projection_enumeration() { vec!["young".to_string(), "old".to_string()], ), state, + dbl("dblx", vec!["x1".to_string(), "x2".to_string()]), + dbl("dbly", vec!["y1".to_string(), "y2".to_string()]), datamodel::Dimension::named( "other".to_string(), vec!["o1".to_string(), "o2".to_string()], @@ -195,14 +216,14 @@ fn dep_element_pins_projection_enumeration() { let age = make_named_dimension("age", &["young", "old"]); let state = make_named_dimension("state", &["west", "east"]); let other = make_named_dimension("other", &["o1", "o2"]); + let dblx = make_named_dimension("dblx", &["x1", "x2"]); + let dbly = make_named_dimension("dbly", &["y1", "y2"]); // Target `growth[Region,Age]` at element `(boston, old)` -- both // coordinates are the SECOND element of their dimension, which is what // makes the positional correspondence observable (`state·east`). - let target_iterated_dims = vec!["region".to_string(), "age".to_string()]; - let mut target_elem_by_dim: HashMap = HashMap::new(); - target_elem_by_dim.insert("region".to_string(), ("boston".to_string(), 1usize)); - target_elem_by_dim.insert("age".to_string(), ("old".to_string(), 1usize)); + let target_dims = vec![region.clone(), age.clone()]; + let target_elements = vec!["boston".to_string(), "young".to_string()]; let pinnable: Vec<(Ident, Vec)> = vec![ // identity, in the target's own order @@ -217,6 +238,9 @@ fn dep_element_pins_projection_enumeration() { (Ident::new("partial"), vec![region.clone(), other.clone()]), // nothing resolves -> absent (Ident::new("unrelated"), vec![other.clone()]), + // P2-2: BOTH axes can map to BOTH target axes. The allocation must be + // one-to-one and in declaration order, matching the compiler. + (Ident::new("doubly"), vec![dblx.clone(), dbly.clone()]), ]; let axes_of = |pins: &HashMap, crate::ltm_augment::DepElementPin>, @@ -230,8 +254,8 @@ fn dep_element_pins_projection_enumeration() { let positional = build_ctx(vec![]); let pins = super::post_transform::dep_element_pins( &pinnable, - &target_iterated_dims, - &target_elem_by_dim, + &target_dims, + &target_elements, &positional, ); @@ -240,7 +264,7 @@ fn dep_element_pins_projection_enumeration() { Some(( vec![ axis("region", "region\u{B7}boston"), - axis("age", "age\u{B7}old") + axis("age", "age\u{B7}young") ], true )), @@ -251,7 +275,7 @@ fn dep_element_pins_projection_enumeration() { axes_of(&pins, "flip"), Some(( vec![ - axis("age", "age\u{B7}old"), + axis("age", "age\u{B7}young"), axis("region", "region\u{B7}boston") ], true @@ -261,7 +285,7 @@ fn dep_element_pins_projection_enumeration() { ); assert_eq!( axes_of(&pins, "sub"), - Some((vec![axis("age", "age\u{B7}old")], true)), + Some((vec![axis("age", "age\u{B7}young")], true)), "a subset-dims dep must be pinned over its own single axis, not the \ target's full tuple" ); @@ -270,7 +294,7 @@ fn dep_element_pins_projection_enumeration() { Some(( vec![ axis("state", "state\u{B7}east"), - axis("age", "age\u{B7}old") + axis("age", "age\u{B7}young") ], true )), @@ -288,6 +312,41 @@ fn dep_element_pins_projection_enumeration() { None, "a dep no axis of which projects has nothing to rewrite and must be absent" ); + // P2-2: each target axis is consumed once, so the second dep axis gets the + // second target axis rather than re-claiming the first. `boston` is Region's + // second element and `old` is Age's, so a one-to-one allocation reads each + // mapped dimension's SECOND element. + assert_eq!( + axes_of(&pins, "doubly"), + Some(( + vec![axis("dblx", "dblx\u{B7}x2"), axis("dbly", "dbly\u{B7}y1")], + true + )), + "two dep axes that can each map to either target axis must be allocated \ + ONE-TO-ONE in declaration order, as the compiler allocates them; an \ + independent per-axis search gives both the first target axis" + ); + + // P2-1: a target that REPEATS a dimension. The two axes are different reads, + // and the simulation gives a bare dep the FIRST -- measured by + // `repeated_target_dimension_reads_the_first_axis`. A map keyed by dimension + // name cannot express this at all; the positional projection can. + let repeated_dims = vec![region.clone(), region.clone()]; + let repeated_elements = vec!["nyc".to_string(), "boston".to_string()]; + let repeated_pins = super::post_transform::dep_element_pins( + &[(Ident::new("w"), vec![region.clone()])], + &repeated_dims, + &repeated_elements, + &positional, + ); + assert_eq!( + repeated_pins + .get(&Ident::::new("w")) + .map(|p| (p.axes.clone(), p.complete)), + Some((vec![axis("region", "region\u{B7}nyc")], true)), + "a subset dep under a repeated-dimension target reads the FIRST axis; a \ + name-keyed map keeps only the last and would say `boston`" + ); // An EXPLICIT element map is declined by `mapped_element_correspondence` // (execution resolves positionally and ignores it, GH #756), so the same @@ -298,13 +357,13 @@ fn dep_element_pins_projection_enumeration() { ]); let pins = super::post_transform::dep_element_pins( &pinnable, - &target_iterated_dims, - &target_elem_by_dim, + &target_dims, + &target_elements, &element_mapped, ); assert_eq!( axes_of(&pins, "mapped"), - Some((vec![axis("age", "age\u{B7}old")], false)), + Some((vec![axis("age", "age\u{B7}young")], false)), "an element-mapped axis must decline: following the map would spell a \ read the positionally-resolving simulation never performs" ); @@ -332,6 +391,14 @@ struct PinFixture { /// The QUALIFIED target element this instantiation emits for -- `region·boston` /// for [`PinFixture::new`], `state·ma` for [`PinFixture::mapped`]. target_element: String, + /// The target equation's dimensions in AXIS order, and this instantiation's + /// element as a positional tuple over them. Every fixture here iterates one + /// dimension, so these are the one-element twins of `target_iterated_dims` + /// and `target_elem_by_dim` -- carried separately because a repeated + /// dimension makes the name-keyed pair unrepresentable, which is what the + /// pin projection now refuses to depend on. + target_dims: Vec, + target_elements: Vec, } impl PinFixture { @@ -377,6 +444,8 @@ impl PinFixture { row_parts_bare: vec!["boston".to_string(), "young".to_string()], target_elem_by_dim, target_element: "region\u{B7}boston".to_string(), + target_dims: vec![make_named_dimension("region", &["nyc", "boston"])], + target_elements: vec!["boston".to_string()], } } @@ -455,6 +524,8 @@ impl PinFixture { row_parts_bare: vec!["boston".to_string(), "young".to_string()], target_elem_by_dim, target_element: "state\u{B7}ma".to_string(), + target_dims: vec![make_named_dimension("state", &["ny", "ma"])], + target_elements: vec!["ma".to_string()], } } @@ -507,6 +578,8 @@ impl PinFixture { row_parts_bare: vec!["boston".to_string(), "old".to_string()], target_elem_by_dim, target_element: "region\u{B7}boston".to_string(), + target_dims: vec![make_named_dimension("region", &["nyc", "boston"])], + target_elements: vec!["boston".to_string()], } } @@ -577,6 +650,8 @@ impl PinFixture { &HashMap::new(), &self.from_dims, &self.target_elem_by_dim, + &self.target_dims, + &self.target_elements, &self.target_iterated_dims, &self.dim_ctx, None, diff --git a/src/simlin-engine/src/ltm_augment_post_transform.rs b/src/simlin-engine/src/ltm_augment_post_transform.rs index 716c2aeed..3a06cfc69 100644 --- a/src/simlin-engine/src/ltm_augment_post_transform.rs +++ b/src/simlin-engine/src/ltm_augment_post_transform.rs @@ -79,14 +79,16 @@ pub(super) struct PerElementRefCtx<'a> { pub(super) row_parts_bare: &'a [String], /// The source's declared dimensions (for index qualification). pub(super) from_dims: &'a [crate::dimensions::Dimension], + /// The target equation's dimensions, in axis order and WITH duplicates -- + /// the positional twin of `target_elem_by_dim`, which a repeated dimension + /// makes unrepresentable. [`dep_row_for_target`] consumes these. + pub(super) target_dims: &'a [crate::dimensions::Dimension], + /// This instantiation's target element, one bare name per axis of + /// `target_dims`. + pub(super) target_elements: &'a [String], /// Target-iterated dim (canonical) -> (element of `e` for that dim, /// its index within the dim) -- the projection data `e` supplies. pub(super) target_elem_by_dim: &'a HashMap, - /// The target equation's iterated dimensions, canonical and IN ORDER. - /// Same set as `target_elem_by_dim`'s keys; carried separately because - /// [`dep_row_for_target`]'s mapped search must be deterministic and a - /// `HashMap`'s iteration order is not. - pub(super) target_iterated_dims: &'a [String], pub(super) dim_ctx: &'a crate::dimensions::DimensionsContext, } @@ -141,71 +143,73 @@ pub(crate) fn per_element_row_for_target( } /// Per DECLARED dimension of a dep, the element it reads at ONE target element -/// -- `None` for a dimension this target element projects onto neither by name -/// nor through a positional mapping. Bare element names, in the dep's own -/// declaration order. +/// -- `None` for a dimension the target element does not supply. Bare element +/// names, in the dep's own declaration order. /// -/// A bare arrayed reference inside an apply-to-all body reads the element the -/// enclosing iteration selects FOR EACH OF ITS OWN AXES, matched by dimension -/// NAME and not by position: `growth[Region,Age] = ... w ...` over a `w[Age]` -/// reads `w[]`, and over a `w[Age,Region]` reads +/// A bare arrayed reference inside an apply-to-all body reads, for each of its +/// OWN axes, the element the enclosing iteration selects on the target axis that +/// axis is allocated to. `growth[Region,Age] = ... w ...` over a `w[Age]` reads +/// `w[]`, and over a `w[Age,Region]` reads /// `w[,]` -- the transpose of the target's own tuple. That is the /// executed behaviour (pinned by -/// `bare_arrayed_dep_is_pinned_over_its_own_declared_dims`' numeric oracle), -/// and it is why pinning such a reference with the target's FULL element tuple -/// is wrong twice over: over- or under-arity when the dep declares a strict -/// subset (a fragment that fails to compile, so the score reads a constant 0), -/// and a compilable, SILENTLY WRONG element when it declares the same -/// dimensions in another order (GH #974). +/// `bare_arrayed_dep_is_pinned_over_its_own_declared_dims`' numeric oracle), and +/// it is why pinning such a reference with the target's FULL element tuple is +/// wrong twice over: over- or under-arity when the dep declares a strict subset +/// (a fragment that fails to compile, so the score reads a constant 0), and a +/// compilable, SILENTLY WRONG element when it declares the same dimensions in +/// another order (GH #974). /// -/// The per-axis question -- which target coordinate projects onto this -/// dimension -- is handed to [`per_element_row_for_target`], the single row -/// derivation, as an [`crate::ltm_agg::AxisRead::Iterated`] pair. That is what -/// makes the identity axis and a positionally-MAPPED one (`w[Region]` read from -/// a `growth[State,..]` body under a `State`/`Region` mapping) one arm instead -/// of two, and it means this accepts exactly the mapped pairs the -/// occurrence-driven pin and `ltm_agg::classify_axis_access` accept. +/// **The ALLOCATION is not decided here.** Which target axis supplies which dep +/// axis is `compiler::dimensions::allocate_implicit_axes` -- the same function +/// the compiler's own `get_implicit_subscripts` calls to resolve exactly this +/// reference -- so a pin cannot spell a row the simulation does not read. This +/// used to be a per-axis `.find` over the target's dimension NAMES, and it got +/// two shapes wrong for the one reason: a name is not an axis identity. A target +/// repeating a dimension (`target[D,D]`) has two axes with one name, and a +/// name-keyed map keeps only the last (the simulation reads the FIRST, measured +/// by `repeated_target_dimension_reads_the_first_axis`); and two dep axes that +/// can each map to the same target axis both claimed it, because an independent +/// search tracks no `used` set (the simulation allocates one-to-one, measured by +/// `doubly_mapped_dep_axes_are_allocated_one_to_one`). That is the SECOND time in +/// this area a table keyed by dimension name produced a wrong row -- GH #986 was +/// the first -- so the rule is now asked for rather than restated. /// -/// The mapped search walks `target_iterated_dims` in the target's declared -/// ORDER, so a dep dimension reachable from two mapped target dimensions -/// resolves the same way on every run (`target_elem_by_dim` is a `HashMap` -/// and cannot be iterated for this). +/// What remains here is the per-axis ELEMENT translation, which is LTM's own and +/// deliberately narrower than the compiler's: an axis allocated to a target axis +/// of a different name resolves through +/// `DimensionsContext::mapped_element_correspondence`, which declines explicit +/// element maps (GH #756 -- the executed A2A lowering resolves positionally and +/// ignores them), so this accepts exactly the mapped pairs the occurrence-driven +/// pin and `ltm_agg::classify_axis_access` accept. fn dep_axis_elements( dep_dims: &[crate::dimensions::Dimension], - target_iterated_dims: &[String], - target_elem_by_dim: &HashMap, + target_dims: &[crate::dimensions::Dimension], + target_elements: &[String], dim_ctx: &crate::dimensions::DimensionsContext, ) -> Vec> { - use crate::common::CanonicalDimensionName; - use crate::ltm_agg::AxisRead; + use crate::common::CanonicalElementName; + if target_dims.len() != target_elements.len() { + return vec![None; dep_dims.len()]; + } + let alloc = + crate::compiler::dimensions::allocate_implicit_axes_partial(dep_dims, target_dims, dim_ctx); dep_dims .iter() - .map(|dep_dim| { - let source_dim = dep_dim.name().to_string(); - // The target iterating the dep's OWN dimension is the identity - // projection and always wins; only a name the target does not - // iterate goes looking for a mapped partner. - let target_dim = if target_elem_by_dim.contains_key(&source_dim) { - source_dim.clone() - } else { - target_iterated_dims - .iter() - .find(|td| { - dim_ctx - .mapped_element_correspondence( - &CanonicalDimensionName::from_raw(td), - dep_dim.canonical_name(), - ) - .is_some() - })? - .clone() - }; - let axis = AxisRead::Iterated { - dim: target_dim, - source_dim, - }; - per_element_row_for_target(std::slice::from_ref(&axis), target_elem_by_dim, dim_ctx) - .map(|row| row[0].clone()) + .zip(alloc) + .map(|(dep_dim, target_pos)| { + let target_dim = target_dims.get(target_pos?)?; + let target_elem = target_elements.get(target_pos?)?; + if dep_dim.canonical_name() == target_dim.canonical_name() { + return Some(target_elem.clone()); + } + // A different-named axis: the element is whatever the executed A2A + // lowering reads there, i.e. the positional correspondence. + let corr = dim_ctx.mapped_element_correspondence( + target_dim.canonical_name(), + dep_dim.canonical_name(), + )?; + let idx = target_dim.get_offset(&CanonicalElementName::from_raw(target_elem))?; + corr.get(idx).map(|e| e.as_str().to_string()) }) .collect() } @@ -217,11 +221,11 @@ fn dep_axis_elements( /// at all. pub(crate) fn dep_row_for_target( dep_dims: &[crate::dimensions::Dimension], - target_iterated_dims: &[String], - target_elem_by_dim: &HashMap, + target_dims: &[crate::dimensions::Dimension], + target_elements: &[String], dim_ctx: &crate::dimensions::DimensionsContext, ) -> Option> { - dep_axis_elements(dep_dims, target_iterated_dims, target_elem_by_dim, dim_ctx) + dep_axis_elements(dep_dims, target_dims, target_elements, dim_ctx) .into_iter() .collect() } @@ -245,15 +249,14 @@ pub(crate) fn dep_row_for_target( /// target equation by the caller; only the projection is per element. pub(crate) fn dep_element_pins( pinnable: &[(Ident, Vec)], - target_iterated_dims: &[String], - target_elem_by_dim: &HashMap, + target_dims: &[crate::dimensions::Dimension], + target_elements: &[String], dim_ctx: &crate::dimensions::DimensionsContext, ) -> HashMap, DepElementPin> { pinnable .iter() .filter_map(|(ident, dep_dims)| { - let elems = - dep_axis_elements(dep_dims, target_iterated_dims, target_elem_by_dim, dim_ctx); + let elems = dep_axis_elements(dep_dims, target_dims, target_elements, dim_ctx); let complete = elems.iter().all(Option::is_some); let axes: Vec<(String, String)> = elems .iter() @@ -432,8 +435,8 @@ pub(super) fn pin_source_subscript_indices( pub(super) fn pin_bare_source_ref(ctx: &PerElementRefCtx<'_>) -> Option> { dep_row_for_target( ctx.from_dims, - ctx.target_iterated_dims, - ctx.target_elem_by_dim, + ctx.target_dims, + ctx.target_elements, ctx.dim_ctx, ) .map(|row| qualified_row_indices(&row, ctx)) diff --git a/src/simlin-engine/src/ltm_augment_tests.rs b/src/simlin-engine/src/ltm_augment_tests.rs index c8c9c59d5..737eb0551 100644 --- a/src/simlin-engine/src/ltm_augment_tests.rs +++ b/src/simlin-engine/src/ltm_augment_tests.rs @@ -22,7 +22,13 @@ fn make_named_dimension(name: &str, elements: &[&str]) -> Dimension { let indexed: HashMap = canonical_elements .iter() .enumerate() - .map(|(i, e)| (e.clone(), i)) + // 1-indexed, matching `impl From<&datamodel::Dimension>` -- production + // subtracts 1 in `get_offset`/`get_element_index`, so a 0-based helper + // silently shifts every element lookup down by one. That went unnoticed + // until the pin projection started calling `get_offset` on a fixture + // dimension, where it turned `boston` (Region's second element) into + // index 0 and read the wrong mapped partner. + .map(|(i, e)| (e.clone(), i + 1)) .collect(); Dimension::Named( CanonicalDimensionName::from_raw(name), From 0619020af718700e67ff7811c6c885bec97e4811 Mon Sep 17 00:00:00 2001 From: Bobby Powers Date: Tue, 28 Jul 2026 10:15:48 -0700 Subject: [PATCH 13/17] engine: an ignored arm is not an unfilled equation Third and last correction to the UnfilledEquation advisory, and the same root as the second: the gap between the arms AS WRITTEN and the arms that REACH A SLOT. ae71ed0d handled an arm the compiler never selects because the other arms already cover the dimension; this handles one it never selects because the subscript names nothing. Given arms a=1, b=2 and typo=NAN over D=[a,b], expand_arrayed_with_hoisting drops the typo'd arm and both slots compile finite, yet the advisory claimed typo simulates as NaN -- on top of the UnknownElementSubscript warning that correctly reported the subscript itself. Both facts come from one place. arm_coverage already built the declared combination keys and the arm keys to answer "does every slot have an arm"; the effective-arm test is that same lookup read the other way, so ArmCoverage now carries the effective set alongside the coverage bit and unfilled_arms classifies over the arms the compiler will select. No second scan and no second membership rule -- the rule stays expand_arrayed_with_hoisting's own element lookup. The UnknownElementSubscript advisory is untouched: a subscript naming nothing is still worth reporting, and what was wrong was the second warning claiming a value that no slot has. Ineffective arms are deliberately NOT a fifth decision-table axis. They are not a state the verdict depends on but a thing that must not matter, so the table stays a 32-cell product and one invariant test asserts that appending an ignored arm -- NaN or filled -- changes no verdict in any cell. That is a stronger claim than the 24 rows a fifth axis would have added, and a smaller one. The corpus count is unchanged at 94 findings across 4 files: exactly one checked-in model has unknown-subscript arms at all, and none of them is a NaN literal, so the defect was reachable by construction rather than by any fixture. ArmCoverage's Debug derive is feature-gated like CanonicalElementName's -- an unconditional one compiles under the test profile and breaks the release build pysimlin uses. --- src/simlin-engine/src/db/diagnostic.rs | 41 ++-- .../src/unfilled_equation_tests.rs | 210 ++++++++++++++---- src/simlin-engine/src/variable.rs | 115 +++++++--- 3 files changed, 279 insertions(+), 87 deletions(-) diff --git a/src/simlin-engine/src/db/diagnostic.rs b/src/simlin-engine/src/db/diagnostic.rs index 9de83302d..b797a832e 100644 --- a/src/simlin-engine/src/db/diagnostic.rs +++ b/src/simlin-engine/src/db/diagnostic.rs @@ -785,7 +785,8 @@ fn resolve_equation_dimensions( .collect() } -/// Do `equation`'s explicit arms already name every slot its dimensions declare? +/// What the compiler will do with `equation`'s arms: which of them reach a slot, +/// and whether any declared slot is left without one. /// /// `None` when the answer cannot be determined -- an unresolvable dimension name /// (already reported as `BadDimensionName`), which the caller treats as "say @@ -796,34 +797,40 @@ fn resolve_equation_dimensions( /// equation with `elements.get(&CanonicalElementName::from_raw(combination.join(",")))` /// and falls back to the EXCEPT default only on a miss, so that lookup IS the /// fact being asked about; deriving it any other way would be re-deriving the -/// compiler's own decision by a second route. (`emit_unknown_element_subscript_warnings` -/// accepts the UNION of that rule and the conveyor init-list matcher's, because -/// it is asking a different question -- "does ANY consumer resolve this entry?" -/// -- and its rustdoc records that the two rules diverge.) +/// compiler's own decision by a second route. Both facts read that one lookup in +/// opposite directions -- a declared slot with no arm leaves the slot uncovered, +/// an arm matching no declared slot is ignored -- so they come from a single +/// pair of sets rather than from two independent scans. +/// +/// (`emit_unknown_element_subscript_warnings` accepts the UNION of that rule and +/// the conveyor init-list matcher's, because it is asking a different question +/// -- "does ANY consumer resolve this entry?" -- and its rustdoc records that +/// the two rules diverge. That advisory keeps reporting an unknown subscript; +/// what this one must not do is additionally claim the ignored arm simulates as +/// NaN when no slot does.) fn arm_coverage( project_dims: &[crate::datamodel::Dimension], equation: &crate::datamodel::Equation, ) -> Option { use crate::common::CanonicalElementName; use crate::variable::ArmCoverage; + use std::collections::HashSet; let crate::datamodel::Equation::Arrayed(dim_names, elements, _, _) = equation else { - // A scalar's single formula, and an apply-to-all's, IS every slot's. - return Some(ArmCoverage::CoversEverySlot); + return Some(ArmCoverage::whole_variable()); }; let dims = resolve_equation_dimensions(project_dims, dim_names)?; - let arm_keys: std::collections::HashSet = elements + let arm_keys: HashSet = elements .iter() .map(|(subscript, _, _, _)| CanonicalElementName::from_raw(subscript)) .collect(); - let covers_all = crate::dimensions::SubscriptIterator::new(&dims).all(|combination| { - arm_keys.contains(&CanonicalElementName::from_raw(&combination.join(","))) - }); - Some(if covers_all { - ArmCoverage::CoversEverySlot - } else { - ArmCoverage::LeavesSlotsUncovered - }) + let declared_keys: HashSet = + crate::dimensions::SubscriptIterator::new(&dims) + .map(|combination| CanonicalElementName::from_raw(&combination.join(","))) + .collect(); + let leaves_slots_uncovered = declared_keys.iter().any(|key| !arm_keys.contains(key)); + let effective_arms = arm_keys.intersection(&declared_keys).cloned().collect(); + Some(ArmCoverage::new(effective_arms, leaves_slots_uncovered)) } /// Is `var`'s own equation only a FALLBACK -- a stand-in for a value a calling @@ -912,7 +919,7 @@ fn emit_unfilled_equation_warnings(db: &dyn Db, model: SourceModel, project: Sou let Some(coverage) = arm_coverage(project_dims, equation) else { continue; }; - let Some(unfilled) = unfilled_arms(equation, coverage) else { + let Some(unfilled) = unfilled_arms(equation, &coverage) else { continue; }; // A stock's `equation` is its INITIAL VALUE, not a formula re-evaluated diff --git a/src/simlin-engine/src/unfilled_equation_tests.rs b/src/simlin-engine/src/unfilled_equation_tests.rs index d53efc73b..2caee0958 100644 --- a/src/simlin-engine/src/unfilled_equation_tests.rs +++ b/src/simlin-engine/src/unfilled_equation_tests.rs @@ -17,7 +17,7 @@ use std::collections::BTreeSet; -use crate::common::ErrorCode; +use crate::common::{CanonicalElementName, ErrorCode}; use crate::datamodel; use crate::db::{ Diagnostic, DiagnosticError, DiagnosticSeverity, SimlinDb, collect_all_diagnostics, @@ -129,10 +129,14 @@ fn arrayed(arms: Vec, default: Option<&str>, live: bool) -> datamodel::Equa // The table is their product and `the_decision_table_covers_every_cell` checks // that mechanically, so the row count is DERIVED rather than asserted. -/// How many of the per-element arms are unfilled. Four states: `unfilled_arms` -/// tests `unfilled.is_empty()` and `unfilled.len() == elements.len()`, and those -/// two comparisons distinguish exactly these -- with the empty-`elements` case -/// separate because it satisfies BOTH (`0 == 0`). +/// How many of the EFFECTIVE per-element arms are unfilled -- effective meaning +/// the compiler would select the arm for some slot. Four states: `unfilled_arms` +/// tests `unfilled.is_empty()` and `unfilled.len() == effective_count`, and those +/// two comparisons distinguish exactly these -- with the no-arms case separate +/// because it satisfies BOTH (`0 == 0`). +/// +/// `no-arms` covers "written with no arms" and "every arm names nothing" alike: +/// after filtering they are the same variable, entirely default- or zero-filled. const ARM_STATES: [&str; 4] = ["no-arms", "none-unfilled", "some-unfilled", "all-unfilled"]; /// The state of the EXCEPT default. Four states: it is dropped unless @@ -157,18 +161,39 @@ const DEFAULT_STATES: [&str; 4] = [ /// reported as wholly unfilled. const COVERAGE_STATES: [&str; 2] = ["covers-every-slot", "leaves-slots-uncovered"]; -fn coverage_state(coverage: ArmCoverage) -> &'static str { - match coverage { - ArmCoverage::CoversEverySlot => "covers-every-slot", - ArmCoverage::LeavesSlotsUncovered => "leaves-slots-uncovered", +fn coverage_state(coverage: &ArmCoverage) -> &'static str { + if coverage.leaves_slots_uncovered() { + "leaves-slots-uncovered" + } else { + "covers-every-slot" } } +/// The coverage an arrayed fixture is classified under: every arm it declares is +/// effective (the fixtures use only real subscripts), plus the coverage state +/// being exercised. +/// +/// Ineffective arms are deliberately NOT a fifth axis. Whether one exists cannot +/// change any verdict once they are filtered, and asserting that INVARIANT over +/// the whole table (`an_arm_the_compiler_never_selects_cannot_change_the_verdict`) +/// is both a stronger claim and a smaller one than doubling the product. +fn coverage_for(eqn: &datamodel::Equation, leaves_slots_uncovered: bool) -> ArmCoverage { + let datamodel::Equation::Arrayed(_, arms, _, _) = eqn else { + return ArmCoverage::whole_variable(); + }; + ArmCoverage::new( + arms.iter() + .map(|(subscript, _, _, _)| CanonicalElementName::from_raw(subscript)) + .collect(), + leaves_slots_uncovered, + ) +} + /// The cell of the decision space a row occupies, computed from the row's /// EQUATION and COVERAGE rather than from its label -- so a mislabelled row /// cannot fake coverage. `Scalar` and `ApplyToAll` carry one arm and no default, /// so their only axis is whether that arm is unfilled. -fn decision_cell(eqn: &datamodel::Equation, coverage: ArmCoverage) -> String { +fn decision_cell(eqn: &datamodel::Equation, coverage: &ArmCoverage) -> String { match eqn { datamodel::Equation::Scalar(s) | datamodel::Equation::ApplyToAll(_, s) => { let filled = if is_nan_literal(s) { @@ -407,44 +432,40 @@ fn decision_table() -> Vec<( .clone(); // The generated fixture must actually LAND in the cell it is filed // under, or the coverage check below would be measuring labels. - assert_eq!(cell, decision_cell(&eqn, coverage), "fixture/cell mismatch"); + assert_eq!( + cell, + decision_cell(&eqn, &coverage), + "fixture/cell mismatch" + ); rows.push((cell, eqn, coverage, expected)); }; - push( - "Scalar/unfilled".to_string(), - datamodel::Equation::Scalar("NAN".to_string()), - ArmCoverage::CoversEverySlot, - ); - push( - "Scalar/filled".to_string(), - datamodel::Equation::Scalar("3 * x".to_string()), - ArmCoverage::CoversEverySlot, - ); - push( - "ApplyToAll/unfilled".to_string(), - datamodel::Equation::ApplyToAll(dims(), "nan".to_string()), - ArmCoverage::CoversEverySlot, - ); - push( - "ApplyToAll/filled".to_string(), - datamodel::Equation::ApplyToAll(dims(), "3 * x".to_string()), - ArmCoverage::CoversEverySlot, - ); + for (cell, eqn) in [ + ("Scalar/unfilled", datamodel::Equation::Scalar("NAN".into())), + ("Scalar/filled", datamodel::Equation::Scalar("3 * x".into())), + ( + "ApplyToAll/unfilled", + datamodel::Equation::ApplyToAll(dims(), "nan".into()), + ), + ( + "ApplyToAll/filled", + datamodel::Equation::ApplyToAll(dims(), "3 * x".into()), + ), + ] { + let coverage = coverage_for(&eqn, false); + push(cell.to_string(), eqn, coverage); + } for arms in ARM_STATES { for default in DEFAULT_STATES { for coverage_name in COVERAGE_STATES { if arms == "no-arms" && coverage_name == "covers-every-slot" { continue; } - let coverage = if coverage_name == "covers-every-slot" { - ArmCoverage::CoversEverySlot - } else { - ArmCoverage::LeavesSlotsUncovered - }; + let eqn = arrayed_fixture(arms, default); + let coverage = coverage_for(&eqn, coverage_name == "leaves-slots-uncovered"); push( format!("Arrayed/{arms}/{default}/{coverage_name}"), - arrayed_fixture(arms, default), + eqn, coverage, ); } @@ -458,12 +479,43 @@ fn the_decision_table_pins_every_row() { for (cell, eqn, coverage, expected) in decision_table() { assert_eq!( expected, - unfilled_arms(&eqn, coverage), + unfilled_arms(&eqn, &coverage), "decision-table cell {cell:?} disagrees" ); } } +/// An arm the compiler never selects cannot change ANY verdict. +/// +/// Stated as an invariant over the whole table rather than as a fifth axis. An +/// ineffective arm is not a state the verdict depends on -- it is a thing that +/// must not matter -- and "adding one changes nothing, anywhere, whatever its +/// text" is both a stronger claim than 24 extra hand-authored rows and a much +/// smaller one. Both texts are exercised because the NaN one is the whole point +/// (it used to be reported as a slot that stops) and the filled one is the +/// control that would catch a filter keyed on the text instead of the subscript. +#[test] +fn an_arm_the_compiler_never_selects_cannot_change_the_verdict() { + for (cell, eqn, coverage, expected) in decision_table() { + let datamodel::Equation::Arrayed(dim_names, arms, default, live) = &eqn else { + continue; + }; + for text in ["NAN", "7"] { + let mut with_typo = arms.clone(); + with_typo.push(arm("typo", text)); + let perturbed = + datamodel::Equation::Arrayed(dim_names.clone(), with_typo, default.clone(), *live); + // `coverage` lists only the fixture's own arms as effective, so the + // appended one is ignored exactly as the compiler ignores it. + assert_eq!( + expected, + unfilled_arms(&perturbed, &coverage), + "cell {cell:?}: an ignored `typo={text}` arm changed the verdict" + ); + } + } +} + /// Every authored verdict must belong to a real cell. Without this an entry left /// behind by a renamed axis state would sit unused and unnoticed. #[test] @@ -494,7 +546,7 @@ fn every_authored_verdict_names_a_real_cell() { fn the_decision_table_covers_every_cell() { let covered: BTreeSet = decision_table() .iter() - .map(|(_, eqn, coverage, _)| decision_cell(eqn, *coverage)) + .map(|(_, eqn, coverage, _)| decision_cell(eqn, coverage)) .collect(); let expected = expected_cells(); assert_eq!( @@ -832,6 +884,86 @@ fn a_sparse_array_of_unfilled_arms_names_the_arms_not_the_variable() { ); } +/// Every `UnknownElementSubscript` diagnostic the project reports, as +/// `(variable, message)`. The unfilled-equation filter must not silence this +/// one: a subscript naming nothing is still worth telling the modeller about. +fn unknown_subscript_findings(project: &datamodel::Project) -> Vec<(String, String)> { + diagnostics(project) + .into_iter() + .filter_map(|d| match &d.error { + DiagnosticError::Model(e) if e.code == ErrorCode::UnknownElementSubscript => Some(( + d.variable.clone().unwrap_or_default(), + e.get_details().unwrap_or_default().to_string(), + )), + _ => None, + }) + .collect() +} + +/// An arm whose subscript names nothing is not an arm: the compiler ignores it, +/// so it cannot be a slot that simulates as NaN. +/// +/// `typo=NAN` beside arms that already cover `D`. Every declared slot is finite +/// (1 and 2), so there is no unfilled equation to report at all -- but the +/// modeller must still hear that `typo` names nothing, which is the SIBLING +/// advisory's job and is asserted here so the filter cannot silence it. +#[test] +fn an_unknown_subscript_arm_is_not_an_unfilled_equation() { + let project = read_xmile( + r#""#, + r#" + + 1 + 2 + NAN + + "#, + ); + assert_eq!( + Vec::<(String, String, String)>::new(), + unfilled_findings(&project), + "the compiler drops the `typo` arm, so no slot simulates as NaN" + ); + assert_eq!( + 1, + unknown_subscript_findings(&project).len(), + "the unknown-subscript advisory must still fire -- what was wrong was \ + the SECOND warning claiming `typo` simulates as NaN, not this one" + ); + assert_eq!(1.0, final_value(&project, "typo_only[a]")); + assert_eq!(2.0, final_value(&project, "typo_only[b]")); +} + +/// A GENUINE unfilled arm alongside a typo'd one still gets reported -- and only +/// the genuine one is named. +/// +/// This is the half that a filter which simply gave up on any variable carrying +/// an unknown subscript would break, so it is the control that keeps the fix +/// from over-correcting. +#[test] +fn a_genuine_unfilled_arm_survives_a_typod_sibling() { + let project = read_xmile( + r#""#, + r#" + + 1 + NAN + NAN + + "#, + ); + let findings = unfilled_findings(&project); + assert_eq!(1, findings.len(), "{findings:#?}"); + assert!( + findings[0].2.contains("no equation for 'b'") && !findings[0].2.contains("typo"), + "only the arm that reaches a slot may be named: {:?}", + findings[0].2 + ); + assert_eq!(1, unknown_subscript_findings(&project).len()); + assert_eq!(1.0, final_value(&project, "mixed[a]")); + assert!(final_value(&project, "mixed[b]").is_nan()); +} + /// The message may not claim that everything downstream of the unfilled variable /// is NaN, because NaN is absorbing through ARITHMETIC only. /// diff --git a/src/simlin-engine/src/variable.rs b/src/simlin-engine/src/variable.rs index 519f2510e..ccf1f471d 100644 --- a/src/simlin-engine/src/variable.rs +++ b/src/simlin-engine/src/variable.rs @@ -532,25 +532,64 @@ pub(crate) enum UnfilledArms { }, } -/// Whether a variable's explicit arms already name every slot its dimensions -/// declare -- the fact that decides what the slots WITHOUT an arm evaluate to, -/// and therefore whether an EXCEPT default is reachable at all. +/// What the COMPILER will do with an arrayed equation's arms: which of them +/// reach a slot, and whether any slot is left over. /// -/// This cannot be read off a `datamodel::Equation`: it needs the equation's +/// Two facts, one source. `compiler::expand_arrayed_with_hoisting` walks the +/// declared element combinations and looks each one up among the arms; both +/// questions below are that same lookup read in opposite directions, so they are +/// derived together from one pair of sets. +/// +/// Neither can be read off a `datamodel::Equation`: both need the equation's /// dimensions RESOLVED against the project's declarations, which is a database -/// read. So it is computed by the caller (`db::diagnostic::arm_coverage`) and -/// passed in, leaving [`unfilled_arms`] a pure classifier. The alternative -- -/// giving the classifier the resolved element set -- would move a salsa read -/// into a pure function for no gain, since a single bit is all the verdict uses. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub(crate) enum ArmCoverage { - /// Every declared slot has an equation of its own. A scalar or apply-to-all - /// variable is always this: its single formula IS every slot's. - CoversEverySlot, - /// At least one declared slot has no arm naming it, so something else - /// decides that slot's value -- the EXCEPT default if one is live, else the - /// compiler's silent `0`. - LeavesSlotsUncovered, +/// read. So they are computed by the caller (`db::diagnostic::arm_coverage`) and +/// passed in, leaving [`unfilled_arms`] a pure classifier. +// `Debug` is feature-gated because `CanonicalElementName`'s is: an +// unconditional derive here compiles under the test profile (which enables +// `debug-derive`) and fails the release build libsimlin/pysimlin use. +#[cfg_attr(feature = "debug-derive", derive(Debug))] +#[derive(Clone, PartialEq, Eq)] +pub(crate) struct ArmCoverage { + /// The canonical keys of the arms the compiler will actually SELECT. + /// + /// An arm outside this set names no declared element combination, so the + /// compiler ignores it wholesale: it contributes no slot, and its text -- + /// NaN or not -- can never be what any slot evaluates to. Classifying such + /// an arm claimed a variable simulated as NaN when every one of its slots + /// was finite. + effective_arms: HashSet, + /// True when at least one declared slot has no arm naming it, so something + /// else decides that slot's value -- the EXCEPT default if one is live, else + /// the compiler's silent `0`. False for a scalar or apply-to-all variable: + /// its single formula IS every slot's. + leaves_slots_uncovered: bool, +} + +impl ArmCoverage { + pub(crate) fn new( + effective_arms: HashSet, + leaves_slots_uncovered: bool, + ) -> Self { + ArmCoverage { + effective_arms, + leaves_slots_uncovered, + } + } + + /// A scalar or apply-to-all variable: no per-element arms to be ineffective, + /// and its one formula leaves no slot uncovered. + pub(crate) fn whole_variable() -> Self { + ArmCoverage::new(HashSet::new(), false) + } + + pub(crate) fn leaves_slots_uncovered(&self) -> bool { + self.leaves_slots_uncovered + } + + fn is_effective(&self, subscript: &str) -> bool { + self.effective_arms + .contains(&CanonicalElementName::from_raw(subscript)) + } } /// Classify `equation` by which of its arms are unfilled, or `None` when none @@ -565,17 +604,22 @@ pub(crate) enum ArmCoverage { /// [`UnfilledArms::Partial`] names the arms so the message can point at them. /// Either way it is at most ONE finding per variable, never one per element. /// -/// `coverage` is load-bearing for both directions of the arrayed verdict, and -/// both were wrong without it. A NaN EXCEPT default that no slot can reach is -/// not an unfilled equation (arms `a=1,b=2` over `D=[a,b]` with default `NAN` -/// warned, though the compiled model contains no NaN at all), and a sparse -/// array's omitted slots are not covered by its listed arms (arms `a=NAN,b=NAN` -/// over `D=[a,b,c]` claimed the WHOLE variable had no equation, though `[c]` -/// compiles to `0`). Both follow from the same rule: what an armless slot -/// evaluates to is not something the arms' text can answer. +/// `coverage` is load-bearing in three ways, and the verdict was wrong in all +/// three without it -- each on a model that runs and simulates finite today. +/// A NaN EXCEPT default that no slot can reach is not an unfilled equation (arms +/// `a=1,b=2` over `D=[a,b]` with default `NAN`). A sparse array's omitted slots +/// are not covered by its listed arms (arms `a=NAN,b=NAN` over `D=[a,b,c]` +/// claimed the WHOLE variable had no equation, though `[c]` compiles to `0`). +/// And an arm whose subscript names nothing is not an arm at all (`typo=NAN` +/// beside `a=1,b=2` claimed `typo` simulated as NaN, though the compiler drops +/// it and both real slots are finite). +/// +/// All three are the same gap -- between the arms AS WRITTEN and the arms that +/// REACH A SLOT -- which is why they are one parameter and not three. What a +/// slot evaluates to is not something the arms' text alone can answer. pub(crate) fn unfilled_arms( equation: &datamodel::Equation, - coverage: ArmCoverage, + coverage: &ArmCoverage, ) -> Option { use crate::datamodel::Equation; match equation { @@ -585,8 +629,18 @@ pub(crate) fn unfilled_arms( is_nan_literal(s).then_some(UnfilledArms::Whole) } Equation::Arrayed(_, elements, default, has_except_default) => { - let unfilled: Vec = elements + // Only the arms the compiler will SELECT can make a slot NaN. An arm + // it ignores is not evidence about any value, so it is dropped here + // rather than being reported as a slot that stops. + // + // Note this also folds "every arm names nothing" into the no-arms + // case, which is right: such a variable is entirely default- or + // zero-filled, exactly as if it had been written with no arms. + let effective = elements .iter() + .filter(|(subscript, _, _, _)| coverage.is_effective(subscript)); + let effective_count = effective.clone().count(); + let unfilled: Vec = effective .filter(|(_, eqn, _, _)| is_nan_literal(eqn)) .map(|(subscript, _, _, _)| subscript.clone()) .collect(); @@ -599,7 +653,7 @@ pub(crate) fn unfilled_arms( // REACHABLE -- `expand_arrayed_with_hoisting` looks at the default // only for a slot the arms do not name, so when the arms cover the // whole dimension the default is dead code whatever the flag says. - let default_unfilled = coverage == ArmCoverage::LeavesSlotsUncovered + let default_unfilled = coverage.leaves_slots_uncovered() && *has_except_default && default.as_deref().is_some_and(is_nan_literal); // What the slots with no arm of their own evaluate to: the default's @@ -607,12 +661,11 @@ pub(crate) fn unfilled_arms( // `0` -- which is finite, so it is NOT an unfilled equation. (The // silent zero is its own reportable shape, and a deliberately // separate one: GH #905 covers it.) - let uncovered_slots_are_nan = - coverage == ArmCoverage::CoversEverySlot || default_unfilled; + let uncovered_slots_are_nan = !coverage.leaves_slots_uncovered() || default_unfilled; if unfilled.is_empty() && !default_unfilled { None - } else if unfilled.len() == elements.len() && uncovered_slots_are_nan { + } else if unfilled.len() == effective_count && uncovered_slots_are_nan { // Every slot the variable has evaluates to NaN. Some(UnfilledArms::Whole) } else { From 4147832c8f6560b28e73fa589f977bf983973bb4 Mon Sep 17 00:00:00 2001 From: Bobby Powers Date: Tue, 28 Jul 2026 10:58:14 -0700 Subject: [PATCH 14/17] engine: classify the arms the compiler selects Two review findings on the UnfilledEquation advisory, one a defect class and one an interactive-path cost. The class first. Three findings on this diagnostic were the same mistake: an arm shadowed because the others already cover the dimension, an arm whose subscript names nothing, and an arm a later duplicate overrides. Each was patched as a case. They are one question -- which arm does the compiler SELECT for a slot -- so the classifier now takes an ArmSelection and db::diagnostic::arm_selection answers it by performing expand_arrayed_with_hoisting's own walk: build the arm map, then for each declared element combination look it up and fall to the EXCEPT default only on a miss. Arms are identified by index rather than canonical key, which is what makes the duplicate case fall out instead of needing handling: a=NAN and A=1 share a key, so a key-based set marks the key selected and leaves both source entries to be classified, reporting the shadowed arm though the compiled slot holds 1. Note parse_equation's map alone is not the answer -- it is canonical-keyed and last-wins but keys unknown subscripts too, so the drop of those happens only in the declared-slot walk. The cost second. emit_unfilled_equation_warnings ran that walk for every arrayed variable before checking whether any arm could report, so an ordinary per-element array paid an O(declared slots) expansion on every diagnostics pass -- and model_all_diagnostics runs on the interactive path. variable::may_have_unfilled_arms is a cheap superset test that now gates it. Measured with a fresh database per iteration, since within one salsa revision the accumulated diagnostics come from cache and a warm loop measures DFS overhead instead of the emitters: on a 40-variable 20x20x20 model (320,000 declared slots) the pass drops from ~6.77s to ~6.06s, about 2.2us of walk per slot; on C-LEARN, the largest arrayed corpus model at 3,255 slots, ~40ms or 1.5%. The corpus was never going to show this; a modeller's own 3-D array on every keystroke would. Probing the gate found that its superset property was unpinned -- every end-to-end fixture with a NaN default also had a NaN arm, so a version scanning only the arms passed everything. A variable whose only NaN is its EXCEPT default now has a test. The corpus warning count is unchanged at 94 across 4 files: the duplicate shape needs subscripts differing in spelling but not canonically, which no checked-in model has. --- src/simlin-engine/src/db/diagnostic.rs | 83 +++++--- .../src/unfilled_equation_tests.rs | 161 +++++++++++---- src/simlin-engine/src/variable.rs | 187 ++++++++++-------- 3 files changed, 282 insertions(+), 149 deletions(-) diff --git a/src/simlin-engine/src/db/diagnostic.rs b/src/simlin-engine/src/db/diagnostic.rs index b797a832e..83d35885b 100644 --- a/src/simlin-engine/src/db/diagnostic.rs +++ b/src/simlin-engine/src/db/diagnostic.rs @@ -785,52 +785,69 @@ fn resolve_equation_dimensions( .collect() } -/// What the compiler will do with `equation`'s arms: which of them reach a slot, -/// and whether any declared slot is left without one. +/// Which of `equation`'s arms the compiler will SELECT, and whether any declared +/// slot falls past all of them. /// /// `None` when the answer cannot be determined -- an unresolvable dimension name /// (already reported as `BadDimensionName`), which the caller treats as "say /// nothing about this variable" rather than guessing. /// -/// The membership rule is deliberately the COMPILER's, not this file's sibling -/// advisory's. `compiler::expand_arrayed_with_hoisting` selects a slot's -/// equation with `elements.get(&CanonicalElementName::from_raw(combination.join(",")))` -/// and falls back to the EXCEPT default only on a miss, so that lookup IS the -/// fact being asked about; deriving it any other way would be re-deriving the -/// compiler's own decision by a second route. Both facts read that one lookup in -/// opposite directions -- a declared slot with no arm leaves the slot uncovered, -/// an arm matching no declared slot is ignored -- so they come from a single -/// pair of sets rather than from two independent scans. +/// This is `compiler::expand_arrayed_with_hoisting`'s own selection, performed +/// the way it performs it: build the arm map, then walk the declared element +/// combinations looking each one up and falling to the EXCEPT default only on a +/// miss. Two properties come from doing it that way rather than from testing +/// arms individually, and each was a review finding when it was absent -- an arm +/// naming no declared combination is never the answer to any lookup, and among +/// arms sharing a canonical key only the LAST survives the map. /// -/// (`emit_unknown_element_subscript_warnings` accepts the UNION of that rule and +/// The last-wins rule is not restated here: `HashMap` collection over `elements` +/// in declaration order is the same operation `variable::parse_equation` performs +/// to build the AST the compiler actually consumes, so the surviving arm is the +/// same one by construction. `a_shadowed_duplicate_arm_is_not_reported` pins that +/// agreement against a SIMULATED value rather than against this reasoning. +/// +/// (`emit_unknown_element_subscript_warnings` accepts the UNION of this rule and /// the conveyor init-list matcher's, because it is asking a different question /// -- "does ANY consumer resolve this entry?" -- and its rustdoc records that /// the two rules diverge. That advisory keeps reporting an unknown subscript; /// what this one must not do is additionally claim the ignored arm simulates as /// NaN when no slot does.) -fn arm_coverage( +/// +/// Callers must gate this on `variable::may_have_unfilled_arms`: the walk is +/// O(declared slots) and this runs on the interactive path. +fn arm_selection( project_dims: &[crate::datamodel::Dimension], equation: &crate::datamodel::Equation, -) -> Option { +) -> Option { use crate::common::CanonicalElementName; - use crate::variable::ArmCoverage; - use std::collections::HashSet; + use crate::variable::ArmSelection; + use std::collections::{HashMap, HashSet}; let crate::datamodel::Equation::Arrayed(dim_names, elements, _, _) = equation else { - return Some(ArmCoverage::whole_variable()); + return Some(ArmSelection::whole_variable()); }; let dims = resolve_equation_dimensions(project_dims, dim_names)?; - let arm_keys: HashSet = elements + // Keyed by canonical name, valued by POSITION, collected in declaration + // order so a later duplicate overwrites an earlier one -- exactly what + // happens to the arm itself in `parse_equation`'s map. + let arm_positions: HashMap = elements .iter() - .map(|(subscript, _, _, _)| CanonicalElementName::from_raw(subscript)) + .enumerate() + .map(|(index, (subscript, _, _, _))| (CanonicalElementName::from_raw(subscript), index)) .collect(); - let declared_keys: HashSet = - crate::dimensions::SubscriptIterator::new(&dims) - .map(|combination| CanonicalElementName::from_raw(&combination.join(","))) - .collect(); - let leaves_slots_uncovered = declared_keys.iter().any(|key| !arm_keys.contains(key)); - let effective_arms = arm_keys.intersection(&declared_keys).cloned().collect(); - Some(ArmCoverage::new(effective_arms, leaves_slots_uncovered)) + + let mut selected_arms: HashSet = HashSet::new(); + let mut default_is_selected = false; + for combination in crate::dimensions::SubscriptIterator::new(&dims) { + let key = CanonicalElementName::from_raw(&combination.join(",")); + match arm_positions.get(&key) { + Some(index) => { + selected_arms.insert(*index); + } + None => default_is_selected = true, + } + } + Some(ArmSelection::new(selected_arms, default_is_selected)) } /// Is `var`'s own equation only a FALLBACK -- a stand-in for a value a calling @@ -901,7 +918,7 @@ fn equation_is_a_module_input_fallback( /// Variables are visited in sorted-name order so accumulation is deterministic. fn emit_unfilled_equation_warnings(db: &dyn Db, model: SourceModel, project: SourceProject) { use crate::common::{Error, ErrorCode, ErrorKind}; - use crate::variable::{UnfilledArms, unfilled_arms}; + use crate::variable::{UnfilledArms, may_have_unfilled_arms, unfilled_arms}; use salsa::Accumulator; let source_vars = model.variables(db); @@ -916,10 +933,18 @@ fn emit_unfilled_equation_warnings(db: &dyn Db, model: SourceModel, project: Sou continue; } let equation = svar.equation(db); - let Some(coverage) = arm_coverage(project_dims, equation) else { + // Cheap superset test FIRST. `arm_selection` walks the equation's whole + // Cartesian element product, and this pass runs on the interactive path, + // so an ordinary arrayed variable must not pay for a warning it cannot + // emit. Every finding names an arm or the default, so a variable with no + // lone-NaN text among them has nothing to report. + if !may_have_unfilled_arms(equation) { + continue; + } + let Some(selection) = arm_selection(project_dims, equation) else { continue; }; - let Some(unfilled) = unfilled_arms(equation, &coverage) else { + let Some(unfilled) = unfilled_arms(equation, &selection) else { continue; }; // A stock's `equation` is its INITIAL VALUE, not a formula re-evaluated diff --git a/src/simlin-engine/src/unfilled_equation_tests.rs b/src/simlin-engine/src/unfilled_equation_tests.rs index 2caee0958..e095b8298 100644 --- a/src/simlin-engine/src/unfilled_equation_tests.rs +++ b/src/simlin-engine/src/unfilled_equation_tests.rs @@ -17,13 +17,13 @@ use std::collections::BTreeSet; -use crate::common::{CanonicalElementName, ErrorCode}; +use crate::common::ErrorCode; use crate::datamodel; use crate::db::{ Diagnostic, DiagnosticError, DiagnosticSeverity, SimlinDb, collect_all_diagnostics, compile_project_incremental, sync_from_datamodel_incremental, }; -use crate::variable::{ArmCoverage, UnfilledArms, is_nan_literal, unfilled_arms}; +use crate::variable::{ArmSelection, UnfilledArms, is_nan_literal, unfilled_arms}; // --------------------------------------------------------------------------- // The atom: is this equation text nothing but the NaN literal? @@ -129,14 +129,14 @@ fn arrayed(arms: Vec, default: Option<&str>, live: bool) -> datamodel::Equa // The table is their product and `the_decision_table_covers_every_cell` checks // that mechanically, so the row count is DERIVED rather than asserted. -/// How many of the EFFECTIVE per-element arms are unfilled -- effective meaning -/// the compiler would select the arm for some slot. Four states: `unfilled_arms` -/// tests `unfilled.is_empty()` and `unfilled.len() == effective_count`, and those +/// How many of the SELECTED per-element arms are unfilled -- selected meaning +/// the compiler evaluates that arm for some slot. Four states: `unfilled_arms` +/// tests `unfilled.is_empty()` and `unfilled.len() == selected_count`, and those /// two comparisons distinguish exactly these -- with the no-arms case separate /// because it satisfies BOTH (`0 == 0`). /// -/// `no-arms` covers "written with no arms" and "every arm names nothing" alike: -/// after filtering they are the same variable, entirely default- or zero-filled. +/// `no-arms` covers "written with no arms" and "no arm is selected" alike: to the +/// compiled model they are the same variable, entirely default- or zero-filled. const ARM_STATES: [&str; 4] = ["no-arms", "none-unfilled", "some-unfilled", "all-unfilled"]; /// The state of the EXCEPT default. Four states: it is dropped unless @@ -152,7 +152,8 @@ const DEFAULT_STATES: [&str; 4] = [ "live-unfilled-default", ]; -/// Whether the explicit arms already name every slot the dimensions declare. +/// Whether any declared slot falls past all the arms, so the EXCEPT default (or +/// the compiler's silent `0`) decides its value. /// /// This axis is the one no amount of staring at the equation's text can supply /// -- it needs the dimensions RESOLVED -- and leaving it out made the verdict @@ -161,39 +162,35 @@ const DEFAULT_STATES: [&str; 4] = [ /// reported as wholly unfilled. const COVERAGE_STATES: [&str; 2] = ["covers-every-slot", "leaves-slots-uncovered"]; -fn coverage_state(coverage: &ArmCoverage) -> &'static str { - if coverage.leaves_slots_uncovered() { +fn coverage_state(selection: &ArmSelection) -> &'static str { + if selection.default_is_selected() { "leaves-slots-uncovered" } else { "covers-every-slot" } } -/// The coverage an arrayed fixture is classified under: every arm it declares is -/// effective (the fixtures use only real subscripts), plus the coverage state -/// being exercised. +/// The selection an arrayed fixture is classified under: every arm it declares +/// is selected (the fixtures use only real, distinct subscripts), plus the +/// coverage state being exercised. /// -/// Ineffective arms are deliberately NOT a fifth axis. Whether one exists cannot -/// change any verdict once they are filtered, and asserting that INVARIANT over -/// the whole table (`an_arm_the_compiler_never_selects_cannot_change_the_verdict`) -/// is both a stronger claim and a smaller one than doubling the product. -fn coverage_for(eqn: &datamodel::Equation, leaves_slots_uncovered: bool) -> ArmCoverage { +/// Arms the compiler does NOT select are deliberately not a further axis. +/// Whether one exists cannot change any verdict once selection is the input, and +/// asserting that INVARIANT over the whole table +/// (`an_arm_the_compiler_never_selects_cannot_change_the_verdict`) is both a +/// stronger claim and a smaller one than multiplying the product. +fn coverage_for(eqn: &datamodel::Equation, default_is_selected: bool) -> ArmSelection { let datamodel::Equation::Arrayed(_, arms, _, _) = eqn else { - return ArmCoverage::whole_variable(); + return ArmSelection::whole_variable(); }; - ArmCoverage::new( - arms.iter() - .map(|(subscript, _, _, _)| CanonicalElementName::from_raw(subscript)) - .collect(), - leaves_slots_uncovered, - ) + ArmSelection::new((0..arms.len()).collect(), default_is_selected) } /// The cell of the decision space a row occupies, computed from the row's /// EQUATION and COVERAGE rather than from its label -- so a mislabelled row /// cannot fake coverage. `Scalar` and `ApplyToAll` carry one arm and no default, /// so their only axis is whether that arm is unfilled. -fn decision_cell(eqn: &datamodel::Equation, coverage: &ArmCoverage) -> String { +fn decision_cell(eqn: &datamodel::Equation, coverage: &ArmSelection) -> String { match eqn { datamodel::Equation::Scalar(s) | datamodel::Equation::ApplyToAll(_, s) => { let filled = if is_nan_literal(s) { @@ -418,12 +415,12 @@ fn expected_verdicts() -> Vec<(&'static str, Option)> { fn decision_table() -> Vec<( String, datamodel::Equation, - ArmCoverage, + ArmSelection, Option, )> { let verdicts = expected_verdicts(); let mut rows = Vec::new(); - let mut push = |cell: String, eqn: datamodel::Equation, coverage: ArmCoverage| { + let mut push = |cell: String, eqn: datamodel::Equation, coverage: ArmSelection| { let expected = verdicts .iter() .find(|(name, _)| *name == cell) @@ -496,21 +493,38 @@ fn the_decision_table_pins_every_row() { /// control that would catch a filter keyed on the text instead of the subscript. #[test] fn an_arm_the_compiler_never_selects_cannot_change_the_verdict() { - for (cell, eqn, coverage, expected) in decision_table() { + for (cell, eqn, selection, expected) in decision_table() { let datamodel::Equation::Arrayed(dim_names, arms, default, live) = &eqn else { continue; }; for text in ["NAN", "7"] { - let mut with_typo = arms.clone(); - with_typo.push(arm("typo", text)); + // Appended: the arm sits at a position the selection does not name, + // which is how an unknown subscript and a losing duplicate both + // reach the classifier. + let mut appended = arms.clone(); + appended.push(arm("ignored", text)); + let perturbed = + datamodel::Equation::Arrayed(dim_names.clone(), appended, default.clone(), *live); + assert_eq!( + expected, + unfilled_arms(&perturbed, &selection), + "cell {cell:?}: an appended unselected `{text}` arm changed the verdict" + ); + + // Prepended, with every real arm's index shifted by one: this is the + // SHADOWED-DUPLICATE shape specifically, where the arm the compiler + // drops comes BEFORE the one it keeps. Appending alone would never + // exercise it, since the survivor is always the later entry. + let mut prepended = vec![arm("shadowed", text)]; + prepended.extend(arms.iter().cloned()); + let shifted = + ArmSelection::new((1..=arms.len()).collect(), selection.default_is_selected()); let perturbed = - datamodel::Equation::Arrayed(dim_names.clone(), with_typo, default.clone(), *live); - // `coverage` lists only the fixture's own arms as effective, so the - // appended one is ignored exactly as the compiler ignores it. + datamodel::Equation::Arrayed(dim_names.clone(), prepended, default.clone(), *live); assert_eq!( expected, - unfilled_arms(&perturbed, &coverage), - "cell {cell:?}: an ignored `typo={text}` arm changed the verdict" + unfilled_arms(&perturbed, &shifted), + "cell {cell:?}: a shadowed leading `{text}` arm changed the verdict" ); } } @@ -934,6 +948,81 @@ fn an_unknown_subscript_arm_is_not_an_unfilled_equation() { assert_eq!(2.0, final_value(&project, "typo_only[b]")); } +/// A variable whose ONLY NaN is its EXCEPT default is still reported. +/// +/// Every arm is filled; the uncovered slot `c` falls to the default and is the +/// one that stops. This is the case that keeps `may_have_unfilled_arms` honest: +/// that pre-scan exists to skip the slot walk for variables that cannot report, +/// and it must stay a SUPERSET of what gets reported. A version testing only the +/// arms passes every other fixture here -- each of those has a NaN arm too -- and +/// silently loses this finding. +#[test] +fn a_variable_whose_only_nan_is_the_default_is_still_reported() { + let project = read_xmile( + r#""#, + r#" + + NAN + 1 + 2 + + "#, + ); + let findings = unfilled_findings(&project); + assert_eq!(1, findings.len(), "{findings:#?}"); + assert!( + findings[0] + .2 + .contains("no equation for every element with no equation of its own"), + "the default is the unfilled arm here: {:?}", + findings[0].2 + ); + assert_eq!(1.0, final_value(&project, "defaulted[a]")); + assert_eq!(2.0, final_value(&project, "defaulted[b]")); + assert!( + final_value(&project, "defaulted[c]").is_nan(), + "the slot that falls to the NaN default is the one that stops" + ); +} + +/// A duplicate arm that a later one SHADOWS is not reported: only the surviving +/// arm is what the slot evaluates to. +/// +/// `a=NAN` followed by `A=1` canonicalize to one key, and `parse_equation` +/// collects arms into a `HashMap`, so the last wins and the slot holds 1. This +/// is the third instance of one root -- an arm the compiler never selects -- and +/// the reason the classifier now takes the SELECTION rather than testing arms +/// individually. +/// +/// The value assertion is the point. It pins the last-wins rule against what the +/// compiler actually evaluates rather than against the reasoning in +/// `arm_selection`'s doc, so if the two ever disagreed this would fail before +/// the finding count did. +#[test] +fn a_shadowed_duplicate_arm_is_not_reported() { + let project = read_xmile( + r#""#, + r#" + + NAN + 1 + 2 + + "#, + ); + assert_eq!( + 1.0, + final_value(&project, "dup[a]"), + "the LAST arm for a canonical key is the one the compiler keeps" + ); + assert_eq!(2.0, final_value(&project, "dup[b]")); + assert_eq!( + Vec::<(String, String, String)>::new(), + unfilled_findings(&project), + "the shadowed `a=NAN` arm is not what any slot evaluates to" + ); +} + /// A GENUINE unfilled arm alongside a typo'd one still gets reported -- and only /// the genuine one is named. /// diff --git a/src/simlin-engine/src/variable.rs b/src/simlin-engine/src/variable.rs index ccf1f471d..3306c1b18 100644 --- a/src/simlin-engine/src/variable.rs +++ b/src/simlin-engine/src/variable.rs @@ -532,63 +532,60 @@ pub(crate) enum UnfilledArms { }, } -/// What the COMPILER will do with an arrayed equation's arms: which of them -/// reach a slot, and whether any slot is left over. +/// Which of an arrayed equation's arms the COMPILER SELECTS, and whether any +/// declared slot falls past all of them. /// -/// Two facts, one source. `compiler::expand_arrayed_with_hoisting` walks the -/// declared element combinations and looks each one up among the arms; both -/// questions below are that same lookup read in opposite directions, so they are -/// derived together from one pair of sets. +/// This is the structure that closes a defect CLASS rather than an instance. +/// Three separate findings -- an arm shadowed because the others already cover +/// the dimension, an arm whose subscript names nothing, and an arm a later +/// duplicate overrides -- were all one mistake: classifying the arms AS WRITTEN +/// when the only ones that can make a slot NaN are the ones the compiler +/// SELECTS. Asking the selection question directly makes all three consequences +/// instead of cases. /// -/// Neither can be read off a `datamodel::Equation`: both need the equation's +/// The selection is `compiler::expand_arrayed_with_hoisting`'s own two steps, +/// per declared element combination: look the combination's key up among the +/// arms, and fall to the EXCEPT default only on a miss. `db::diagnostic::arm_selection` +/// performs exactly that walk. +/// +/// **Arms are identified by INDEX, not by canonical key**, which is what makes +/// the duplicate case fall out. `a=NAN` followed by `A=1` canonicalizes to one +/// key, and `variable::parse_equation` collects arms into a `HashMap` so the +/// LAST one wins; a key-based set would mark that key selected and leave the +/// classifier examining both source entries, reporting the shadowed `a=NAN` +/// though the compiled slot holds 1. An index names the surviving arm exactly. +/// +/// It cannot be read off a `datamodel::Equation`: the walk needs the equation's /// dimensions RESOLVED against the project's declarations, which is a database -/// read. So they are computed by the caller (`db::diagnostic::arm_coverage`) and -/// passed in, leaving [`unfilled_arms`] a pure classifier. -// `Debug` is feature-gated because `CanonicalElementName`'s is: an -// unconditional derive here compiles under the test profile (which enables -// `debug-derive`) and fails the release build libsimlin/pysimlin use. +/// read. So the caller computes it and [`unfilled_arms`] stays pure. #[cfg_attr(feature = "debug-derive", derive(Debug))] -#[derive(Clone, PartialEq, Eq)] -pub(crate) struct ArmCoverage { - /// The canonical keys of the arms the compiler will actually SELECT. - /// - /// An arm outside this set names no declared element combination, so the - /// compiler ignores it wholesale: it contributes no slot, and its text -- - /// NaN or not -- can never be what any slot evaluates to. Classifying such - /// an arm claimed a variable simulated as NaN when every one of its slots - /// was finite. - effective_arms: HashSet, - /// True when at least one declared slot has no arm naming it, so something - /// else decides that slot's value -- the EXCEPT default if one is live, else - /// the compiler's silent `0`. False for a scalar or apply-to-all variable: - /// its single formula IS every slot's. - leaves_slots_uncovered: bool, +#[derive(Clone, PartialEq, Eq, Default)] +pub(crate) struct ArmSelection { + /// Positions in the equation's `elements` list of the arms that are the + /// selected equation for at least one declared slot. + selected_arms: HashSet, + /// True when at least one declared slot has no arm at all, so its value + /// comes from the EXCEPT default if one is live and from the compiler's + /// silent `0` otherwise. + default_is_selected: bool, } -impl ArmCoverage { - pub(crate) fn new( - effective_arms: HashSet, - leaves_slots_uncovered: bool, - ) -> Self { - ArmCoverage { - effective_arms, - leaves_slots_uncovered, +impl ArmSelection { + pub(crate) fn new(selected_arms: HashSet, default_is_selected: bool) -> Self { + ArmSelection { + selected_arms, + default_is_selected, } } - /// A scalar or apply-to-all variable: no per-element arms to be ineffective, - /// and its one formula leaves no slot uncovered. + /// A scalar or apply-to-all variable: no per-element arms, and its one + /// formula is every slot's, so nothing falls through. pub(crate) fn whole_variable() -> Self { - ArmCoverage::new(HashSet::new(), false) - } - - pub(crate) fn leaves_slots_uncovered(&self) -> bool { - self.leaves_slots_uncovered + ArmSelection::default() } - fn is_effective(&self, subscript: &str) -> bool { - self.effective_arms - .contains(&CanonicalElementName::from_raw(subscript)) + pub(crate) fn default_is_selected(&self) -> bool { + self.default_is_selected } } @@ -604,22 +601,19 @@ impl ArmCoverage { /// [`UnfilledArms::Partial`] names the arms so the message can point at them. /// Either way it is at most ONE finding per variable, never one per element. /// -/// `coverage` is load-bearing in three ways, and the verdict was wrong in all -/// three without it -- each on a model that runs and simulates finite today. -/// A NaN EXCEPT default that no slot can reach is not an unfilled equation (arms -/// `a=1,b=2` over `D=[a,b]` with default `NAN`). A sparse array's omitted slots -/// are not covered by its listed arms (arms `a=NAN,b=NAN` over `D=[a,b,c]` -/// claimed the WHOLE variable had no equation, though `[c]` compiles to `0`). -/// And an arm whose subscript names nothing is not an arm at all (`typo=NAN` -/// beside `a=1,b=2` claimed `typo` simulated as NaN, though the compiler drops -/// it and both real slots are finite). +/// `selection` is what makes the verdict a statement about SLOTS rather than +/// about text. Three review findings were the same mistake -- an arm shadowed by +/// full coverage, an arm naming nothing, an arm overridden by a later duplicate +/// -- and each reported a variable as NaN on a model that simulates finite. They +/// are not three cases here: an arm the compiler does not select is simply not +/// among the arms this looks at. /// -/// All three are the same gap -- between the arms AS WRITTEN and the arms that -/// REACH A SLOT -- which is why they are one parameter and not three. What a -/// slot evaluates to is not something the arms' text alone can answer. +/// Reporting stays at ARM granularity even though the decision is per slot, +/// because the message must stay bounded: a live NaN default covering 400 slots +/// is one phrase, not 400 element names. pub(crate) fn unfilled_arms( equation: &datamodel::Equation, - coverage: &ArmCoverage, + selection: &ArmSelection, ) -> Option { use crate::datamodel::Equation; match equation { @@ -629,43 +623,42 @@ pub(crate) fn unfilled_arms( is_nan_literal(s).then_some(UnfilledArms::Whole) } Equation::Arrayed(_, elements, default, has_except_default) => { - // Only the arms the compiler will SELECT can make a slot NaN. An arm - // it ignores is not evidence about any value, so it is dropped here - // rather than being reported as a slot that stops. + // Only an arm the compiler SELECTS can be what a slot evaluates to. + // Every other arm is inert text: it names no slot, or a duplicate + // overrides it, so its NaN is not a series that stops. // - // Note this also folds "every arm names nothing" into the no-arms - // case, which is right: such a variable is entirely default- or - // zero-filled, exactly as if it had been written with no arms. - let effective = elements + // This also folds "no arm is selected" into the no-arms case, which + // is right -- such a variable is entirely default- or zero-filled, + // exactly as if it had been written with no arms at all. + let selected = elements .iter() - .filter(|(subscript, _, _, _)| coverage.is_effective(subscript)); - let effective_count = effective.clone().count(); - let unfilled: Vec = effective + .enumerate() + .filter(|(index, _)| selection.selected_arms.contains(index)) + .map(|(_, arm)| arm); + let selected_count = selected.clone().count(); + let unfilled: Vec = selected .filter(|(_, eqn, _, _)| is_nan_literal(eqn)) .map(|(subscript, _, _, _)| subscript.clone()) .collect(); // Two independent things have to hold before the default's TEXT - // means anything. It must be LIVE -- `has_except_default` is what - // makes it apply to a slot with no arm, and the MDL converter keeps - // a dead one as round-trip metadata (`mdl::convert::variables`) - // while the compiler consults it only under the flag - // (`compiler::expand_arrayed_with_hoisting`). And it must be - // REACHABLE -- `expand_arrayed_with_hoisting` looks at the default - // only for a slot the arms do not name, so when the arms cover the - // whole dimension the default is dead code whatever the flag says. - let default_unfilled = coverage.leaves_slots_uncovered() + // means anything. It must be SELECTED -- `expand_arrayed_with_hoisting` + // consults it only for a slot no arm names, so with the arms covering + // the dimension it is dead code. And it must be LIVE -- + // `has_except_default` is what lets it apply at all, and the MDL + // converter keeps a dead one as round-trip metadata + // (`mdl::convert::variables`). + let default_unfilled = selection.default_is_selected() && *has_except_default && default.as_deref().is_some_and(is_nan_literal); - // What the slots with no arm of their own evaluate to: the default's - // formula when it is live and reachable, else the compiler's silent - // `0` -- which is finite, so it is NOT an unfilled equation. (The - // silent zero is its own reportable shape, and a deliberately - // separate one: GH #905 covers it.) - let uncovered_slots_are_nan = !coverage.leaves_slots_uncovered() || default_unfilled; + // What the slots with no arm evaluate to: the default's formula when + // it is live, else the compiler's silent `0` -- which is finite, so + // it is NOT an unfilled equation. (That silent zero is its own + // reportable shape and a deliberately separate one: GH #905.) + let slots_past_the_arms_are_nan = !selection.default_is_selected() || default_unfilled; if unfilled.is_empty() && !default_unfilled { None - } else if unfilled.len() == effective_count && uncovered_slots_are_nan { + } else if unfilled.len() == selected_count && slots_past_the_arms_are_nan { // Every slot the variable has evaluates to NaN. Some(UnfilledArms::Whole) } else { @@ -678,6 +671,32 @@ pub(crate) fn unfilled_arms( } } +/// Could `equation` possibly produce an unfilled-equation finding? +/// +/// A cheap superset test, and the ONLY thing an ordinary variable pays. Every +/// finding names either an arm or the EXCEPT default, so if no arm text and no +/// default text is a lone NaN there is nothing to report and the caller can skip +/// the slot walk entirely -- which for an arrayed variable means skipping a +/// Cartesian product over its declared elements. +/// +/// That matters because `db::diagnostic::model_all_diagnostics` runs on the +/// interactive path: without this gate every arrayed variable in a model paid an +/// O(slot-count) allocation on every keystroke, for a warning almost none of +/// them will ever emit. +/// +/// It must stay a SUPERSET of what [`unfilled_arms`] reports, which it is by +/// construction: that function only ever looks at these same texts. +pub(crate) fn may_have_unfilled_arms(equation: &datamodel::Equation) -> bool { + use crate::datamodel::Equation; + match equation { + Equation::Scalar(s) | Equation::ApplyToAll(_, s) => is_nan_literal(s), + Equation::Arrayed(_, elements, default, _) => { + elements.iter().any(|(_, eqn, _, _)| is_nan_literal(eqn)) + || default.as_deref().is_some_and(is_nan_literal) + } + } +} + #[cfg(test)] mod is_lookup_only_tests { use super::*; From 3af1001342bd62d970400653384c0f6a2f16d9a2 Mon Sep 17 00:00:00 2001 From: Bobby Powers Date: Tue, 28 Jul 2026 11:21:27 -0700 Subject: [PATCH 15/17] engine: no unfilled claim when the text is ambiguous The UnfilledEquation advisory read a stored equation of `NAN` and asserted the modeller had written no formula. In a model that declares a variable NAMED `nan` that assertion is undecidable, and it was wrong on a model this branch itself makes reachable: importing MDL `nan = 3` / `b = nan` stores b's equation as the bare text, so `b has no equation` named a formula the modeller did write. This is two changes on this branch meeting rather than a defect in either. Batch 1 deliberately excluded `nan` from the MDL importer's keyword quoting, because quoting it would bind Vensim's `A FUNCTION OF(...)` placeholder -- which we store as `NAN` -- to any like-named variable, and a round-tripped model would compute a value for a variable that has none. That trade is disclosed and pinned by keyword_ident_tests::a_bare_nan_reference_in_mdl_is_still_the_literal. Its consequence is that the two shapes are stored identically. So the diagnostic declines to make a claim: may_have_unfilled_arms returns false when the model declares a variable named `nan`. Note this is not the declared-name resolution rule batch 1 rejected -- that resolved the ambiguity in favour of one reading and would have shipped a wrong value. This ships no claim at all, which is the correct behaviour for a warning whose entire content is an assertion about the model: its value is that a practitioner can trust it and skip the backward hunt through the dependency graph, so a warning that might be false is worse than one that is absent. Only the bare literal is affected, and only within the model that declares the name, since that is the scope a bare reference resolves in; a sub-model's `nan` cannot silence its parent. The corpus count is unchanged at 94 findings across 4 files -- no checked-in model declares a variable named `nan`. Note the cost of declining: in such a model a genuine placeholder now goes unwarned. Recovering it would need import provenance carried in the datamodel. --- src/simlin-engine/src/db/diagnostic.rs | 6 +- .../src/unfilled_equation_tests.rs | 91 +++++++++++++++++++ src/simlin-engine/src/variable.rs | 38 +++++++- 3 files changed, 133 insertions(+), 2 deletions(-) diff --git a/src/simlin-engine/src/db/diagnostic.rs b/src/simlin-engine/src/db/diagnostic.rs index 83d35885b..974cc3443 100644 --- a/src/simlin-engine/src/db/diagnostic.rs +++ b/src/simlin-engine/src/db/diagnostic.rs @@ -927,6 +927,10 @@ fn emit_unfilled_equation_warnings(db: &dyn Db, model: SourceModel, project: Sou let project_dims = project.dimensions(db); let model_name = model.name(db); + // A variable literally named `nan` makes the stored text `NAN` ambiguous -- + // see `may_have_unfilled_arms`. The map is canonically keyed, so this covers + // every spelling (`NaN`, `NAN`, ...) that canonicalizes to the same name. + let nan_names_a_variable = source_vars.contains_key("nan"); for var_name in var_names { let svar = source_vars[var_name]; if equation_is_a_module_input_fallback(db, model, svar) { @@ -938,7 +942,7 @@ fn emit_unfilled_equation_warnings(db: &dyn Db, model: SourceModel, project: Sou // so an ordinary arrayed variable must not pay for a warning it cannot // emit. Every finding names an arm or the default, so a variable with no // lone-NaN text among them has nothing to report. - if !may_have_unfilled_arms(equation) { + if !may_have_unfilled_arms(equation, nan_names_a_variable) { continue; } let Some(selection) = arm_selection(project_dims, equation) else { diff --git a/src/simlin-engine/src/unfilled_equation_tests.rs b/src/simlin-engine/src/unfilled_equation_tests.rs index e095b8298..d81bb8450 100644 --- a/src/simlin-engine/src/unfilled_equation_tests.rs +++ b/src/simlin-engine/src/unfilled_equation_tests.rs @@ -948,6 +948,97 @@ fn an_unknown_subscript_arm_is_not_an_unfilled_equation() { assert_eq!(2.0, final_value(&project, "typo_only[b]")); } +/// A model that declares a variable NAMED `nan` gets no unfilled-equation +/// findings, because in such a model the stored text `NAN` has two readings. +/// +/// This is two of this branch's changes meeting, not a defect in either. Batch 1 +/// deliberately excluded `nan` from the MDL importer's keyword quoting -- quoting +/// it would bind Vensim's `A FUNCTION OF(...)` placeholder to any like-named +/// variable -- and disclosed the residual in +/// `keyword_ident_tests::a_bare_nan_reference_in_mdl_is_still_the_literal`. So +/// `b = nan` here IS a formula the modeller wrote, stored identically to a +/// placeholder they never wrote, and "b has no equation" was false. +/// +/// The assertions bracket the ambiguity from both sides: `b` really is NaN at +/// runtime (batch 1's residual, unchanged by this), so the NaN half of the old +/// message was true and only the "has no equation" half was false -- which is +/// exactly why declining to claim beats guessing. +#[test] +fn a_model_declaring_a_variable_named_nan_gets_no_findings() { + let mdl = concat!( + "nan = 3\n\t~\t\n\t~\t|\n\n", + "b = nan\n\t~\t\n\t~\t|\n\n", + "\\\\\\---/// Sketch information - do not modify anything except names\n" + ); + let project = crate::compat::open_vensim(mdl).expect("MDL must parse"); + assert_eq!( + Vec::<(String, String, String)>::new(), + unfilled_findings(&project), + "`b = nan` is a reference the modeller wrote; we cannot tell it from a \ + placeholder, so we must not claim `b` has no equation" + ); + assert_eq!(3.0, final_value(&project, "nan")); + assert!( + final_value(&project, "b").is_nan(), + "batch 1's disclosed residual: the bare reference still reads as the \ + literal, so the NaN itself was never the false part of the message" + ); +} + +/// The ambiguity is scoped to the MODEL that declares `nan`, because that is the +/// scope in which a bare reference resolves. +/// +/// A sub-model declaring `nan` says nothing about how the main model's `NAN` +/// reads, so the main model's placeholder is still reported. Without this, one +/// oddly-named variable anywhere in a project would silence the diagnostic +/// everywhere in it. +#[test] +fn a_nan_variable_in_another_model_does_not_suppress_this_one() { + let project = read_xmile_with_models( + "", + r#" + NAN + "#, + r#" + 0 + 3 + port + nan + "#, + ); + let findings = unfilled_findings(&project); + assert_eq!( + vec!["marketing".to_string()], + findings.iter().map(|f| f.1.clone()).collect::>(), + "the sub-model's `nan` cannot make the MAIN model's text ambiguous: \ + {findings:#?}" + ); + assert_eq!("main", findings[0].0); +} + +/// The suppression is total within such a model: every equation the decision +/// table says IS reportable stops being reportable. +/// +/// Stated as an invariant over the table rather than as a fifth axis, for the +/// same reason as the unselected-arm invariant: `nan` naming a variable is not a +/// state the verdict varies with, it is a condition under which no claim can be +/// made at all. Thirty-two checks from one loop rather than 32 duplicated rows. +#[test] +fn declaring_nan_suppresses_every_reportable_cell() { + for (cell, eqn, _selection, expected) in decision_table() { + if expected.is_none() { + continue; + } + assert!( + crate::variable::may_have_unfilled_arms(&eqn, false), + "cell {cell:?} reports, so it must pass the gate in an ordinary model" + ); + assert!( + !crate::variable::may_have_unfilled_arms(&eqn, true), + "cell {cell:?} must make no claim when `nan` names a variable" + ); + } +} + /// A variable whose ONLY NaN is its EXCEPT default is still reported. /// /// Every arm is filled; the uncovered slot `c` falls to the default and is the diff --git a/src/simlin-engine/src/variable.rs b/src/simlin-engine/src/variable.rs index 3306c1b18..f991da02e 100644 --- a/src/simlin-engine/src/variable.rs +++ b/src/simlin-engine/src/variable.rs @@ -686,7 +686,43 @@ pub(crate) fn unfilled_arms( /// /// It must stay a SUPERSET of what [`unfilled_arms`] reports, which it is by /// construction: that function only ever looks at these same texts. -pub(crate) fn may_have_unfilled_arms(equation: &datamodel::Equation) -> bool { +/// +/// # `nan_names_a_variable`: declining to make an undecidable claim +/// +/// When the model declares a variable actually NAMED `nan`, the stored text +/// `NAN` has two readings and nothing here can tell them apart, so this returns +/// `false` and the variable is not reported at all. +/// +/// The ambiguity is one we chose. The MDL importer quotes every keyword-shaped +/// variable reference EXCEPT `nan` (`mdl::xmile_compat`'s `quote_reference`), +/// because quoting it would bind Vensim's `A FUNCTION OF(...)` placeholder -- +/// which we store as the text `NAN` -- to any like-named variable, and a +/// round-tripped model would then compute a value for a variable that has none. +/// That trade is deliberate and is pinned by +/// `keyword_ident_tests::a_bare_nan_reference_in_mdl_is_still_the_literal`, +/// which records the residual it leaves: `b = nan` referring to a declared `nan` +/// still reads as the literal. So a model containing both shapes stores them +/// identically, and `b = nan` -- a formula the modeller really did write -- +/// looked exactly like a placeholder they never filled in. +/// +/// Silence is the right answer rather than a guess in either direction. This +/// warning's whole value is that a practitioner can trust it and skip the +/// backward hunt (`crate::float`), so a warning that MIGHT be false is worse +/// than one that is absent. Note this is not the declared-name resolution rule +/// batch 1 rejected: that one resolved the ambiguity in favour of one reading +/// and would have shipped a wrong VALUE. This ships no claim. +/// +/// Scoped to the ambiguity, not to the model: the flag only ever suppresses +/// equations whose text IS the bare literal, which is the only text with two +/// readings. Every other diagnostic in the pass is untouched, and in a model +/// with no variable named `nan` -- every ordinary model -- nothing changes. +pub(crate) fn may_have_unfilled_arms( + equation: &datamodel::Equation, + nan_names_a_variable: bool, +) -> bool { + if nan_names_a_variable { + return false; + } use crate::datamodel::Equation; match equation { Equation::Scalar(s) | Equation::ApplyToAll(_, s) => is_nan_literal(s), From 05a90c4c3884b38f4568f486af0564a0f32b8596 Mon Sep 17 00:00:00 2001 From: Bobby Powers Date: Tue, 28 Jul 2026 11:56:02 -0700 Subject: [PATCH 16/17] engine: classify the parsed equation, not the raw arms Fourth review finding on the UnfilledEquation advisory, and the first FALSE NEGATIVE: an arm with an empty equation was treated as covering its slot, so a live NAN default looked unreachable and nothing was reported while the slot really did simulate as NaN. A spurious warning costs a moment; a missing one costs the backward hunt this diagnostic exists to remove. The three earlier findings were each fixed by re-deriving one more stage of the pipeline between the arms as written and the arms the compiler evaluates. That approach is what failed here -- the hand-built selection was missing a stage, and missed it silently. So this stops re-deriving. unfilled_arms now takes the parsed Ast, which IS the result of that pipeline: empty and unparseable arms dropped, duplicate canonical subscripts collapsed last-wins, dimensions resolved. ArmSelection and arm_selection are deleted; the only stage still performed here is the declared-slot walk, which cannot come from the Ast because the parser does not know about slots, and which is the compiler's own SubscriptIterator/elements.get/default-on-miss sequence. Reading the parse costs nothing: model_all_diagnostics already triggers compile_var_fragment for every variable, which parses under the same module-ident context key, so this is a memo read. The cheap text gate added for the interactive path still runs first, so an ordinary variable reaches neither the parse nor the slot walk. One behaviour change to note: arrayed element names in the message are now canonical and in row-major declared order, because that is how the parsed map is keyed. Recovering the as-written spelling would require re-deriving the last-wins rule to know which spelling survived, which is exactly what this removes. The corpus count is unchanged at 94 findings across 4 files -- no checked-in model contains an empty arm. --- src/simlin-engine/src/db/diagnostic.rs | 100 ++-- .../src/unfilled_equation_tests.rs | 486 ++++++++++-------- src/simlin-engine/src/variable.rs | 183 +++---- 3 files changed, 369 insertions(+), 400 deletions(-) diff --git a/src/simlin-engine/src/db/diagnostic.rs b/src/simlin-engine/src/db/diagnostic.rs index 974cc3443..da3a3eae2 100644 --- a/src/simlin-engine/src/db/diagnostic.rs +++ b/src/simlin-engine/src/db/diagnostic.rs @@ -785,69 +785,28 @@ fn resolve_equation_dimensions( .collect() } -/// Which of `equation`'s arms the compiler will SELECT, and whether any declared -/// slot falls past all of them. +/// The parsed `Ast` for `var`'s equation, or `None` when it has none (an +/// unparseable equation or an unresolvable dimension name, both of which +/// `compile_var_fragment` already reports). /// -/// `None` when the answer cannot be determined -- an unresolvable dimension name -/// (already reported as `BadDimensionName`), which the caller treats as "say -/// nothing about this variable" rather than guessing. +/// This is a memo READ, not a parse. `model_all_diagnostics` triggers +/// `compile_var_fragment` for every variable before this pass runs, and that +/// path parses through `parse_source_variable_with_module_context` under +/// `model_module_ident_context(db, model, project, vec![])` -- the same key used +/// here, so the memo is already populated and shared rather than a second parse +/// under a second key. /// -/// This is `compiler::expand_arrayed_with_hoisting`'s own selection, performed -/// the way it performs it: build the arm map, then walk the declared element -/// combinations looking each one up and falling to the EXCEPT default only on a -/// miss. Two properties come from doing it that way rather than from testing -/// arms individually, and each was a review finding when it was absent -- an arm -/// naming no declared combination is never the answer to any lookup, and among -/// arms sharing a canonical key only the LAST survives the map. -/// -/// The last-wins rule is not restated here: `HashMap` collection over `elements` -/// in declaration order is the same operation `variable::parse_equation` performs -/// to build the AST the compiler actually consumes, so the surviving arm is the -/// same one by construction. `a_shadowed_duplicate_arm_is_not_reported` pins that -/// agreement against a SIMULATED value rather than against this reasoning. -/// -/// (`emit_unknown_element_subscript_warnings` accepts the UNION of this rule and -/// the conveyor init-list matcher's, because it is asking a different question -/// -- "does ANY consumer resolve this entry?" -- and its rustdoc records that -/// the two rules diverge. That advisory keeps reporting an unknown subscript; -/// what this one must not do is additionally claim the ignored arm simulates as -/// NaN when no slot does.) -/// -/// Callers must gate this on `variable::may_have_unfilled_arms`: the walk is -/// O(declared slots) and this runs on the interactive path. -fn arm_selection( - project_dims: &[crate::datamodel::Dimension], - equation: &crate::datamodel::Equation, -) -> Option { - use crate::common::CanonicalElementName; - use crate::variable::ArmSelection; - use std::collections::{HashMap, HashSet}; - - let crate::datamodel::Equation::Arrayed(dim_names, elements, _, _) = equation else { - return Some(ArmSelection::whole_variable()); - }; - let dims = resolve_equation_dimensions(project_dims, dim_names)?; - // Keyed by canonical name, valued by POSITION, collected in declaration - // order so a later duplicate overwrites an earlier one -- exactly what - // happens to the arm itself in `parse_equation`'s map. - let arm_positions: HashMap = elements - .iter() - .enumerate() - .map(|(index, (subscript, _, _, _))| (CanonicalElementName::from_raw(subscript), index)) - .collect(); - - let mut selected_arms: HashSet = HashSet::new(); - let mut default_is_selected = false; - for combination in crate::dimensions::SubscriptIterator::new(&dims) { - let key = CanonicalElementName::from_raw(&combination.join(",")); - match arm_positions.get(&key) { - Some(index) => { - selected_arms.insert(*index); - } - None => default_is_selected = true, - } - } - Some(ArmSelection::new(selected_arms, default_is_selected)) +/// Callers must still gate on `variable::may_have_unfilled_arms` first: this is +/// cheap, but the classification it feeds walks the equation's declared slots. +fn parsed_equation_ast( + db: &dyn Db, + model: SourceModel, + project: SourceProject, + var: SourceVariable, +) -> Option> { + let module_ident_context = model_module_ident_context(db, model, project, vec![]); + let parsed = parse_source_variable_with_module_context(db, var, project, module_ident_context); + parsed.variable.ast().cloned() } /// Is `var`'s own equation only a FALLBACK -- a stand-in for a value a calling @@ -925,7 +884,6 @@ fn emit_unfilled_equation_warnings(db: &dyn Db, model: SourceModel, project: Sou let mut var_names: Vec<&String> = source_vars.keys().collect(); var_names.sort_unstable(); - let project_dims = project.dimensions(db); let model_name = model.name(db); // A variable literally named `nan` makes the stored text `NAN` ambiguous -- // see `may_have_unfilled_arms`. The map is canonically keyed, so this covers @@ -936,19 +894,19 @@ fn emit_unfilled_equation_warnings(db: &dyn Db, model: SourceModel, project: Sou if equation_is_a_module_input_fallback(db, model, svar) { continue; } - let equation = svar.equation(db); - // Cheap superset test FIRST. `arm_selection` walks the equation's whole - // Cartesian element product, and this pass runs on the interactive path, - // so an ordinary arrayed variable must not pay for a warning it cannot - // emit. Every finding names an arm or the default, so a variable with no - // lone-NaN text among them has nothing to report. - if !may_have_unfilled_arms(equation, nan_names_a_variable) { + // Cheap TEXT test FIRST, so an ordinary variable pays only this. The + // classification below walks the equation's declared slots, and this + // pass runs on the interactive path, so an arrayed variable must not pay + // for a warning it cannot emit. Every finding names an arm or the + // default, so a variable with no lone-NaN text among them has nothing to + // report. + if !may_have_unfilled_arms(svar.equation(db), nan_names_a_variable) { continue; } - let Some(selection) = arm_selection(project_dims, equation) else { + let Some(ast) = parsed_equation_ast(db, model, project, svar) else { continue; }; - let Some(unfilled) = unfilled_arms(equation, &selection) else { + let Some(unfilled) = unfilled_arms(&ast) else { continue; }; // A stock's `equation` is its INITIAL VALUE, not a formula re-evaluated diff --git a/src/simlin-engine/src/unfilled_equation_tests.rs b/src/simlin-engine/src/unfilled_equation_tests.rs index d81bb8450..f8dae4aaa 100644 --- a/src/simlin-engine/src/unfilled_equation_tests.rs +++ b/src/simlin-engine/src/unfilled_equation_tests.rs @@ -15,15 +15,16 @@ //! fixture that hand-built a `datamodel::Equation::Scalar("NAN")` would prove //! nothing about that. -use std::collections::BTreeSet; +use std::collections::{BTreeSet, HashMap}; -use crate::common::ErrorCode; +use crate::ast::{Ast, Expr0}; +use crate::common::{CanonicalElementName, ErrorCode}; use crate::datamodel; use crate::db::{ Diagnostic, DiagnosticError, DiagnosticSeverity, SimlinDb, collect_all_diagnostics, compile_project_incremental, sync_from_datamodel_incremental, }; -use crate::variable::{ArmSelection, UnfilledArms, is_nan_literal, unfilled_arms}; +use crate::variable::{UnfilledArms, is_nan_literal, unfilled_arms}; // --------------------------------------------------------------------------- // The atom: is this equation text nothing but the NaN literal? @@ -81,70 +82,105 @@ fn the_atom_rejects_everything_that_is_not_a_lone_nan() { } // --------------------------------------------------------------------------- -// The per-shape decision, enumerated from `datamodel::Equation`'s variants +// The per-shape decision, over the PARSED equation // --------------------------------------------------------------------------- - -/// The `datamodel::Equation` variant `eqn` is, as an EXHAUSTIVE match with no -/// catch-all arm: a new equation shape added to the datamodel is a compile error -/// right here, which is what makes the decision table's coverage checkable -/// rather than asserted. -fn equation_shape(eqn: &datamodel::Equation) -> &'static str { - match eqn { - datamodel::Equation::Scalar(_) => "Scalar", - datamodel::Equation::ApplyToAll(..) => "ApplyToAll", - datamodel::Equation::Arrayed(..) => "Arrayed", +// +// The classifier now takes `Ast`, so these fixtures are parsed ASTs. That +// is deliberate rather than incidental: the four stages between an equation as +// written and the arms the compiler evaluates (empty-arm drop, last-wins over +// duplicate canonical subscripts, dimension resolution, declared-slot lookup) +// are the parser's and the compiler's, and every review finding on this +// diagnostic came from re-deriving one of them by hand. The table pins the +// verdict given a parsed equation; the stages that produce it are pinned by the +// end-to-end tests further down, which go through a real reader. + +/// The `Ast` shape `ast` is, as an EXHAUSTIVE match with no catch-all arm: a new +/// shape is a compile error right here, which is what makes the decision table's +/// coverage checkable rather than asserted. +fn equation_shape(ast: &Ast) -> &'static str { + match ast { + Ast::Scalar(_) => "Scalar", + Ast::ApplyToAll(..) => "ApplyToAll", + Ast::Arrayed(..) => "Arrayed", } } -/// Counted from `datamodel::Equation`: three variants, three shapes. +/// Counted from `ast::Ast`: three variants, three shapes. const EQUATION_SHAPES: [&str; 3] = ["Scalar", "ApplyToAll", "Arrayed"]; -fn dims() -> Vec { - vec!["items".to_string()] +/// Parse one arm's or formula's text the way `variable::parse_equation` does. +/// `None` for text with no expression at all, which is exactly what that +/// function drops before collecting its map. +fn expr(text: &str) -> Option { + Expr0::new(text, crate::lexer::LexerType::Equation).expect("fixture must parse") } -/// One per-element arm of an `Equation::Arrayed`: -/// `(subscript, equation, initial equation, graphical function)`. -type Arm = ( - String, - String, - Option, - Option, -); - -fn arm(subscript: &str, eqn: &str) -> Arm { - (subscript.to_string(), eqn.to_string(), None, None) +fn scalar_ast(text: &str) -> Ast { + Ast::Scalar(expr(text).expect("fixture must have an expression")) } -/// An `Equation::Arrayed` with `default` as its EXCEPT default. `live` is the -/// `has_except_default` flag, which is what makes the default apply to elements -/// with no entry of their own -- a `Some` default with the flag clear is dead. -fn arrayed(arms: Vec, default: Option<&str>, live: bool) -> datamodel::Equation { - datamodel::Equation::Arrayed(dims(), arms, default.map(str::to_string), live) +fn a2a_ast(text: &str) -> Ast { + Ast::ApplyToAll( + test_dims(&["a", "b"]), + expr(text).expect("fixture must have an expression"), + ) +} + +/// Resolved dimensions declaring exactly `elements`, built through the same +/// `From<&datamodel::Dimension>` conversion the parser uses. +fn test_dims(elements: &[&str]) -> Vec { + vec![crate::dimensions::Dimension::from( + &datamodel::Dimension::named( + "d".to_string(), + elements.iter().map(|e| e.to_string()).collect(), + ), + )] } -// THREE axes decide an `Arrayed` verdict. The first two are properties of the -// equation TEXT; the third is not derivable from it at all, which is how the -// first version of this table came to be wrong in both directions at once. -// The table is their product and `the_decision_table_covers_every_cell` checks -// that mechanically, so the row count is DERIVED rather than asserted. - -/// How many of the SELECTED per-element arms are unfilled -- selected meaning -/// the compiler evaluates that arm for some slot. Four states: `unfilled_arms` -/// tests `unfilled.is_empty()` and `unfilled.len() == selected_count`, and those -/// two comparisons distinguish exactly these -- with the no-arms case separate -/// because it satisfies BOTH (`0 == 0`). +/// Build an arrayed `Ast` the way `parse_equation` would: arms keyed +/// canonically with empty ones dropped, over `declared` elements. +fn arrayed_ast( + declared: &[&str], + arms: &[(&str, &str)], + default: Option<&str>, + apply_default_to_missing: bool, +) -> Ast { + let elements: HashMap = arms + .iter() + .filter_map(|(subscript, text)| { + expr(text).map(|e| (CanonicalElementName::from_raw(subscript), e)) + }) + .collect(); + Ast::Arrayed( + test_dims(declared), + elements, + default.and_then(expr), + apply_default_to_missing, + ) +} + +// THREE axes decide an `Arrayed` verdict, and every one of them is now a +// property of the parsed fixture rather than a parameter -- which is why +// `decision_cell` can compute the cell from the `Ast` alone. + +/// How many of the arms a declared slot SELECTS are unfilled. Four states: +/// `unfilled_arms` tests `unfilled.is_empty()` and +/// `unfilled.len() == slots_with_an_arm`, and those two comparisons distinguish +/// exactly these -- with the no-arms case separate because it satisfies BOTH +/// (`0 == 0`). /// -/// `no-arms` covers "written with no arms" and "no arm is selected" alike: to the -/// compiled model they are the same variable, entirely default- or zero-filled. +/// `no-arms` covers every way a variable ends up with no selected arm at all: +/// none written, every arm empty (the parser drops those), or every subscript +/// naming nothing. To the compiled model they are one variable, entirely +/// default- or zero-filled. const ARM_STATES: [&str; 4] = ["no-arms", "none-unfilled", "some-unfilled", "all-unfilled"]; -/// The state of the EXCEPT default. Four states: it is dropped unless -/// `has_except_default` is set, and only then is its text read. +/// The state of the EXCEPT default: absent, present but not applied to missing +/// slots, applied-and-filled, or applied-and-unfilled. /// /// A dead default whose text is FILLED is deliberately not a fifth state: the -/// flag drops the default before its text is ever looked at, so it is the same -/// state as `dead-default` by construction, not by coincidence. +/// flag is checked before the text is read, so it is the same state as +/// `dead-default` by construction, not by coincidence. const DEFAULT_STATES: [&str; 4] = [ "no-default", "dead-default", @@ -154,73 +190,59 @@ const DEFAULT_STATES: [&str; 4] = [ /// Whether any declared slot falls past all the arms, so the EXCEPT default (or /// the compiler's silent `0`) decides its value. -/// -/// This axis is the one no amount of staring at the equation's text can supply -/// -- it needs the dimensions RESOLVED -- and leaving it out made the verdict -/// wrong in both directions on models that run today: a NaN default no slot can -/// reach was reported, and a sparse array whose omitted slots compile to `0` was -/// reported as wholly unfilled. const COVERAGE_STATES: [&str; 2] = ["covers-every-slot", "leaves-slots-uncovered"]; -fn coverage_state(selection: &ArmSelection) -> &'static str { - if selection.default_is_selected() { - "leaves-slots-uncovered" - } else { - "covers-every-slot" - } -} - -/// The selection an arrayed fixture is classified under: every arm it declares -/// is selected (the fixtures use only real, distinct subscripts), plus the -/// coverage state being exercised. -/// -/// Arms the compiler does NOT select are deliberately not a further axis. -/// Whether one exists cannot change any verdict once selection is the input, and -/// asserting that INVARIANT over the whole table -/// (`an_arm_the_compiler_never_selects_cannot_change_the_verdict`) is both a -/// stronger claim and a smaller one than multiplying the product. -fn coverage_for(eqn: &datamodel::Equation, default_is_selected: bool) -> ArmSelection { - let datamodel::Equation::Arrayed(_, arms, _, _) = eqn else { - return ArmSelection::whole_variable(); - }; - ArmSelection::new((0..arms.len()).collect(), default_is_selected) -} - -/// The cell of the decision space a row occupies, computed from the row's -/// EQUATION and COVERAGE rather than from its label -- so a mislabelled row -/// cannot fake coverage. `Scalar` and `ApplyToAll` carry one arm and no default, -/// so their only axis is whether that arm is unfilled. -fn decision_cell(eqn: &datamodel::Equation, coverage: &ArmSelection) -> String { - match eqn { - datamodel::Equation::Scalar(s) | datamodel::Equation::ApplyToAll(_, s) => { - let filled = if is_nan_literal(s) { +/// The cell of the decision space a fixture occupies, computed from the `Ast` +/// itself rather than from a label -- so a mislabelled row cannot fake coverage. +fn decision_cell(ast: &Ast) -> String { + match ast { + Ast::Scalar(e) | Ast::ApplyToAll(_, e) => { + let filled = if crate::variable::is_nan_constant_for_test(e) { "unfilled" } else { "filled" }; - format!("{}/{filled}", equation_shape(eqn)) + format!("{}/{filled}", equation_shape(ast)) } - datamodel::Equation::Arrayed(_, elements, default, live) => { - let unfilled = elements - .iter() - .filter(|(_, e, _, _)| is_nan_literal(e)) - .count(); - let arms = if elements.is_empty() { + Ast::Arrayed(dims, elements, default, live) => { + let mut with_arm = 0usize; + let mut unfilled = 0usize; + let mut uncovered = false; + for combination in crate::dimensions::SubscriptIterator::new(dims) { + let key = CanonicalElementName::from_raw(&combination.join(",")); + match elements.get(&key) { + Some(e) => { + with_arm += 1; + if crate::variable::is_nan_constant_for_test(e) { + unfilled += 1; + } + } + None => uncovered = true, + } + } + let arms = if with_arm == 0 { "no-arms" } else if unfilled == 0 { "none-unfilled" - } else if unfilled == elements.len() { + } else if unfilled == with_arm { "all-unfilled" } else { "some-unfilled" }; - let default = match (default.as_deref(), live) { + let default = match (default.as_ref(), live) { (None, _) => "no-default", (Some(_), false) => "dead-default", - (Some(d), true) if is_nan_literal(d) => "live-unfilled-default", + (Some(d), true) if crate::variable::is_nan_constant_for_test(d) => { + "live-unfilled-default" + } (Some(_), true) => "live-filled-default", }; - format!("Arrayed/{arms}/{default}/{}", coverage_state(coverage)) + let coverage = if uncovered { + "leaves-slots-uncovered" + } else { + "covers-every-slot" + }; + format!("Arrayed/{arms}/{default}/{coverage}") } } } @@ -230,10 +252,8 @@ fn decision_cell(eqn: &datamodel::Equation, coverage: &ArmSelection) -> String { /// **2 (Scalar) + 2 (ApplyToAll) + [(4 arm-states x 4 default-states x 2 /// coverage-states) - 4 impossible] = 32.** /// -/// The four excluded cells are `no-arms` x `covers-every-slot`: zero arms cannot -/// name every slot of a dimension that has at least one. Feeding the classifier -/// that pair would pin behaviour for an input the emitter cannot construct, -/// which is worth less than saying why it is absent. +/// The four excluded cells are `no-arms` x `covers-every-slot`: with no selected +/// arm, every declared slot is by definition uncovered. fn expected_cells() -> BTreeSet { let mut cells = BTreeSet::new(); for shape in ["Scalar", "ApplyToAll"] { @@ -254,16 +274,21 @@ fn expected_cells() -> BTreeSet { cells } -/// Build the arrayed fixture for one `(arms, default)` cell. Generated rather -/// than hand-written so the table cannot drift out of the axis product; the -/// EXPECTED VERDICTS in [`decision_table`] stay authored, which is the half that -/// must not be derived from the implementation. -fn arrayed_fixture(arms: &str, default: &str) -> datamodel::Equation { - let arms = match arms { - "no-arms" => vec![], - "none-unfilled" => vec![arm("a", "1"), arm("b", "2")], - "some-unfilled" => vec![arm("a", "1"), arm("b", "NAN")], - "all-unfilled" => vec![arm("a", "NAN"), arm("b", "nan")], +/// Build the arrayed fixture for one `(arms, default, coverage)` cell. +/// +/// Coverage is expressed by DECLARING a third element the arms do not name, +/// rather than by a flag, because that is how a real sparse array expresses it. +fn arrayed_fixture(arms: &str, default: &str, coverage: &str) -> Ast { + let declared: &[&str] = if coverage == "covers-every-slot" { + &["a", "b"] + } else { + &["a", "b", "c"] + }; + let arms: &[(&str, &str)] = match arms { + "no-arms" => &[], + "none-unfilled" => &[("a", "1"), ("b", "2")], + "some-unfilled" => &[("a", "1"), ("b", "NAN")], + "all-unfilled" => &[("a", "NAN"), ("b", "nan")], other => panic!("unknown arm state {other:?}"), }; let (default, live) = match default { @@ -273,7 +298,7 @@ fn arrayed_fixture(arms: &str, default: &str) -> datamodel::Equation { "live-unfilled-default" => (Some("nan"), true), other => panic!("unknown default state {other:?}"), }; - arrayed(arms, default, live) + arrayed_ast(declared, arms, default, live) } /// The verdict each cell must produce, authored cell by cell. @@ -281,8 +306,9 @@ fn arrayed_fixture(arms: &str, default: &str) -> datamodel::Equation { /// Deliberately a flat list of literals rather than a `match` with grouped or /// wildcard arms: a grouped expectation is a second copy of the rule under test, /// and would agree with a wrong implementation for the same wrong reason. Every -/// entry here was worked out from what the compiled model actually evaluates, -/// slot by slot. +/// entry was worked out from what the compiled model evaluates, slot by slot. +/// +/// Element names are CANONICAL, because that is how the parsed arm map is keyed. #[allow(clippy::type_complexity)] fn expected_verdicts() -> Vec<(&'static str, Option)> { let whole = || Some(UnfilledArms::Whole); @@ -298,8 +324,8 @@ fn expected_verdicts() -> Vec<(&'static str, Option)> { ("Scalar/filled", None), ("ApplyToAll/unfilled", whole()), ("ApplyToAll/filled", None), - // -- COVERS EVERY SLOT. No slot can reach the default, so it is dead - // code whatever its text says and all four default states agree. + // -- COVERS EVERY SLOT. No slot reaches the default, so it is dead code + // whatever its text says and all four default states agree. ("Arrayed/none-unfilled/no-default/covers-every-slot", None), ("Arrayed/none-unfilled/dead-default/covers-every-slot", None), ( @@ -307,8 +333,6 @@ fn expected_verdicts() -> Vec<(&'static str, Option)> { None, ), ( - // P2-1, first half: this warned before the coverage axis existed, - // on a model whose compiled slots hold 1 and 2 and no NaN at all. "Arrayed/none-unfilled/live-unfilled-default/covers-every-slot", None, ), @@ -350,7 +374,6 @@ fn expected_verdicts() -> Vec<(&'static str, Option)> { None, ), ( - // Every slot falls to the default, and the default is unfilled. "Arrayed/no-arms/live-unfilled-default/leaves-slots-uncovered", whole(), ), @@ -383,14 +406,13 @@ fn expected_verdicts() -> Vec<(&'static str, Option)> { partial(&["b"], false), ), ( - // The only cell producing a non-empty element list AND `default`, - // i.e. the message branch that joins the two. + // The only cell producing a non-empty element list AND `default`. "Arrayed/some-unfilled/live-unfilled-default/leaves-slots-uncovered", partial(&["b"], true), ), ( - // P2-1, second half: every ARM is unfilled, but the uncovered slots - // compile to a finite 0, so the variable is not wholly unfilled. + // Every ARM is unfilled, but the uncovered slot compiles to a finite + // 0, so the variable is not wholly unfilled. "Arrayed/all-unfilled/no-default/leaves-slots-uncovered", partial(&["a", "b"], false), ), @@ -412,15 +434,10 @@ fn expected_verdicts() -> Vec<(&'static str, Option)> { /// The decision table: one row per cell, fixtures generated from the axes and /// verdicts taken from [`expected_verdicts`]. #[allow(clippy::type_complexity)] -fn decision_table() -> Vec<( - String, - datamodel::Equation, - ArmSelection, - Option, -)> { +fn decision_table() -> Vec<(String, Ast, Option)> { let verdicts = expected_verdicts(); let mut rows = Vec::new(); - let mut push = |cell: String, eqn: datamodel::Equation, coverage: ArmSelection| { + let mut push = |cell: String, ast: Ast| { let expected = verdicts .iter() .find(|(name, _)| *name == cell) @@ -429,41 +446,27 @@ fn decision_table() -> Vec<( .clone(); // The generated fixture must actually LAND in the cell it is filed // under, or the coverage check below would be measuring labels. - assert_eq!( - cell, - decision_cell(&eqn, &coverage), - "fixture/cell mismatch" - ); - rows.push((cell, eqn, coverage, expected)); + assert_eq!(cell, decision_cell(&ast), "fixture/cell mismatch"); + rows.push((cell, ast, expected)); }; - for (cell, eqn) in [ - ("Scalar/unfilled", datamodel::Equation::Scalar("NAN".into())), - ("Scalar/filled", datamodel::Equation::Scalar("3 * x".into())), - ( - "ApplyToAll/unfilled", - datamodel::Equation::ApplyToAll(dims(), "nan".into()), - ), - ( - "ApplyToAll/filled", - datamodel::Equation::ApplyToAll(dims(), "3 * x".into()), - ), + for (cell, ast) in [ + ("Scalar/unfilled", scalar_ast("NAN")), + ("Scalar/filled", scalar_ast("3 * x")), + ("ApplyToAll/unfilled", a2a_ast("nan")), + ("ApplyToAll/filled", a2a_ast("3 * x")), ] { - let coverage = coverage_for(&eqn, false); - push(cell.to_string(), eqn, coverage); + push(cell.to_string(), ast); } for arms in ARM_STATES { for default in DEFAULT_STATES { - for coverage_name in COVERAGE_STATES { - if arms == "no-arms" && coverage_name == "covers-every-slot" { + for coverage in COVERAGE_STATES { + if arms == "no-arms" && coverage == "covers-every-slot" { continue; } - let eqn = arrayed_fixture(arms, default); - let coverage = coverage_for(&eqn, coverage_name == "leaves-slots-uncovered"); push( - format!("Arrayed/{arms}/{default}/{coverage_name}"), - eqn, - coverage, + format!("Arrayed/{arms}/{default}/{coverage}"), + arrayed_fixture(arms, default, coverage), ); } } @@ -473,10 +476,10 @@ fn decision_table() -> Vec<( #[test] fn the_decision_table_pins_every_row() { - for (cell, eqn, coverage, expected) in decision_table() { + for (cell, ast, expected) in decision_table() { assert_eq!( expected, - unfilled_arms(&eqn, &coverage), + unfilled_arms(&ast), "decision-table cell {cell:?} disagrees" ); } @@ -484,47 +487,34 @@ fn the_decision_table_pins_every_row() { /// An arm the compiler never selects cannot change ANY verdict. /// -/// Stated as an invariant over the whole table rather than as a fifth axis. An -/// ineffective arm is not a state the verdict depends on -- it is a thing that +/// Stated as an invariant over the whole table rather than as a further axis. An +/// unselected arm is not a state the verdict depends on -- it is a thing that /// must not matter -- and "adding one changes nothing, anywhere, whatever its -/// text" is both a stronger claim than 24 extra hand-authored rows and a much -/// smaller one. Both texts are exercised because the NaN one is the whole point -/// (it used to be reported as a slot that stops) and the filled one is the -/// control that would catch a filter keyed on the text instead of the subscript. +/// text" is a stronger claim than duplicating every row, and a much smaller one. +/// +/// The arm is added with a subscript naming no declared element, which is the +/// only way an unselected arm can still be present in the PARSED map: the parser +/// has already dropped empty arms and collapsed duplicates, so those two ways of +/// going unselected cannot reach this classifier at all. That is the point of +/// classifying the parsed form -- two of the four pipeline stages stop being +/// something this code can get wrong. #[test] fn an_arm_the_compiler_never_selects_cannot_change_the_verdict() { - for (cell, eqn, selection, expected) in decision_table() { - let datamodel::Equation::Arrayed(dim_names, arms, default, live) = &eqn else { + for (cell, ast, expected) in decision_table() { + let Ast::Arrayed(dims, elements, default, live) = &ast else { continue; }; for text in ["NAN", "7"] { - // Appended: the arm sits at a position the selection does not name, - // which is how an unknown subscript and a losing duplicate both - // reach the classifier. - let mut appended = arms.clone(); - appended.push(arm("ignored", text)); - let perturbed = - datamodel::Equation::Arrayed(dim_names.clone(), appended, default.clone(), *live); - assert_eq!( - expected, - unfilled_arms(&perturbed, &selection), - "cell {cell:?}: an appended unselected `{text}` arm changed the verdict" + let mut with_extra = elements.clone(); + with_extra.insert( + CanonicalElementName::from_raw("ignored"), + expr(text).unwrap(), ); - - // Prepended, with every real arm's index shifted by one: this is the - // SHADOWED-DUPLICATE shape specifically, where the arm the compiler - // drops comes BEFORE the one it keeps. Appending alone would never - // exercise it, since the survivor is always the later entry. - let mut prepended = vec![arm("shadowed", text)]; - prepended.extend(arms.iter().cloned()); - let shifted = - ArmSelection::new((1..=arms.len()).collect(), selection.default_is_selected()); - let perturbed = - datamodel::Equation::Arrayed(dim_names.clone(), prepended, default.clone(), *live); + let perturbed = Ast::Arrayed(dims.clone(), with_extra, default.clone(), *live); assert_eq!( expected, - unfilled_arms(&perturbed, &shifted), - "cell {cell:?}: a shadowed leading `{text}` arm changed the verdict" + unfilled_arms(&perturbed), + "cell {cell:?}: an unselected `{text}` arm changed the verdict" ); } } @@ -546,21 +536,14 @@ fn every_authored_verdict_names_a_real_cell() { /// The table must cover the decision space EXACTLY: every cell present, and no /// row outside it. /// -/// This is the check that makes the row count derived rather than asserted. -/// Set equality both ways is the point -- a subset test would let a shrinking -/// table pass, and the previous version of this file asserted only that the -/// three `Equation` variants appeared, which cannot see a missing combination -/// of the two `Arrayed` axes. Three cells were in fact missing. -/// -/// What the pieces guarantee together: `equation_shape` and `decision_cell` are -/// catch-all-free matches over `datamodel::Equation`, so a new variant is a -/// compile error in both; the author must then name its cells, and this -/// assertion fails until `expected_cells` and the table agree about them. +/// This is the check that makes the row count derived rather than asserted. Set +/// equality both ways is the point -- a subset test would let a shrinking table +/// pass. #[test] fn the_decision_table_covers_every_cell() { let covered: BTreeSet = decision_table() .iter() - .map(|(_, eqn, coverage, _)| decision_cell(eqn, coverage)) + .map(|(_, ast, _)| decision_cell(ast)) .collect(); let expected = expected_cells(); assert_eq!( @@ -578,11 +561,9 @@ fn the_decision_table_covers_every_cell() { "2 + 2 + (4 arm-states x 4 default-states x 2 coverage-states - 4 impossible)" ); - // The shape list is still worth pinning on its own: it is what the two - // catch-all-free matches above are enumerated FROM. let shapes: BTreeSet<&str> = decision_table() .iter() - .map(|(_, eqn, _, _)| equation_shape(eqn)) + .map(|(_, ast, _)| equation_shape(ast)) .collect(); assert_eq!(BTreeSet::from(EQUATION_SHAPES), shapes); } @@ -1015,26 +996,53 @@ fn a_nan_variable_in_another_model_does_not_suppress_this_one() { assert_eq!("main", findings[0].0); } -/// The suppression is total within such a model: every equation the decision -/// table says IS reportable stops being reportable. +/// The suppression is total: every equation shape that can reach a finding +/// stops being reportable when the model declares a variable named `nan`. /// -/// Stated as an invariant over the table rather than as a fifth axis, for the -/// same reason as the unselected-arm invariant: `nan` naming a variable is not a -/// state the verdict varies with, it is a condition under which no claim can be -/// made at all. Thirty-two checks from one loop rather than 32 duplicated rows. +/// Stated over the GATE rather than over the decision table, because the gate is +/// where the rule lives and the gate reads the datamodel equation while the +/// table reads the parsed one. The four rows are the four ways the gate can say +/// yes -- one per `datamodel::Equation` variant, plus the default-only case that +/// probe O showed was unpinned. #[test] -fn declaring_nan_suppresses_every_reportable_cell() { - for (cell, eqn, _selection, expected) in decision_table() { - if expected.is_none() { - continue; - } +fn declaring_nan_suppresses_every_reportable_shape() { + let dim = || vec!["d".to_string()]; + let elem = |subscript: &str, text: &str| { + ( + subscript.to_string(), + text.to_string(), + None, + None::, + ) + }; + let reportable = [ + ("scalar", datamodel::Equation::Scalar("NAN".to_string())), + ( + "apply-to-all", + datamodel::Equation::ApplyToAll(dim(), "nan".to_string()), + ), + ( + "arrayed arm", + datamodel::Equation::Arrayed(dim(), vec![elem("a", "NAN")], None, false), + ), + ( + "arrayed default only", + datamodel::Equation::Arrayed( + dim(), + vec![elem("a", "1")], + Some("NAN".to_string()), + true, + ), + ), + ]; + for (label, eqn) in reportable { assert!( crate::variable::may_have_unfilled_arms(&eqn, false), - "cell {cell:?} reports, so it must pass the gate in an ordinary model" + "{label} must pass the gate in an ordinary model" ); assert!( !crate::variable::may_have_unfilled_arms(&eqn, true), - "cell {cell:?} must make no claim when `nan` names a variable" + "{label} must make no claim when `nan` names a variable" ); } } @@ -1076,6 +1084,46 @@ fn a_variable_whose_only_nan_is_the_default_is_still_reported() { ); } +/// An arm with an EMPTY equation does not cover its slot: the parser drops it, +/// so the slot falls to the EXCEPT default like any other armless slot. +/// +/// This was the fourth instance of the arms-as-written class and the first FALSE +/// NEGATIVE -- `a` looked covered, so the live `NAN` default looked unreachable +/// and nothing was reported, while `a` really does simulate as NaN. It is fixed +/// not by mirroring the parser's empty-drop but by classifying the parsed form, +/// where the drop has already happened. +#[test] +fn an_empty_arm_does_not_cover_its_slot() { + let project = read_xmile( + r#""#, + r#" + + NAN + + 2 + + "#, + ); + // The premise: the slot really is NaN, so a missing warning is a missing + // one and not a difference of opinion. `b` is the control -- it has a real + // arm, so it is finite and the finding must not name it. + assert!( + final_value(&project, "hollow[a]").is_nan(), + "the empty arm is dropped and the slot takes the NaN default" + ); + assert_eq!(2.0, final_value(&project, "hollow[b]")); + let findings = unfilled_findings(&project); + assert_eq!(1, findings.len(), "{findings:#?}"); + assert_eq!("hollow", findings[0].1); + assert!( + findings[0] + .2 + .contains("no equation for every element with no equation of its own"), + "the slot that stops is the one taking the default: {:?}", + findings[0].2 + ); +} + /// A duplicate arm that a later one SHADOWS is not reported: only the surviving /// arm is what the slot evaluates to. /// diff --git a/src/simlin-engine/src/variable.rs b/src/simlin-engine/src/variable.rs index f991da02e..8d47d0558 100644 --- a/src/simlin-engine/src/variable.rs +++ b/src/simlin-engine/src/variable.rs @@ -532,65 +532,8 @@ pub(crate) enum UnfilledArms { }, } -/// Which of an arrayed equation's arms the COMPILER SELECTS, and whether any -/// declared slot falls past all of them. -/// -/// This is the structure that closes a defect CLASS rather than an instance. -/// Three separate findings -- an arm shadowed because the others already cover -/// the dimension, an arm whose subscript names nothing, and an arm a later -/// duplicate overrides -- were all one mistake: classifying the arms AS WRITTEN -/// when the only ones that can make a slot NaN are the ones the compiler -/// SELECTS. Asking the selection question directly makes all three consequences -/// instead of cases. -/// -/// The selection is `compiler::expand_arrayed_with_hoisting`'s own two steps, -/// per declared element combination: look the combination's key up among the -/// arms, and fall to the EXCEPT default only on a miss. `db::diagnostic::arm_selection` -/// performs exactly that walk. -/// -/// **Arms are identified by INDEX, not by canonical key**, which is what makes -/// the duplicate case fall out. `a=NAN` followed by `A=1` canonicalizes to one -/// key, and `variable::parse_equation` collects arms into a `HashMap` so the -/// LAST one wins; a key-based set would mark that key selected and leave the -/// classifier examining both source entries, reporting the shadowed `a=NAN` -/// though the compiled slot holds 1. An index names the surviving arm exactly. -/// -/// It cannot be read off a `datamodel::Equation`: the walk needs the equation's -/// dimensions RESOLVED against the project's declarations, which is a database -/// read. So the caller computes it and [`unfilled_arms`] stays pure. -#[cfg_attr(feature = "debug-derive", derive(Debug))] -#[derive(Clone, PartialEq, Eq, Default)] -pub(crate) struct ArmSelection { - /// Positions in the equation's `elements` list of the arms that are the - /// selected equation for at least one declared slot. - selected_arms: HashSet, - /// True when at least one declared slot has no arm at all, so its value - /// comes from the EXCEPT default if one is live and from the compiler's - /// silent `0` otherwise. - default_is_selected: bool, -} - -impl ArmSelection { - pub(crate) fn new(selected_arms: HashSet, default_is_selected: bool) -> Self { - ArmSelection { - selected_arms, - default_is_selected, - } - } - - /// A scalar or apply-to-all variable: no per-element arms, and its one - /// formula is every slot's, so nothing falls through. - pub(crate) fn whole_variable() -> Self { - ArmSelection::default() - } - - pub(crate) fn default_is_selected(&self) -> bool { - self.default_is_selected - } -} - -/// Classify `equation` by which of its arms are unfilled, or `None` when none -/// are. +/// Classify a variable's PARSED equation by which of its arms are unfilled, or +/// `None` when none are. /// /// Arms rather than whole variables, deliberately. An arrayed variable's /// elements are separate simulated series, so an unfilled `x[b]` stops `x[b]`'s @@ -601,64 +544,68 @@ impl ArmSelection { /// [`UnfilledArms::Partial`] names the arms so the message can point at them. /// Either way it is at most ONE finding per variable, never one per element. /// -/// `selection` is what makes the verdict a statement about SLOTS rather than -/// about text. Three review findings were the same mistake -- an arm shadowed by -/// full coverage, an arm naming nothing, an arm overridden by a later duplicate -/// -- and each reported a variable as NaN on a model that simulates finite. They -/// are not three cases here: an arm the compiler does not select is simply not -/// among the arms this looks at. +/// # Why this takes the parsed `Ast`, not the `datamodel::Equation` /// -/// Reporting stays at ARM granularity even though the decision is per slot, -/// because the message must stay bounded: a live NaN default covering 400 slots -/// is one phrase, not 400 element names. -pub(crate) fn unfilled_arms( - equation: &datamodel::Equation, - selection: &ArmSelection, -) -> Option { - use crate::datamodel::Equation; - match equation { +/// Four review findings on this diagnostic were the same mistake, each caught +/// one at a time: an arm shadowed because the others cover the dimension, an arm +/// whose subscript names nothing, an arm a later duplicate overrides, and an arm +/// whose equation is EMPTY. Every one is a gap between the arms AS WRITTEN and +/// the arms the compiler EVALUATES, and the first three were each fixed by +/// re-deriving one more stage of that pipeline by hand. The fourth proved the +/// approach wrong: the hand-derived selection was missing a stage, and missing +/// one silently -- it reported nothing where a slot really was NaN. +/// +/// So this no longer re-derives anything. [`parse_equation`] already performs +/// the pipeline, and its `Ast` IS the result: empty and unparseable arms +/// dropped, duplicate canonical subscripts collapsed last-wins, dimensions +/// resolved. The one stage that is not in the `Ast` -- which declared slot takes +/// which arm -- is the `SubscriptIterator` walk below, and it is the compiler's +/// own (`compiler::expand_arrayed_with_hoisting` looks each combination's key up +/// in this same map and falls to the EXCEPT default only on a miss). Nothing +/// here mirrors a stage that exists elsewhere. +/// +/// The consequence for arrayed reporting: element names come out CANONICAL and +/// in row-major declared order, because that is how the map is keyed and how the +/// slots are walked. The as-written spelling is not recoverable from the `Ast`, +/// and asking the datamodel for it would mean re-deriving the last-wins rule to +/// know which spelling survived -- exactly the re-derivation this avoids. +pub(crate) fn unfilled_arms(ast: &Ast) -> Option { + match ast { // One arm covering the whole variable: a scalar formula, or one formula // applied to every element. - Equation::Scalar(s) | Equation::ApplyToAll(_, s) => { - is_nan_literal(s).then_some(UnfilledArms::Whole) + Ast::Scalar(expr) | Ast::ApplyToAll(_, expr) => { + is_nan_constant(expr).then_some(UnfilledArms::Whole) } - Equation::Arrayed(_, elements, default, has_except_default) => { - // Only an arm the compiler SELECTS can be what a slot evaluates to. - // Every other arm is inert text: it names no slot, or a duplicate - // overrides it, so its NaN is not a series that stops. - // - // This also folds "no arm is selected" into the no-arms case, which - // is right -- such a variable is entirely default- or zero-filled, - // exactly as if it had been written with no arms at all. - let selected = elements - .iter() - .enumerate() - .filter(|(index, _)| selection.selected_arms.contains(index)) - .map(|(_, arm)| arm); - let selected_count = selected.clone().count(); - let unfilled: Vec = selected - .filter(|(_, eqn, _, _)| is_nan_literal(eqn)) - .map(|(subscript, _, _, _)| subscript.clone()) - .collect(); - // Two independent things have to hold before the default's TEXT - // means anything. It must be SELECTED -- `expand_arrayed_with_hoisting` - // consults it only for a slot no arm names, so with the arms covering - // the dimension it is dead code. And it must be LIVE -- - // `has_except_default` is what lets it apply at all, and the MDL - // converter keeps a dead one as round-trip metadata - // (`mdl::convert::variables`). - let default_unfilled = selection.default_is_selected() - && *has_except_default - && default.as_deref().is_some_and(is_nan_literal); - // What the slots with no arm evaluate to: the default's formula when - // it is live, else the compiler's silent `0` -- which is finite, so - // it is NOT an unfilled equation. (That silent zero is its own - // reportable shape and a deliberately separate one: GH #905.) - let slots_past_the_arms_are_nan = !selection.default_is_selected() || default_unfilled; + Ast::Arrayed(dims, elements, default, apply_default_to_missing) => { + let mut unfilled: Vec = vec![]; + let mut slots_with_an_arm = 0usize; + let mut default_is_selected = false; + for combination in crate::dimensions::SubscriptIterator::new(dims) { + let key = CanonicalElementName::from_raw(&combination.join(",")); + match elements.get(&key) { + Some(expr) => { + slots_with_an_arm += 1; + if is_nan_constant(expr) { + unfilled.push(key.as_str().to_string()); + } + } + // No arm names this slot, so its value comes from the EXCEPT + // default when one is live and from the compiler's silent + // `0` otherwise. + None => default_is_selected = true, + } + } + let default_unfilled = default_is_selected + && *apply_default_to_missing + && default.as_ref().is_some_and(is_nan_constant); + // The silent `0` is finite, so a slot that falls to it is NOT an + // unfilled equation. (It is its own reportable shape, and a + // deliberately separate one: GH #905.) + let slots_past_the_arms_are_nan = !default_is_selected || default_unfilled; if unfilled.is_empty() && !default_unfilled { None - } else if unfilled.len() == selected_count && slots_past_the_arms_are_nan { + } else if unfilled.len() == slots_with_an_arm && slots_past_the_arms_are_nan { // Every slot the variable has evaluates to NaN. Some(UnfilledArms::Whole) } else { @@ -671,6 +618,22 @@ pub(crate) fn unfilled_arms( } } +/// Is `expr` exactly a NaN constant -- the whole formula, not a NaN inside one? +/// +/// A root-level `Expr0::Const` IS the whole equation: the parser builds no node +/// for parentheses, so this is the parsed twin of [`is_nan_literal`]'s +/// single-token rule and agrees with it on every text that reaches here. +fn is_nan_constant(expr: &Expr0) -> bool { + matches!(expr, Expr0::Const(_, literal, _) if literal.value().is_nan()) +} + +/// [`is_nan_constant`] for the decision table, which must compute a fixture's +/// cell by the same rule the classifier uses. +#[cfg(test)] +pub(crate) fn is_nan_constant_for_test(expr: &Expr0) -> bool { + is_nan_constant(expr) +} + /// Could `equation` possibly produce an unfilled-equation finding? /// /// A cheap superset test, and the ONLY thing an ordinary variable pays. Every From eea33c7e248db9cf2acd33fee69ea2b6fbd1d423 Mon Sep 17 00:00:00 2001 From: Bobby Powers Date: Tue, 28 Jul 2026 12:54:41 -0700 Subject: [PATCH 17/17] engine: an indexed axis has no dim-element spelling MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The ceteris-paribus wrap qualified a resolved subscript element as `dim·elem` whatever kind of dimension the axis was. DimensionsContext::lookup resolves that spelling only for a NAMED dimension, so an index into an indexed axis came out as `d·1`, which resolves to nothing: the PREVIOUS-capture helper holding it failed to compile and both of that target's link scores read a constant 0 behind Assembly warnings. The qualification now goes through qualify_axis_element, the one qualifier, which leaves an indexed axis's position verbatim -- and verbatim is what resolves. The resolution itself was right and is unchanged, which is worth recording because it is easy to read the other way. A quoted numeric-named variable used as an index -- `q["1"]` -- is not a runtime read of that variable: normalize_subscripts3 declines it, but declining routes the subscript to the dynamic lowering, whose first move is Dimension::get_offset, and that parses an indexed axis's identifier text into a position. Measured on a running model, with the variable holding two different values and the result unchanged. Refusing to resolve it instead would make the wrap freeze the index, and a frozen index is a dynamic one -- the partial would read whatever the variable held last step where the equation reads position 1, turning a loud zero into a silent wrong row. DimName.N remains a deliberate gap in the shared resolver, now pinned by a test asserting that shape does not compile at all, with instructions for the day it does. ltm_augment.rs sat exactly on the per-file line cap, so the wrap's subscript-index pass moves to a #[path]-mounted sibling -- a cohesive unit and a pure move. --- src/simlin-engine/src/db/ltm_char_tests.rs | 144 +++++++++ src/simlin-engine/src/ltm_augment.rs | 326 +------------------ src/simlin-engine/src/ltm_augment_index.rs | 353 +++++++++++++++++++++ 3 files changed, 505 insertions(+), 318 deletions(-) create mode 100644 src/simlin-engine/src/ltm_augment_index.rs diff --git a/src/simlin-engine/src/db/ltm_char_tests.rs b/src/simlin-engine/src/db/ltm_char_tests.rs index 91ad6b601..6654df4a5 100644 --- a/src/simlin-engine/src/db/ltm_char_tests.rs +++ b/src/simlin-engine/src/db/ltm_char_tests.rs @@ -2367,3 +2367,147 @@ fn per_element_edge_declines_a_repeated_dimension_target() { "a declined edge must emit no per-element score; got {per_element:?}" ); } + +// --------------------------------------------------------------------------- +// P2-8: an index on an INDEXED axis has no `dim·elem` spelling. +// +// The reported mechanism does not hold, and the measurements are in the test +// below rather than only in the report. `q["1"]` -- a legal quoted +// numeric-named variable used as an index -- is NOT a runtime read: the +// compiler's dynamic-subscript lowering resolves it through +// `Dimension::get_offset`, which for an indexed dimension parses the +// identifier's text, so `q["1"]` and `q[1]` are the same static position. The +// defect at that line was the QUALIFICATION, not the resolution. +// --------------------------------------------------------------------------- + +/// The `q["1"]` fixture: an indexed dimension, a variable legally named `1`, +/// and a target that indexes `q` with it. `index_expr` is the spelling under +/// test. +fn indexed_axis_index_model(index_expr: &str, one_value: &str) -> datamodel::Project { + let mut project = TestProject::new("indexed_axis_index") + .with_sim_time(0.0, 2.0, 1.0) + .aux("\"1\"", one_value, None) + .flow("inflow", "1", None) + .stock("source", "1", &["inflow"], &[], None) + .aux("target", &format!("source + q[{index_expr}]"), None) + .build_datamodel(); + project + .dimensions + .push(datamodel::Dimension::indexed("D".to_string(), 3)); + project + .models + .get_mut(0) + .expect("main model") + .variables + .push(crate::datamodel::Variable::Aux(datamodel::Aux { + ident: "q".to_string(), + equation: crate::datamodel::Equation::Arrayed( + vec!["D".to_string()], + vec![ + ("1".to_string(), "100".to_string(), None, None), + ("2".to_string(), "200".to_string(), None, None), + ("3".to_string(), "300".to_string(), None, None), + ], + None, + false, + ), + documentation: String::new(), + units: None, + gf: None, + ai_state: None, + uid: None, + compat: datamodel::Compat::default(), + })); + project +} + +/// A quoted numeric-named variable used as an index into an INDEXED axis is a +/// STATIC position, and the ceteris-paribus wrap must leave it spellable. +/// +/// Batch 1 made `needs_quoting` leading-digit-aware so a variable named `1` +/// round-trips as `"1"`, which is what makes this shape appear in generated +/// text at all; the two changes meet here. +/// +/// The first half is the measurement that settles what the reference means. A +/// review pass read `compiler::subscript::normalize_subscripts3` -- which does +/// decline a bare identifier on an indexed axis -- and concluded the compiler +/// treats `q["1"]` as a runtime read of the variable. It does not: declining +/// there routes to the DYNAMIC lowering (`compiler::context`), whose first move +/// is `Dimension::get_offset(ident)`, and that parses an indexed axis's +/// identifier text into a position. So `q["1"]` is position 1 no matter what +/// the variable holds -- asserted here by running the model with two different +/// values for it. +/// +/// The second half is the defect that WAS there: the wrap qualified the +/// resolved element as `d·1`, a spelling `DimensionsContext::lookup` resolves +/// only for a NAMED dimension. The capture helper that froze it could not +/// compile, so both link scores read a constant 0 behind `Assembly` warnings. +/// A loud degradation rather than a wrong row -- and the reported fix, refusing +/// to resolve the identifier at all, would have turned it INTO a wrong row by +/// freezing an index the compiler resolves statically (`q[PREVIOUS("1")]` reads +/// whatever the variable held last step; the equation reads `q[1]`). +#[test] +fn quoted_numeric_index_on_an_indexed_axis_is_a_static_position() { + // 1. What the reference MEANS, measured: independent of the variable's value. + for (one_value, label) in [("2", "variable 1 = 2"), ("3", "variable 1 = 3")] { + let project = indexed_axis_index_model("\"1\"", one_value); + assert_eq!( + final_value(&project, "target"), + final_value(&project, "source") + final_value(&project, "q[1]"), + "{label}: `q[\"1\"]` is the static position 1, not a read of the \ + variable named 1" + ); + // Non-vacuity: the candidate rows differ, so a runtime read would show. + assert_ne!(final_value(&project, "q[1]"), final_value(&project, "q[2]")); + assert_ne!(final_value(&project, "q[1]"), final_value(&project, "q[3]")); + } + + // 2. ...so the wrap must spell it the way the equation does, and the + // fragments must compile. `q["1"]` and `q[1]` are the same position, so + // both spellings are checked. + for (index_expr, expected) in [("\"1\"", "PREVIOUS(q[\"1\"])"), ("1", "PREVIOUS(q[1])")] { + let project = indexed_axis_index_model(index_expr, "2"); + let (db, model, source_project) = char_fixture_db(&project); + assert!( + fragment_compile_failures(&db, model, source_project).is_empty(), + "q[{index_expr}]: every link-score fragment must compile; a `d·1` \ + qualification does not resolve on an indexed axis and zeroes the score" + ); + let text = link_score_text( + &db, + model, + source_project, + &["link_score\u{205A}source\u{2192}target"], + ); + assert!( + text.contains(expected), + "q[{index_expr}]: expected {expected:?} in the partial; got: {text}" + ); + assert!( + !text.contains("d\u{B7}1"), + "q[{index_expr}]: an indexed axis has no `dim·elem` spelling; got: {text}" + ); + } +} + +/// `DimName.N` on an indexed axis is unreachable, pinned rather than handled. +/// +/// `compiler::subscript` and `compiler::context` both accept `D.2` as a static +/// position, and `dimensions::resolve_axis_index_name` deliberately does not -- +/// a disclosed conservative gap. It stays a gap because the shape does not +/// compile in the first place: the TARGET's own fragment fails, so no link +/// score for it is ever asked for. If this test starts failing, the gap has +/// become reachable and the resolver needs the `DimName.N` form. +#[test] +fn dim_name_dot_index_does_not_compile_so_the_resolver_gap_is_inert() { + use crate::db::compile_project_incremental; + let project = indexed_axis_index_model("D.2", "2"); + let db = SimlinDb::default(); + let sp = sync_from_datamodel(&db, &project).project; + assert!( + compile_project_incremental(&db, sp, "main").is_err(), + "`q[D.2]` is expected NOT to compile; if it now does, \ + `resolve_axis_index_name` must learn the `DimName.N` static position \ + or the wrap will freeze it and read the wrong row" + ); +} diff --git a/src/simlin-engine/src/ltm_augment.rs b/src/simlin-engine/src/ltm_augment.rs index 260e144fc..2aed4a4d6 100644 --- a/src/simlin-engine/src/ltm_augment.rs +++ b/src/simlin-engine/src/ltm_augment.rs @@ -983,324 +983,6 @@ fn freeze_lookup_table_indices( Expr0::Subscript(ident, indices, loc) } -/// If `index` is a bare identifier that unambiguously names a dimension -/// element (per `dims_ctx`), return its qualified `dimension·element` form; -/// otherwise `None`. -/// -/// Subscript indices that name dimension elements are *element selectors*, -/// not causal references. Treating them like variable references and -/// PREVIOUS-wrapping them (the pre-GH#587 behavior) turns a statically -/// resolvable index into a dynamic expression: the resulting -/// `dep[PREVIOUS(elem)]` needs a helper-aux chain whose innermost helper -/// (`$arg = elem`, a bare element name as an equation) cannot compile, so the -/// link score silently stubs to zero. Qualifying instead keeps the index a -/// compile-time constant: it can never be confused with a variable reference -/// (XMILE forbids dimension/variable name collisions, so `dim·elem` is -/// unambiguous), and `PREVIOUS(dep[dim·elem])` compiles to a direct LoadPrev -/// at the element's slot. -/// -/// Qualification requires knowing *which* dimension the element belongs to, and -/// this helper does not know the subscripted variable's declared dimensions: it -/// qualifies only names that exactly one PROJECT dimension declares -/// (`dimension_uniquely_containing_element`), and names shared by several -/// dimensions keep the conservative wrapping behavior. -/// -/// **That is why this is now a FALLBACK.** When the axis IS known, -/// [`index_axis_verdict`] answers the same question against the indexed -/// variable's own axis and this helper is not consulted at all. Ranging over -/// the project's dimensions instead of the axis's own elements is not merely -/// imprecise -- it disagrees with `compiler::subscript::normalize_subscripts3`, -/// so a variable colliding with an UNRELATED dimension's element name is -/// qualified onto that dimension and the frozen read lands on a slot the -/// `PREVIOUS(target)` anchor never touched. -/// -/// **It still runs on two PRODUCTION paths, not only at the test entry points** -/// -- every `LOOKUP` table index and everything under -/// `generate_per_element_link_equation` -- so the defect above is present tense -/// there. [`index_axis_verdict`]'s rustdoc enumerates the three shapes and says -/// why each reaches this. -fn qualify_element_index( - index: &IndexExpr0, - dims_ctx: Option<&crate::dimensions::DimensionsContext>, -) -> Option { - let ctx = dims_ctx?; - let IndexExpr0::Expr(Expr0::Var(name, loc)) = index else { - return None; - }; - let canonical = canonicalize(name.as_str()); - // Already-qualified `dim·element` references resolve via `lookup`; keep - // them verbatim (they are already static). - if ctx.lookup(&canonical).is_some() { - return Some(index.clone()); - } - let elem = crate::common::CanonicalElementName::from_raw(&canonical); - let dim_name = ctx.dimension_uniquely_containing_element(&elem)?; - Some(IndexExpr0::Expr(Expr0::Var( - RawIdent::new_from_str(&format!("{}\u{B7}{}", dim_name.as_str(), canonical)), - *loc, - ))) -} - -/// What a bare-identifier index NAMES on the axis it indexes -- the engine's own -/// precedence rule, applied to the wrap. -/// -/// `axis` is the declared `Dimension` at this index's position of the -/// subscripted variable, looked up in [`axis_dim_at`]. When it is known, -/// every "is this a selector or a runtime read?" question below is answered by -/// [`crate::dimensions::resolve_axis_index_name`] -- the SAME predicate -/// `compiler::subscript::normalize_subscripts3` implements and that GH #986 -/// unified `ltm_agg::classify_axis_access` and -/// `post_transform::pin_dimension_name_indices` onto. The wrap was the third -/// consumer and was left on two project-wide predicates -/// (`dimension_uniquely_containing_element` and `is_element_of_any_dimension`), -/// which range over EVERY dimension in the project while the compiler ranges -/// over the axis's own declared elements. -/// -/// That disagreement was a wrong NUMBER, not a missed optimization. A model -/// variable whose canonical name happens to be an element of some UNRELATED -/// dimension (a `Scenario = [base, high, low]` beside a variable named `base`) -/// read as a runtime value to the simulation and as a static selector to the -/// wrap, so the ceteris-paribus partial left it LIVE: the "frozen" partial moved -/// with it and the link score reported real influence for an edge with no causal -/// dependence at all. Worse, `qualify_element_index` then rewrote it to -/// `otherdim·name`, naming an element of a dimension the subscripted variable is -/// not declared over -- which still compiles, and reads a different slot than the -/// anchor did. -/// -/// Returning `None` means the axis is unknown and the caller keeps the -/// pre-existing project-wide behaviour. **That is NOT merely the test entry -/// points, and it is NOT conservative**: on a collision the fallback does not -/// decline to answer, it answers wrongly. Three shapes reach it, and two are -/// production: -/// -/// - a `LOOKUP` **table index** -- always, because a graphical-function holder is -/// by construction absent from `dep_dims` (GH #606 keeps it off the dependency -/// graph). See `freeze_lookup_table_indices`, which passes `None` explicitly. -/// - **`generate_per_element_link_equation`**, which threads `dep_dims: None`. -/// - the text-in/text-out test entry points, which have no table to thread. -/// -/// So the wrap is not one consumer of this precedence rule but three call-site -/// families with two different answers. GH #986's unification reaches the -/// arrayed / scalar / A2A / stock-flow emitters and no others; the remaining two -/// are named at their call sites and are their own change. -fn index_axis_verdict( - name: &str, - axis: Option<&crate::dimensions::Dimension>, - iter_ctx: Option<&IteratedDimCtx<'_>>, -) -> Option { - let axis = axis?; - Some(crate::dimensions::resolve_axis_index_name( - name, - axis, - |dim| { - iter_ctx.is_some_and(|ic| { - ic.target_iterated_dims - .iter() - .any(|d| d.eq_ignore_ascii_case(dim)) - }) - }, - )) -} - -/// The declared dimension at index position `pos` of `ident`. -/// -/// Reads `IteratedDimCtx::dep_dims`, the target's dep -> declared-dimensions -/// table the db layer already threads here for the GH #526 other-dep -/// correspondence check. Reusing it rather than adding a second table is what -/// keeps the two questions -- "does this dep's axis correspond positionally?" -/// and "what does this index NAME on that axis?" -- answered from one source; -/// a second table built by a second route is how the wrap and the compiler came -/// to disagree in the first place. -/// -/// A dep absent from the table (a scalar, an implicit/synthetic name, or a -/// caller that threads no `iter_ctx`) yields `None`, and the caller falls back. -fn axis_dim_at<'a>( - ctx: &WrapCtx<'a>, - ident: &Ident, - pos: usize, -) -> Option<&'a crate::dimensions::Dimension> { - ctx.iter_ctx?.dep_dims?.get(ident.as_str())?.get(pos) -} - -fn wrap_index_non_matching_in_previous( - index: IndexExpr0, - ctx: &WrapCtx<'_>, - out: &mut WrapOutcome, - skip_element_qualification: bool, - path: &[u16], - frozen: bool, - axis: Option<&crate::dimensions::Dimension>, -) -> IndexExpr0 { - // Only `dims_ctx` (element/dimension recognition) and `iter_ctx` (the - // iterated-dim-name guard) are read directly here; the full wrap context - // rides `ctx` into the recursive `wrap_non_matching_in_previous` calls. - let &WrapCtx { - dims_ctx, iter_ctx, .. - } = ctx; - // The axis-resolved verdict, when the axis is known. It SUPERSEDES the two - // project-wide element guards below -- see [`index_axis_verdict`]. - let axis_verdict = match &index { - IndexExpr0::Expr(Expr0::Var(name, _)) => index_axis_verdict(name.as_str(), axis, iter_ctx), - _ => None, - }; - if let Some(verdict) = &axis_verdict { - use crate::dimensions::AxisIndexName; - match verdict { - // An element THIS axis declares: a static selector. Qualify it with - // the axis's own dimension -- the only qualification that is right - // for a name several dimensions declare, and the one the pre-GH #986 - // `dimension_uniquely_containing_element` could not produce. - AxisIndexName::Element(elem) => { - if skip_element_qualification { - return index; - } - let IndexExpr0::Expr(Expr0::Var(_, loc)) = &index else { - unreachable!("axis_verdict is Some only for a bare Var index") - }; - return IndexExpr0::Expr(Expr0::Var( - RawIdent::new_from_str(&format!( - "{}\u{B7}{}", - axis.expect("verdict implies an axis").name(), - elem.as_str() - )), - *loc, - )); - } - // A dimension the enclosing equation iterates: an iteration - // reference, left verbatim exactly as the guard below does. - AxisIndexName::IteratedDim => return index, - // Neither: a runtime read. Fall through to the recursive wrap, which - // freezes it if it is a dep. This is the arm the project-wide - // predicates got wrong. - AxisIndexName::Unresolved => {} - } - } - // The project-wide fallbacks, reached only when the axis is UNKNOWN (see - // [`index_axis_verdict`]). They answer the same questions over the project's - // whole element/dimension namespace instead of the axis's own, which is - // sound only when there is no axis to consult. - let axis_unknown = axis_verdict.is_none(); - // An index that unambiguously names a dimension element is an element - // selector, never a causal reference: qualify it and leave it unwrapped - // (GH #587). This must be checked BEFORE the recursive wrap below, which - // would otherwise treat the element name as a dep reference. - // - // `skip_element_qualification` is set on the `PerElement` frozen-live-source - // path (Track A stage 1, finding 1): there the row-pinning lowering owns - // source-index qualification from `from_dims`, so a literal element index is - // left bare here (the element-verbatim guards below still keep it unwrapped) - // and the lowering qualifies it consistently. The recursive wrap still runs - // for a genuinely-dynamic index below, so a frozen `pop[Region, idx]` keeps - // its `PREVIOUS(idx)` lag. - if axis_unknown - && !skip_element_qualification - && let Some(qualified) = qualify_element_index(&index, dims_ctx) - { - return qualified; - } - // An ALREADY-`dim·element`-qualified index is a static element selector - // whatever the qualification mode: it carries its own dimension and resolves - // to a constant offset. `qualify_element_index` returns it verbatim too, but - // that call is suppressed on the paths that own qualification themselves, so - // the guard has to stand on its own. It was latent before GH #984 -- the - // suppressed path only wrapped an ident present in `other_deps`, and no - // variable is named `dim·elem` -- and became reachable when the table-index - // freeze started widening that set with the index's own idents. - if let IndexExpr0::Expr(Expr0::Var(name, _)) = &index - && let Some(ctx) = dims_ctx - && ctx.lookup(&canonicalize(name.as_str())).is_some() - { - return index; - } - // An index that names a dimension element which *cannot* be qualified - // (declared by multiple dimensions at different positions, e.g. C-LEARN's - // region elements) is still left verbatim rather than PREVIOUS-wrapped. - // Like `qualify_element_index` above, this ranges over the whole project and - // so runs only when the axis is unknown -- see `index_axis_verdict`. - // Wrapping it would make the subscript dynamic (`dep[PREVIOUS(elem)]`), - // forcing a synthesized helper aux per call site -- the dominant residual - // helper source on large arrayed models (GH #654) -- and is also - // semantically wrong for a genuinely-dynamic index (the index would be - // read from two steps ago instead of one). The downstream parse decides: - // a non-shadowed element compiles to a static subscript (direct - // LoadPrev), a genuinely-dynamic index still synthesizes its helper - // there, with single-lag semantics. - if axis_unknown - && let IndexExpr0::Expr(Expr0::Var(name, _)) = &index - && let Some(ctx) = dims_ctx - && ctx.is_element_of_any_dimension(&crate::common::CanonicalElementName::from_raw( - &canonicalize(name.as_str()), - )) - { - return index; - } - // An index that names a DIMENSION (`matrix[D1, c1]`'s `D1`, - // `SUM(matrix[State, *])`'s `State` -- the iterated-dim reference form) - // is a dimension selector, never a causal reference (GH #759). The two - // guards above cover dimension *elements*; a dimension *name* is - // neither an element nor qualifiable, so it previously fell through to - // the recursive wrap whenever a caller's (over-collected) dep set - // contained it: the frozen reference became `dep[PREVIOUS(d1), ..]`, - // whose PREVIOUS-capture helper cannot compile, silently stubbing the - // score to 0. Leave it verbatim -- the A2A expansion resolves it per - // element downstream, exactly as in the target's own equation. The - // `iter_ctx` leg covers callers without a project dims context (the - // iterated/source dims are dimension names by construction). - if axis_unknown && let IndexExpr0::Expr(Expr0::Var(name, _)) = &index { - let canonical = canonicalize(name.as_str()); - let names_project_dim = - dims_ctx.is_some_and(|ctx| ctx.is_dimension_name(canonical.as_ref())); - let names_iterated_dim = iter_ctx.is_some_and(|ctx| { - ctx.target_iterated_dims - .iter() - .chain(ctx.source_dim_names.iter()) - .any(|d| d.as_str() == canonical.as_ref()) - }); - if names_project_dim || names_iterated_dim { - return index; - } - } - // Indices are inner content of a live reference (or of a - // PREVIOUS-wrapped one); a `live_source` occurrence reachable only - // through an index is not the live reference itself, so do not - // capture it -- pass a throwaway live-ref sink. The GH #526 - // `other_dep_mismatch` doom DOES propagate: an index-nested mismatched - // collapse dooms the changed-first partial just the same. - let mut idx_out = WrapOutcome::default(); - // Everything below here contributes to an INDEX value, so every freeze - // performed under it takes the un-lagged operand as its first-DT initial - // value rather than the desugared `0`, which is out of range for a 1-based - // subscript (GH #975; see [`freeze_at_previous`]). - let index_ctx = WrapCtx { - in_subscript_index: true, - ..*ctx - }; - let ctx = &index_ctx; - // The index expression is at `path` (the walk pushes the index position - // before descending); a `Range`'s two operands are children 0 and 1 of - // that, mirroring `walk_all_in_expr`. - let result = match index { - IndexExpr0::Expr(e) => IndexExpr0::Expr(wrap_non_matching_in_previous( - e, - ctx, - &mut idx_out, - path, - frozen, - )), - IndexExpr0::Range(l, r, loc) => IndexExpr0::Range( - wrap_non_matching_in_previous(l, ctx, &mut idx_out, &child_path(path, 0), frozen), - wrap_non_matching_in_previous(r, ctx, &mut idx_out, &child_path(path, 1), frozen), - loc, - ), - other => other, - }; - out.other_dep_mismatch |= idx_out.other_dep_mismatch; - // A live-source subscript nested inside an index (`other[from[x]]`) desyncs - // the same way; propagate its loud miss flag out of the throwaway sink. - out.missing_occurrence |= idx_out.missing_occurrence; - result -} - /// A parse failure in a ceteris-paribus partial-equation builder. /// /// The ceteris-paribus PREVIOUS-wrapping transform ([`wrap_non_matching_in_previous`]) @@ -5984,6 +5666,14 @@ fn generate_nonlinear_partial( } } +/// The wrap's subscript-INDEX pass, in its own file only to keep this one under +/// the project line-count lint. Re-exported so callers keep naming these +/// `crate::ltm_augment::*`. +#[path = "ltm_augment_index.rs"] +mod index_pass; + +use index_pass::{axis_dim_at, wrap_index_non_matching_in_previous}; + #[cfg(test)] #[path = "ltm_augment_tests.rs"] mod tests; diff --git a/src/simlin-engine/src/ltm_augment_index.rs b/src/simlin-engine/src/ltm_augment_index.rs new file mode 100644 index 000000000..20a68d757 --- /dev/null +++ b/src/simlin-engine/src/ltm_augment_index.rs @@ -0,0 +1,353 @@ +// Copyright 2026 The Simlin Authors. All rights reserved. +// Use of this source code is governed by the Apache License, +// Version 2.0, that can be found in the LICENSE file. + +//! The ceteris-paribus wrap's SUBSCRIPT-INDEX pass: what happens to the things +//! between the brackets, as opposed to the reference the brackets hang off. +//! +//! A subscript index is the one position where "is this a causal reference?" has +//! a different answer than everywhere else in an equation. A bare identifier +//! there may be an element selector, a dimension name the apply-to-all expansion +//! resolves per element, or a genuine variable read -- and only the last is a +//! ceteris-paribus dependency. Guessing wrong is expensive in both directions: +//! freezing a selector emits a `PREVIOUS`-capture helper that cannot compile +//! (a silently-zeroed score), and leaving a read live breaks the isolation the +//! partial exists to express. +//! +//! Everything that decides that question lives here: [`index_axis_verdict`] (the +//! axis-aware verdict, which supersedes the project-wide guards when the axis is +//! known), [`qualify_element_index`] (the project-wide element guard), and +//! [`wrap_index_non_matching_in_previous`] itself. +//! +//! Split out of `ltm_augment.rs` only to keep that file under the project +//! line-count lint; included via `#[path]`, so `super::*` resolves the parent's +//! private items and callers keep naming these `crate::ltm_augment::*`. + +use crate::ast::{Expr0, IndexExpr0}; +use crate::common::{Canonical, Ident, RawIdent, canonicalize}; + +use super::post_transform::qualify_axis_element; +use super::{IteratedDimCtx, WrapCtx, WrapOutcome, child_path, wrap_non_matching_in_previous}; + +/// If `index` is a bare identifier that unambiguously names a dimension +/// element (per `dims_ctx`), return its qualified `dimension·element` form; +/// otherwise `None`. +/// +/// Subscript indices that name dimension elements are *element selectors*, +/// not causal references. Treating them like variable references and +/// PREVIOUS-wrapping them (the pre-GH#587 behavior) turns a statically +/// resolvable index into a dynamic expression: the resulting +/// `dep[PREVIOUS(elem)]` needs a helper-aux chain whose innermost helper +/// (`$arg = elem`, a bare element name as an equation) cannot compile, so the +/// link score silently stubs to zero. Qualifying instead keeps the index a +/// compile-time constant: it can never be confused with a variable reference +/// (XMILE forbids dimension/variable name collisions, so `dim·elem` is +/// unambiguous), and `PREVIOUS(dep[dim·elem])` compiles to a direct LoadPrev +/// at the element's slot. +/// +/// Qualification requires knowing *which* dimension the element belongs to, and +/// this helper does not know the subscripted variable's declared dimensions: it +/// qualifies only names that exactly one PROJECT dimension declares +/// (`dimension_uniquely_containing_element`), and names shared by several +/// dimensions keep the conservative wrapping behavior. +/// +/// **That is why this is now a FALLBACK.** When the axis IS known, +/// [`index_axis_verdict`] answers the same question against the indexed +/// variable's own axis and this helper is not consulted at all. Ranging over +/// the project's dimensions instead of the axis's own elements is not merely +/// imprecise -- it disagrees with `compiler::subscript::normalize_subscripts3`, +/// so a variable colliding with an UNRELATED dimension's element name is +/// qualified onto that dimension and the frozen read lands on a slot the +/// `PREVIOUS(target)` anchor never touched. +/// +/// **It still runs on two PRODUCTION paths, not only at the test entry points** +/// -- every `LOOKUP` table index and everything under +/// `generate_per_element_link_equation` -- so the defect above is present tense +/// there. [`index_axis_verdict`]'s rustdoc enumerates the three shapes and says +/// why each reaches this. +fn qualify_element_index( + index: &IndexExpr0, + dims_ctx: Option<&crate::dimensions::DimensionsContext>, +) -> Option { + let ctx = dims_ctx?; + let IndexExpr0::Expr(Expr0::Var(name, loc)) = index else { + return None; + }; + let canonical = canonicalize(name.as_str()); + // Already-qualified `dim·element` references resolve via `lookup`; keep + // them verbatim (they are already static). + if ctx.lookup(&canonical).is_some() { + return Some(index.clone()); + } + let elem = crate::common::CanonicalElementName::from_raw(&canonical); + let dim_name = ctx.dimension_uniquely_containing_element(&elem)?; + Some(IndexExpr0::Expr(Expr0::Var( + RawIdent::new_from_str(&format!("{}\u{B7}{}", dim_name.as_str(), canonical)), + *loc, + ))) +} + +/// What a bare-identifier index NAMES on the axis it indexes -- the engine's own +/// precedence rule, applied to the wrap. +/// +/// `axis` is the declared `Dimension` at this index's position of the +/// subscripted variable, looked up in [`axis_dim_at`]. When it is known, +/// every "is this a selector or a runtime read?" question below is answered by +/// [`crate::dimensions::resolve_axis_index_name`] -- the SAME predicate +/// `compiler::subscript::normalize_subscripts3` implements and that GH #986 +/// unified `ltm_agg::classify_axis_access` and +/// `post_transform::pin_dimension_name_indices` onto. The wrap was the third +/// consumer and was left on two project-wide predicates +/// (`dimension_uniquely_containing_element` and `is_element_of_any_dimension`), +/// which range over EVERY dimension in the project while the compiler ranges +/// over the axis's own declared elements. +/// +/// That disagreement was a wrong NUMBER, not a missed optimization. A model +/// variable whose canonical name happens to be an element of some UNRELATED +/// dimension (a `Scenario = [base, high, low]` beside a variable named `base`) +/// read as a runtime value to the simulation and as a static selector to the +/// wrap, so the ceteris-paribus partial left it LIVE: the "frozen" partial moved +/// with it and the link score reported real influence for an edge with no causal +/// dependence at all. Worse, `qualify_element_index` then rewrote it to +/// `otherdim·name`, naming an element of a dimension the subscripted variable is +/// not declared over -- which still compiles, and reads a different slot than the +/// anchor did. +/// +/// Returning `None` means the axis is unknown and the caller keeps the +/// pre-existing project-wide behaviour. **That is NOT merely the test entry +/// points, and it is NOT conservative**: on a collision the fallback does not +/// decline to answer, it answers wrongly. Three shapes reach it, and two are +/// production: +/// +/// - a `LOOKUP` **table index** -- always, because a graphical-function holder is +/// by construction absent from `dep_dims` (GH #606 keeps it off the dependency +/// graph). See `freeze_lookup_table_indices`, which passes `None` explicitly. +/// - **`generate_per_element_link_equation`**, which threads `dep_dims: None`. +/// - the text-in/text-out test entry points, which have no table to thread. +/// +/// So the wrap is not one consumer of this precedence rule but three call-site +/// families with two different answers. GH #986's unification reaches the +/// arrayed / scalar / A2A / stock-flow emitters and no others; the remaining two +/// are named at their call sites and are their own change. +fn index_axis_verdict( + name: &str, + axis: Option<&crate::dimensions::Dimension>, + iter_ctx: Option<&IteratedDimCtx<'_>>, +) -> Option { + let axis = axis?; + Some(crate::dimensions::resolve_axis_index_name( + name, + axis, + |dim| { + iter_ctx.is_some_and(|ic| { + ic.target_iterated_dims + .iter() + .any(|d| d.eq_ignore_ascii_case(dim)) + }) + }, + )) +} + +/// The declared dimension at index position `pos` of `ident`. +/// +/// Reads `IteratedDimCtx::dep_dims`, the target's dep -> declared-dimensions +/// table the db layer already threads here for the GH #526 other-dep +/// correspondence check. Reusing it rather than adding a second table is what +/// keeps the two questions -- "does this dep's axis correspond positionally?" +/// and "what does this index NAME on that axis?" -- answered from one source; +/// a second table built by a second route is how the wrap and the compiler came +/// to disagree in the first place. +/// +/// A dep absent from the table (a scalar, an implicit/synthetic name, or a +/// caller that threads no `iter_ctx`) yields `None`, and the caller falls back. +pub(super) fn axis_dim_at<'a>( + ctx: &WrapCtx<'a>, + ident: &Ident, + pos: usize, +) -> Option<&'a crate::dimensions::Dimension> { + ctx.iter_ctx?.dep_dims?.get(ident.as_str())?.get(pos) +} + +pub(super) fn wrap_index_non_matching_in_previous( + index: IndexExpr0, + ctx: &WrapCtx<'_>, + out: &mut WrapOutcome, + skip_element_qualification: bool, + path: &[u16], + frozen: bool, + axis: Option<&crate::dimensions::Dimension>, +) -> IndexExpr0 { + // Only `dims_ctx` (element/dimension recognition) and `iter_ctx` (the + // iterated-dim-name guard) are read directly here; the full wrap context + // rides `ctx` into the recursive `wrap_non_matching_in_previous` calls. + let &WrapCtx { + dims_ctx, iter_ctx, .. + } = ctx; + // The axis-resolved verdict, when the axis is known. It SUPERSEDES the two + // project-wide element guards below -- see [`index_axis_verdict`]. + let axis_verdict = match &index { + IndexExpr0::Expr(Expr0::Var(name, _)) => index_axis_verdict(name.as_str(), axis, iter_ctx), + _ => None, + }; + if let Some(verdict) = &axis_verdict { + use crate::dimensions::AxisIndexName; + match verdict { + // An element THIS axis declares: a static selector. Qualify it with + // the axis's own dimension -- the only qualification that is right + // for a name several dimensions declare, and the one the pre-GH #986 + // `dimension_uniquely_containing_element` could not produce. + // + // Through `qualify_axis_element`, the ONE qualifier, rather than + // spelling `dim·elem` again here: an INDEXED axis has no such + // spelling (`DimensionsContext::lookup` resolves it only for a NAMED + // dimension), so this used to emit `d·1` for a position and the + // capture helper that froze it could not compile -- two link scores + // reading a constant 0 behind `Assembly` warnings. That helper leaves + // an indexed axis's position verbatim, which is what resolves. + AxisIndexName::Element(elem) => { + if skip_element_qualification { + return index; + } + let IndexExpr0::Expr(Expr0::Var(_, loc)) = &index else { + unreachable!("axis_verdict is Some only for a bare Var index") + }; + let axis = axis.expect("verdict implies an axis"); + return IndexExpr0::Expr(Expr0::Var( + RawIdent::new_from_str(&qualify_axis_element(elem.as_str(), axis)), + *loc, + )); + } + // A dimension the enclosing equation iterates: an iteration + // reference, left verbatim exactly as the guard below does. + AxisIndexName::IteratedDim => return index, + // Neither: a runtime read. Fall through to the recursive wrap, which + // freezes it if it is a dep. This is the arm the project-wide + // predicates got wrong. + AxisIndexName::Unresolved => {} + } + } + // The project-wide fallbacks, reached only when the axis is UNKNOWN (see + // [`index_axis_verdict`]). They answer the same questions over the project's + // whole element/dimension namespace instead of the axis's own, which is + // sound only when there is no axis to consult. + let axis_unknown = axis_verdict.is_none(); + // An index that unambiguously names a dimension element is an element + // selector, never a causal reference: qualify it and leave it unwrapped + // (GH #587). This must be checked BEFORE the recursive wrap below, which + // would otherwise treat the element name as a dep reference. + // + // `skip_element_qualification` is set on the `PerElement` frozen-live-source + // path (Track A stage 1, finding 1): there the row-pinning lowering owns + // source-index qualification from `from_dims`, so a literal element index is + // left bare here (the element-verbatim guards below still keep it unwrapped) + // and the lowering qualifies it consistently. The recursive wrap still runs + // for a genuinely-dynamic index below, so a frozen `pop[Region, idx]` keeps + // its `PREVIOUS(idx)` lag. + if axis_unknown + && !skip_element_qualification + && let Some(qualified) = qualify_element_index(&index, dims_ctx) + { + return qualified; + } + // An ALREADY-`dim·element`-qualified index is a static element selector + // whatever the qualification mode: it carries its own dimension and resolves + // to a constant offset. `qualify_element_index` returns it verbatim too, but + // that call is suppressed on the paths that own qualification themselves, so + // the guard has to stand on its own. It was latent before GH #984 -- the + // suppressed path only wrapped an ident present in `other_deps`, and no + // variable is named `dim·elem` -- and became reachable when the table-index + // freeze started widening that set with the index's own idents. + if let IndexExpr0::Expr(Expr0::Var(name, _)) = &index + && let Some(ctx) = dims_ctx + && ctx.lookup(&canonicalize(name.as_str())).is_some() + { + return index; + } + // An index that names a dimension element which *cannot* be qualified + // (declared by multiple dimensions at different positions, e.g. C-LEARN's + // region elements) is still left verbatim rather than PREVIOUS-wrapped. + // Like `qualify_element_index` above, this ranges over the whole project and + // so runs only when the axis is unknown -- see `index_axis_verdict`. + // Wrapping it would make the subscript dynamic (`dep[PREVIOUS(elem)]`), + // forcing a synthesized helper aux per call site -- the dominant residual + // helper source on large arrayed models (GH #654) -- and is also + // semantically wrong for a genuinely-dynamic index (the index would be + // read from two steps ago instead of one). The downstream parse decides: + // a non-shadowed element compiles to a static subscript (direct + // LoadPrev), a genuinely-dynamic index still synthesizes its helper + // there, with single-lag semantics. + if axis_unknown + && let IndexExpr0::Expr(Expr0::Var(name, _)) = &index + && let Some(ctx) = dims_ctx + && ctx.is_element_of_any_dimension(&crate::common::CanonicalElementName::from_raw( + &canonicalize(name.as_str()), + )) + { + return index; + } + // An index that names a DIMENSION (`matrix[D1, c1]`'s `D1`, + // `SUM(matrix[State, *])`'s `State` -- the iterated-dim reference form) + // is a dimension selector, never a causal reference (GH #759). The two + // guards above cover dimension *elements*; a dimension *name* is + // neither an element nor qualifiable, so it previously fell through to + // the recursive wrap whenever a caller's (over-collected) dep set + // contained it: the frozen reference became `dep[PREVIOUS(d1), ..]`, + // whose PREVIOUS-capture helper cannot compile, silently stubbing the + // score to 0. Leave it verbatim -- the A2A expansion resolves it per + // element downstream, exactly as in the target's own equation. The + // `iter_ctx` leg covers callers without a project dims context (the + // iterated/source dims are dimension names by construction). + if axis_unknown && let IndexExpr0::Expr(Expr0::Var(name, _)) = &index { + let canonical = canonicalize(name.as_str()); + let names_project_dim = + dims_ctx.is_some_and(|ctx| ctx.is_dimension_name(canonical.as_ref())); + let names_iterated_dim = iter_ctx.is_some_and(|ctx| { + ctx.target_iterated_dims + .iter() + .chain(ctx.source_dim_names.iter()) + .any(|d| d.as_str() == canonical.as_ref()) + }); + if names_project_dim || names_iterated_dim { + return index; + } + } + // Indices are inner content of a live reference (or of a + // PREVIOUS-wrapped one); a `live_source` occurrence reachable only + // through an index is not the live reference itself, so do not + // capture it -- pass a throwaway live-ref sink. The GH #526 + // `other_dep_mismatch` doom DOES propagate: an index-nested mismatched + // collapse dooms the changed-first partial just the same. + let mut idx_out = WrapOutcome::default(); + // Everything below here contributes to an INDEX value, so every freeze + // performed under it takes the un-lagged operand as its first-DT initial + // value rather than the desugared `0`, which is out of range for a 1-based + // subscript (GH #975; see [`freeze_at_previous`]). + let index_ctx = WrapCtx { + in_subscript_index: true, + ..*ctx + }; + let ctx = &index_ctx; + // The index expression is at `path` (the walk pushes the index position + // before descending); a `Range`'s two operands are children 0 and 1 of + // that, mirroring `walk_all_in_expr`. + let result = match index { + IndexExpr0::Expr(e) => IndexExpr0::Expr(wrap_non_matching_in_previous( + e, + ctx, + &mut idx_out, + path, + frozen, + )), + IndexExpr0::Range(l, r, loc) => IndexExpr0::Range( + wrap_non_matching_in_previous(l, ctx, &mut idx_out, &child_path(path, 0), frozen), + wrap_non_matching_in_previous(r, ctx, &mut idx_out, &child_path(path, 1), frozen), + loc, + ), + other => other, + }; + out.other_dep_mismatch |= idx_out.other_dep_mismatch; + // A live-source subscript nested inside an index (`other[from[x]]`) desyncs + // the same way; propagate its loud miss flag out of the throwaway sink. + out.missing_occurrence |= idx_out.missing_occurrence; + result +}