You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Shipping the memory node (#5226) and testing it live exposed that dedup-via-memory-node is fundamentally broken:
The builder produced the textbook recall·flow → condition → remember·flow graph, but it failed at run — remember keyed on =item.html_url, which an intermediate transform node had dropped → null key.
Deeper: semantic recall cannot do exact keyed membership.shape_recall_result returns {scope,query,results:[...]} with store-internal UUIDs and no found field. Three reviewers (Greptile P1 ×2, Codex) independently confirmed the recipe "never suppresses anything — every item re-processed every run."
A single node the author drops once — where the item still has its ID — that owns both the lookup and the commit internally, so there is no fragile key to thread through the graph.
Config:{ "key": "=item.id" } — an =-expression yielding a stable per-item key.
Filter (mid-run, in the node): unseen keys pass through and are written tentative to the durable per-flow flow_state KV; already-committed keys are dropped. Null/empty resolved key → pass through unrecorded (fail-open, never silently drop).
Commit-on-success (host, on FlowRunFinished): run succeeds → tentative keys merge into the committed seen-set; run fails/cancels → tentative keys released, so those items retry next run. A failed action therefore never marks an item done.
Commit granularity
Run-level (the engine has no per-item success isolation — one item's failure fails the node). True exactly-once for single-action flows (the newsletter: one digest → one send); at-least-once for flows with N separate per-item side-effects (safe direction: retry, never silently drop). Documented, not hidden.
Delivery (follows the just-merged memory-node pattern)
host:DedupCommitSubscriber on FlowRunFinished (commit/release tentative), kv_delete in flows/store.rs, node_contracts.rs overlay (13→14), builder-prompt teaching (a CORRECT dedup recipe this time — one node), about_app, submodule bump.
frontend:lib/flows/{types,nodeKindMeta}.ts (13→14), a dedupFields config panel (the key expression), i18n ×14.
Supersedes the originally-shelved #5224 — now justified by empirical evidence.
Why (proven, not theoretical)
Shipping the
memorynode (#5226) and testing it live exposed that dedup-via-memory-node is fundamentally broken:recall·flow → condition → remember·flowgraph, but it failed at run —rememberkeyed on=item.html_url, which an intermediate transform node had dropped → null key.recallcannot do exact keyed membership.shape_recall_resultreturns{scope,query,results:[...]}with store-internal UUIDs and nofoundfield. Three reviewers (Greptile P1 ×2, Codex) independently confirmed the recipe "never suppresses anything — every item re-processed every run."memorynode — in-graph memory read + flow-scoped write #5227 (guard test now asserts its absence), deferring exact dedup to this dedicated primitive.What:
NodeKind::Dedup(14th kind)A single node the author drops once — where the item still has its ID — that owns both the lookup and the commit internally, so there is no fragile key to thread through the graph.
{ "key": "=item.id" }— an=-expression yielding a stable per-item key.flow_stateKV; already-committed keys are dropped. Null/empty resolved key → pass through unrecorded (fail-open, never silently drop).FlowRunFinished): run succeeds → tentative keys merge into the committed seen-set; run fails/cancels → tentative keys released, so those items retry next run. A failed action therefore never marks an item done.Commit granularity
Run-level (the engine has no per-item success isolation — one item's failure fails the node). True exactly-once for single-action flows (the newsletter: one digest → one send); at-least-once for flows with N separate per-item side-effects (safe direction: retry, never silently drop). Documented, not hidden.
Delivery (follows the just-merged memory-node pattern)
NodeKind::Dedup+DedupNodeexecutor (filter + tentative marking viaStateStore) +catalog.rs(13→14) +validate.rs(requirekey) +caps/mock.rs+ tests (incl. failed-action-does-not-commit).DedupCommitSubscriberonFlowRunFinished(commit/release tentative),kv_deleteinflows/store.rs,node_contracts.rsoverlay (13→14), builder-prompt teaching (a CORRECT dedup recipe this time — one node),about_app, submodule bump.lib/flows/{types,nodeKindMeta}.ts(13→14), adedupFieldsconfig panel (the key expression), i18n ×14.Supersedes the originally-shelved #5224 — now justified by empirical evidence.