Skip to content

Flows: dedicated dedup / exactly-once node (the real 'never do this twice' primitive) #5263

Description

@graycyrus

Why (proven, not theoretical)

Shipping the memory node (#5226) and testing it live exposed that dedup-via-memory-node is fundamentally broken:

  • The builder produced the textbook recall·flow → condition → remember·flow graph, but it failed at run — remember keyed on =item.html_url, which an intermediate transform node had dropped → null key.
  • Deeper: semantic recall cannot do exact keyed membership. shape_recall_result returns {scope,query,results:[...]} with store-internal UUIDs and no found field. Three reviewers (Greptile P1 ×2, Codex) independently confirmed the recipe "never suppresses anything — every item re-processed every run."
  • So the dedup recipe was removed from the builder prompt in feat(flows): host adapter for the memory node — in-graph memory read + flow-scoped write #5227 (guard test now asserts its absence), deferring exact dedup to this dedicated primitive.

What: NodeKind::Dedup (14th kind)

A single node the author drops once — where the item still has its ID — that owns both the lookup and the commit internally, so there is no fragile key to thread through the graph.

  • Config: { "key": "=item.id" } — an =-expression yielding a stable per-item key.
  • Filter (mid-run, in the node): unseen keys pass through and are written tentative to the durable per-flow flow_state KV; already-committed keys are dropped. Null/empty resolved key → pass through unrecorded (fail-open, never silently drop).
  • Commit-on-success (host, on FlowRunFinished): run succeeds → tentative keys merge into the committed seen-set; run fails/cancels → tentative keys released, so those items retry next run. A failed action therefore never marks an item done.

Commit granularity

Run-level (the engine has no per-item success isolation — one item's failure fails the node). True exactly-once for single-action flows (the newsletter: one digest → one send); at-least-once for flows with N separate per-item side-effects (safe direction: retry, never silently drop). Documented, not hidden.

Delivery (follows the just-merged memory-node pattern)

  1. tinyflows (vendor): NodeKind::Dedup + DedupNode executor (filter + tentative marking via StateStore) + catalog.rs (13→14) + validate.rs (require key) + caps/mock.rs + tests (incl. failed-action-does-not-commit).
  2. host: DedupCommitSubscriber on FlowRunFinished (commit/release tentative), kv_delete in flows/store.rs, node_contracts.rs overlay (13→14), builder-prompt teaching (a CORRECT dedup recipe this time — one node), about_app, submodule bump.
  3. frontend: lib/flows/{types,nodeKindMeta}.ts (13→14), a dedupFields config panel (the key expression), i18n ×14.

Supersedes the originally-shelved #5224 — now justified by empirical evidence.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions