demo: add at-least-once DS simulation with HLC/VC stats

[dexcompiler]

Summary

Adds a new distributed-systems console demo that simulates at-least-once message delivery with realistic network faults while showcasing Clockworks’ time and causality features.

What’s included:

New demo command: ds-atleastonce

• Single-process, deterministic simulation driven by SimulatedTimeProvider

• Fault injection: drop / duplicate / reorder / delay

• Idempotent consumer behavior via an inbox-style dedupe (MessageId)

• Retry behavior via Timeouts.CreateTimeoutHandle(...) + TimeoutStatistics

• Messages carry both:

1. HLC timestamps (via HlcCoordinator + HlcStatistics)
2. Vector clocks (via VectorClockCoordinator + VectorClockStatistics)

• Explicit vector-clock concurrency detection (IsConcurrentWith(VectorClock)) with rolling-window sampling printed in snapshots

Instrumentation / Output

Periodic snapshots print:

• SimulatedTimeProviderStatistics (timer/scheduler pressure)
• network messaging counters (sent/delivered/dropped/duplicated/reordered + concurrent-pair samples)
• per-node HlcStatistics, VectorClockStatistics, and TimeoutStatistics
• per-node inbox processed/deduped counts

Usage:

dotnet run --project demo/Clockworks.Demo -- ds-atleastonce

Optional knobs: --orders=, --maxSteps=, --tickMs=, --printEvery=, --seed=, --drop=, --dup=, --reorder=, --maxDelayMs=

[dexcompiler]

@cursoragent review this pr

[cursor]

High-level take

This PR is a strong demo addition: it clearly exercises at-least-once delivery (dup/drop/reorder/delay), showcases deterministic simulation via SimulatedTimeProvider, and uses the library’s strengths (HLC/VC/Timeouts) with useful periodic snapshots. The “single-process but realistic faults” framing is a good fit for a console demo.

What looks solid

• Determinism: the --seed= knob + simulated time makes runs reproducible, which is perfect for debugging and for demoing causality.
• Correct at-least-once story: retries + possible duplicates + an inbox-style dedupe (MessageId) communicates the core contract well.
• Instrumentation is on point: per-node HlcStatistics, VectorClockStatistics, TimeoutStatistics plus global network counters and simulated-time pressure is exactly what readers will want.
• Concurrency visibility: explicitly sampling IsConcurrentWith(...) pairs is a nice way to make “VC ≠ total order” tangible.

Things I’d double-check / potential footguns

• Inbox dedupe growth: if the simulation can run long (--maxSteps= high), make sure the dedupe set is bounded (LRU/rolling window) or at least called out as “demo-only, unbounded memory”.
• MessageId uniqueness: ensure IDs are unique across all senders and retries (e.g., (producerNode, orderId)), and that retry doesn’t accidentally generate a new ID (which would defeat dedupe).
• Retry timer lifecycle: verify timeout handles are cancelled/disposed when an ack is observed, so TimeoutStatistics doesn’t count “ghost” timers and you don’t keep scheduling pointless callbacks.
• Fault injection semantics:
  • If you “duplicate”, make sure it duplicates the same message (same MessageId, timestamps, VC) rather than creating a new logical send.
  • If you “reorder”, make sure you’re reordering within a queue in a way that’s still deterministic under the seed.
• Snapshot consistency: consider printing nodes/messages in a stable order so snapshots diff cleanly between runs.

Suggested polish (nice-to-have)

• Help/usage discoverability: ensure ds-atleastonce shows up in the demo’s command list / --help, and that the optional knobs describe units (ms, steps, probabilities as 0–1, etc.).
• Safety rails on knobs: validate ranges (drop/dup/reorder in ([0,1]), tickMs > 0, printEvery > 0, maxDelayMs >= 0) and fail fast with a clear message.
• One deterministic “golden run”: even a lightweight test (or a documented expected snapshot excerpt for --seed=...) would help prevent accidental behavior drift.

Bottom line

I’m in favor of merging: the demo is cohesive, showcases the right primitives, and the stats/output design makes the causality story easy to see. The main thing I’d want addressed (or explicitly documented) is dedupe memory bounds + timeout cancellation correctness under retries/acks.

 

[dexcompiler]

@copilot review this pr

[Copilot]

@dexcompiler I've opened a new pull request, #39, to work on those changes. Once the pull request is ready, I'll request review from you.

[dexcompiler]

@cursoragent copilot fixed some issues listed here: #39 and I've merged in the changes

[cursor]

Nice — thanks for following up and merging the fixes from PR #39.

A couple quick things I’d sanity-check now that those changes are in:

• Inbox dedupe: confirm it’s bounded (or clearly documented as unbounded demo behavior) and that MessageId remains stable across retries.
• Timeout lifecycle: verify retry timeout handles are cancelled/disposed on ack to avoid “ghost” retries and misleading TimeoutStatistics.
• Fault injection semantics: duplication should preserve the same logical message (same MessageId/timestamps/VC), and reorder should stay deterministic under --seed.

If those are covered in the merged fixes, I’m happy with this PR as-is.

 

[dexcompiler]

@copilot cursor agent worked on some other issues and pushed commits to this branch: https://github.com/dexcompiler/Clockworks/tree/demo/ds-atleastonce-idempotency-fixes

Can you review the fixes and open a PR to merge them into the original feature branch: demo/ds-protocols-engineering ?

[Copilot]

@dexcompiler I've opened a new pull request, #40, to work on those changes. Once the pull request is ready, I'll request review from you.