feat(durable): journal writer actor, sealed backend, local durable.db backend (#4946)#4968
Merged
Merged
Conversation
… backend (#4946) Add the durable persistence engine to zeph-durable (spec-064 C3): - LocalBackend owns a dedicated durable.db pool (INV-14, schema via zeph_db::run_migrations), implements the Journal trait (append with RETURNING seq, full and ranged reads, finalize via begin_write, prune stub), AEAD-seals payload-bearing entries through an injected Option<Arc<dyn PayloadCipher>> with the entry location bound as AAD, and stamps a keyed-BLAKE3 row HMAC over control entries when a key is set. Enforces max_payload on both write and read (INV-11). - JournalWriter background actor decouples writes from the calling path: buffered appends group-commit on a flush interval (dropped with WARN under backpressure), exactly-once appends flush all causally-preceding entries before committing (INV-4) and return their JournalSeq over a oneshot bounded by journal_ack_timeout_ms (INV-12, FR-DE-11), and the writer resumes from MAX(seq) on restart (FR-DE-12). Emits the durable.journal.writer.queue_depth gauge per commit cycle. - Sealed ExecutionBackend trait + BackendCapabilities + DurableBackendEnum enum dispatch keep the backend surface closed with no Box<dyn> on the hot path. open/resolve_promise/due_timers join the sealed trait with C4/C5. Promise, timer, and checkpoint journal entries fail closed with DurableError::UnsupportedEntryKind until the promise/timer/retention layers land. All SQL uses sql!() and dialect-agnostic i64/Vec<u8> column types; compiles under sqlite, postgres, and both. 50 unit/integration tests pass.
2b0a186 to
cb868e3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements C3 of the durable execution epic (#4707, spec-064): the persistence engine behind a durable execution. Builds on C2 (#4967, now merged). Part of milestone M28.
Closes #4946.
What lands
LocalBackend(backend/local.rs) — owns a dedicateddurable.dbpool (INV-14; schema applied viazeph_db::run_migrations, no.sql/sqlx::migrate!in this crate). Implements theJournaltrait:append(single,RETURNING seq),read_execution,read_execution_range,finalize(viabegin_write), and aprunestub. AEAD-seals payload-bearing entries through an injectedOption<Arc<dyn PayloadCipher>>with the entry location bound as associated data; stamps a keyed-BLAKE3 row HMAC over control entries when an HMAC key is configured. Enforcesmax_payload_byteson both write and read (INV-11, fail-closed).JournalWriteractor +JournalWriterHandle(writer.rs) — decouples DB writes from the calling path:journal_flush_interval_msand drop with aWARNon a full channel (re-run safely on resume);JournalSeqover a oneshot bounded byjournal_ack_timeout_ms(INV-12 / FR-DE-11 — never blocks indefinitely);MAX(seq)on (re)start (FR-DE-12);durable.journal.writer.queue_depthgauge per commit cycle.ExecutionBackend+BackendCapabilities+DurableBackendEnum(backend.rs) — enum dispatch, noBox<dyn>on the hot path; the trait is sealed (un-implementable outside the crate), soopen/resolve_promise/due_timersjoin it in C4/C5 without breaking callers.Scope boundaries
DurableError::UnsupportedEntryKind— their persistence (and any schema additions) lands with the promise/timer (C5) and retention layers, which thedurable_journalcolumns do not yet cover.DurableContext/DurableStep, the replay cursor, and the daemonTaskSupervisorwiring are C4.Spans
durable.backend.open,durable.journal.append,durable.journal.read,durable.journal.read_segment,durable.journal.finalize,durable.journal.prune, and thedurable.journal.writer.queue_depthgauge.Tests
50 tests pass. Backend + writer integration tests are SQLite-gated (
:memory:, mirroringzeph-scheduler); the handle ACK-timeout / buffered-drop tests are dialect-agnostic.cargo nextest run -p zeph-durable --no-default-features --features sqlite— 50 passedcargo check -p zeph-durableundersqlite,postgres, and bothcargo clippy --all-targets -- -D warningsclean under sqlite and postgrescargo test --doc(23) +RUSTDOCFLAGS="--deny rustdoc::broken_intra_doc_links" cargo doc --no-depscleanRUSTFLAGS="-D warnings" cargo check --all-targetsclean; downstreamzeph-corestill compilesAcceptance criteria
durable.dbpool + migrations viazeph_db::run_migrationsAppendAckedtimes out toJournalUnavailable; caller never blocksMAX(seq)restart resume, no gap/duplicationsql!(),i64/Vec<u8>) compiles under both backendsExecutionBackendsealed;DurableBackendEnumenum dispatch (noBox<dyn>)durable.journal.append,durable.backend.open,durable.journal.writer.queue_depth)