Guarantee idempotent transactional delivery and ambiguity-safe provider state

[alexeygrigorev]

External parent epic: DataTalksClub/website#21

Template/credential prerequisite: #1

Audited baseline: c0fdfac632f057f1e6a467111613928c4899b13b

Outcome

Make one accepted Relay transactional-send command converge to one immutable Relay message and at most one automatic provider submission across request replay, concurrent first insert, worker crash, lease expiry, provider timeout, and response loss.

Relay must require request-hash idempotency, use guarded lease/fencing state transitions, distinguish provider acceptance from delivery, and quarantine uncertain provider acknowledgement as ambiguous. It must recheck suppression and sender eligibility atomically at claim immediately before provider submission.

Baseline gaps

At the audited commit:

• idempotency_key is optional and an empty key is replaced internally;
• an existing client/key returns without comparing delivery-affecting request content;
• the check-before-insert path does not recover a concurrent first-insert uniqueness race;
• messages identify mutable template keys rather than immutable versions;
• status is limited to queued, sending, sent, failed, skipped, bounced, and complained;
• sending has no lease owner/token/expiry or recorded attempt boundary;
• uncertain provider calls return rows to queued and may be retried blindly;
• sent conflates provider acceptance with later delivery; and
• bounce/complaint/category suppression is checked before queueing, not atomically at claim.

Scope

Mandatory request-hash idempotency

• Require a non-empty caller idempotency_key for every real transactional send.
• Compute a versioned canonical SHA-256 request hash over tenant/client plus every delivery-affecting field:
  • normalized primary recipient;
  • purpose/classification;
  • immutable template_key and template_version from Harden scoped credentials and versioned transactional template rendering #1;
  • canonical typed context;
  • resolved sender ID/address and reply-to;
  • normalized CC/BCC;
  • normalized headers and structured message parts/options.
• Persist the hash with the message and record the hash-contract version.
• Exact tenant/key/hash replay returns the original message/status and marks the response as an idempotent replay.
• Same tenant/key with a different hash returns 409 idempotency_conflict and creates/enqueues nothing.
• Concurrent first submissions converge through the database uniqueness constraint: the loser reloads the winner, compares the canonical hash, and returns replay or conflict. Integrity errors must not escape as 500.
• Response loss after commit is recoverable by exact replay of the original command.

Immutable preparation

• Require the immutable versioned template contract from Harden scoped credentials and versioned transactional template rendering #1.
• Validate/render/resolve sender and persist the recipient, version, rendered subject/plain/HTML, sender, options, and request hash before queueing provider work.
• A queued message never re-renders from a mutable draft/current version.
• Queue payloads contain scalar identifiers and an immutable/fenced message reference, not raw protected bodies or credentials.

Guarded lifecycle and attempts

Implement explicit guarded states:

• queued: committed and eligible for claim;
• leased: held by one owner/token until a bounded expiry;
• provider_accepted: provider returned an accepted message identifier;
• delivered: authoritative later delivery event;
• retryable: a known pre-provider/transient failure eligible after bounded backoff;
• ambiguous: provider may have accepted but Relay cannot prove the outcome;
• suppressed: claim-time policy prevented submission;
• dead: permanent failure or retry exhaustion;
• hard_bounced and complained: terminal deliverability outcomes.

Record append-only attempts with start/end, lease owner/token, outcome, safe reason/error code, provider correlation, and timestamps. Do not store raw provider payloads or credentials.

• Claim with row locking plus owner/token/expiry fencing.
• Only the current lease token may transition or complete the attempt.
• An expired lease may be reclaimed only when the attempt is provably pre-provider.
• Known failures before provider invocation may become retryable with bounded exponential backoff.
• Any timeout/connection loss/crash during or after the provider request without a definitive result becomes ambiguous and is never automatically retried.
• Permanent validation/provider rejection or exhausted retries becomes dead.
• Provider acceptance is provider_accepted, never displayed or serialized as delivered.
• Guard against status regression and duplicate terminal transitions.

Claim-time suppression and sender eligibility

Within the claim transaction immediately before provider invocation:

• reload current hard-bounce, complaint, applicable transactional/category suppression, sender eligibility, and tenant/client state;
• if blocked, record suppressed plus a safe reason and make no provider call;
• prove a suppression or credential/client-state change after queueing but before claim prevents submission; and
• avoid holding a database transaction open across the network request while preserving the fenced decision/attempt boundary.

Redacted status contract

Update GET /api/transactional/messages/{message_id} and OpenAPI to return:

• authoritative current state;
• immutable template key/version and safe sender ID;
• request/correlation identifiers safe for client use;
• ordered redacted transition/attempt timeline;
• lease/attempt summary without worker secrets;
• provider-accepted versus delivered timestamps;
• safe reason codes and retry/ambiguity eligibility.

Do not expose bodies, context, full provider payload, credentials, or unrelated tenant identifiers.

Expected files

Implementation should remain within Relay-owned delivery surfaces such as:

• mailing/models.py and new forward-only migrations;
• mailing/services/transactional.py;
• mailing/services/transactional_sender.py;
• mailing/queue_contracts.py and mailing/enqueue.py;
• mailing/services/contacts.py, mailing/services/senders.py, and focused transition helpers;
• mailing/views.py, mailing/services/api_docs.py, and docs/api.md;
• mailing/tests/test_transactional_api.py;
• mailing/tests/test_transactional_sender.py;
• queue/worker and PostgreSQL concurrency tests under mailing/tests/.

Do not redesign campaigns, inbound mail, Mailchimp, Taskdeck, website jobs, or infrastructure.

Acceptance criteria

• A caller idempotency key is mandatory and one canonical versioned request hash covers every delivery-affecting field.
• Exact replay returns the original message; same key/different hash returns 409 with no new row, event, queue item, attempt, or provider call.
• Concurrent first insert returns replay/conflict deterministically and produces one message plus one queue item without a 500.
• Immutable template/version, rendered content, sender, recipient/options, and request hash are snapshotted before queueing.
• State transitions are guarded and provider acceptance is never reported as delivery.
• Claims have owner/token/expiry fencing; expired provably pre-provider work can recover, while uncertain provider work cannot re-enter automatic send.
• Before-provider transient failures use bounded backoff; exhausted/permanent failures become dead.
• During/after-provider timeout, response loss, or crash becomes ambiguous and no automatic retry can submit it again.
• Claim-time hard-bounce/complaint/category suppression and sender/client eligibility prevent a provider call when state changes after queueing.
• Ordered attempts/transitions and the message status response are redacted, tenant-scoped, and safe.
• Queue payloads, logs, errors, audit, test output, and OpenAPI examples contain no credentials, protected body/context, or full provider payload.
• Focused unit, migration, queue/worker, and real PostgreSQL concurrency tests pass with deterministic provider fakes and no external network/AWS credentials.

Mandatory failure tests

1. Exact replay, changed recipient/version/context/sender/options, and canonical JSON/key ordering.
2. Two concurrent first requests with the same key/same hash and same key/different hash.
3. Crash before claim, after claim/before provider, during provider call, after provider acceptance/before local acknowledgement, and after local acknowledgement.
4. Lease expiry before provider, stale lease-token completion, competing claims, retry exhaustion, and forbidden transition from ambiguous to automatic send.
5. Hard bounce, complaint, category suppression, client disablement, sender disablement, or credential policy change after queue and before claim.
6. Provider throttle/transient failure, permanent rejection, accepted response, delivered event, late bounce/complaint, and duplicate/out-of-order terminal events.
7. Tenant isolation plus redaction canaries in responses, logs, errors, events, queue payloads, and test artifacts.

Validation

Use uv and provider fakes/stubs only:

git diff --check
uv run python manage.py makemigrations --check --dry-run
uv run pytest mailing/tests/test_transactional_api.py mailing/tests/test_transactional_sender.py mailing/tests/test_sqs_worker.py
make lint
make test

The tester must run the concurrency cases against PostgreSQL in the repository-supported isolated test path. No live Relay sandbox call, AWS credential, SES request, or email send is permitted.

Browser and screenshots

This is a backend delivery/API contract. No visible Relay page is required to change, so browser testing and screenshots are not applicable unless implementation deliberately expands UI scope and obtains PM approval.

Dependencies

• External ownership and consumer requirements: website #21.
• Depends on Harden scoped credentials and versioned transactional template rendering #1 for scoped credentials, immutable template_key + template_version, safe rendering, and snapshot inputs.
• The generic callback/reconciliation issue depends on this lifecycle's states, attempts, and redacted transition events.
• Website real-send remains disabled until Harden scoped credentials and versioned transactional template rendering #1, this issue, and the callback/reconciliation issue are deployed at exact pinned commit/OpenAPI versions and pass consumer tests.

Non-goals

• No website EmailDelivery model, job, callback endpoint, Studio/admin API, or Datamailer migration.
• No template authoring/version/rendering implementation beyond consuming Harden scoped credentials and versioned transactional template rendering #1's contract.
• No generic outbound callback signing/delivery or unmatched provider-event reconciliation; owned by the callback/reconciliation issue.
• No automatic operator resolution or resend from ambiguous.
• No live credential provisioning, infrastructure mutation, deployment, production sender/domain, broad recipient access, or external email.
• No campaign/newsletter, tracking-pixel, or engagement-analytics redesign.

Process handoff

Follow docs/PROCESS.md in an isolated Relay issue worktree. The engineer implements and tests without committing; a separate tester verifies exact failure/concurrency criteria; a product manager accepts; only then is the issue committed with Closes #N and moved through the repository lifecycle without a pull request.

[alexeygrigorev]

Groomed on creation

Owned Relay slice for mandatory canonical request-hash idempotency, concurrent first-insert recovery, fenced leases/attempts, accepted-versus-delivered state, ambiguity quarantine, and claim-time suppression required by website #21.

Pinned to audited baseline c0fdfac632f057f1e6a467111613928c4899b13b and dependent on #1's immutable template/snapshot contract. Failure/concurrency tests are deterministic and no live credential, deployment, provider call, or email send is authorized.