perf: sender pool for the event transport (removes head-of-line blocking)

[cat-ph]

Stacked on #145 (base branch ci/rust-event-transport) — review/merge that first.

What

Addresses the head-of-line blocking raised in review: the single transport worker did the blocking POST inline, so one slow or stalled endpoint blocked draining and could fill the bounded queue.

Splits the worker into:

• a dispatcher thread that owns the buffer, retry queue, and schedule (single-owner, no lock), and
• a small sender pool (min(cores, 4)) that runs the blocking POSTs concurrently and reports each outcome back.

A slow endpoint now stalls at most one sender instead of the whole pipeline. flush()/shutdown() stay barriers that complete once the sends they dispatched report back. Cross-batch ordering becomes best-effort, which is fine now that event timestamps are stamped at capture time (#145).

Adds crossbeam-channel for the MPMC job queue plus a select over the control/outcome channels — std mpsc is single-consumer, so a pool would otherwise serialize on a mutex-wrapped receiver. Deliberately not a tokio task: the worker stays runtime-independent (blocking client + a future panic hook).

Review (autoreview)

Ran a structured review over the diff across several rounds; it caught 5 real bugs in the shutdown/barrier semantics, all fixed with regression tests:

• queued sender jobs kept POSTing past shutdown_timeout → abandon flag
• a pre-shutdown in-flight batch that failed transiently was dropped instead of getting its final attempt → re-dispatch as final
• flush() could wait on captures that arrived after the call → defer + replay
• abandoned in-flight retries leaked the pending_events count → accounted on send failure
• v0 sent_at was stale on retries → stamped per attempt

Known limitation (deferred, documented in code): flush() under concurrent multi-threaded calls can under-wait — a pre-existing global-barrier limitation; a precise fix needs per-flush generation tracking. Tracked as a follow-up.

Tests

Full gate green: fmt, clippy (-D warnings, v0 + v1), public-api snapshot, publish --dry-run, and every unit-test feature config (default, --workspace, capture-v1 async + blocking, error-tracking blocking, error-tracking + capture-v1, e2e). Added deterministic regression tests for the pool.

[greptile-apps]

Comments Outside Diff (1)

1. src/client/transport.rs, line 1057-1163 (link)

   Tests could be parameterised per the repo convention

   The three new regression tests (flush_waits_for_every_batch_across_the_pool, sender_abandons_queued_jobs_without_posting, sender_accounts_retry_when_dispatcher_is_gone) each exercise one specific scenario. The repo prefers parameterised tests where applicable. While these scenarios have meaningfully different setups, a parameterised approach for, for example, the abandon-flag test (varying event counts, or abandon = true/false) and the dispatcher-gone test (varying max_capture_attempts) would make it easier to extend coverage later without adding boilerplate.

   Prompt To Fix With AI

   This is a comment left during a code review.
   Path: src/client/transport.rs
   Line: 1057-1163
   
   Comment:
   **Tests could be parameterised per the repo convention**
   
   The three new regression tests (`flush_waits_for_every_batch_across_the_pool`, `sender_abandons_queued_jobs_without_posting`, `sender_accounts_retry_when_dispatcher_is_gone`) each exercise one specific scenario. The repo prefers parameterised tests where applicable. While these scenarios have meaningfully different setups, a parameterised approach for, for example, the abandon-flag test (varying event counts, or `abandon = true/false`) and the dispatcher-gone test (varying `max_capture_attempts`) would make it easier to extend coverage later without adding boilerplate.
   
   How can I resolve this? If you propose a fix, please make it concise.

   Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Prompt To Fix All With AI

Fix the following 3 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 3
src/client/transport.rs:510-517
`drop_deferred` silently discards events without any log message. `drop_buffer` logs a warning in the same codepath (zero-grace shutdown), so a user debugging missing events on the `drop_buffer` path would see a message, but an equivalent loss from the deferred queue would be invisible. Given the deferred queue can hold events that arrived during an active flush (a common concurrent-capture scenario), this asymmetry makes the zero-grace shutdown harder to reason about.

```suggestion
    /// Drop held captures (zero-grace shutdown), accounting for them in `len`.
    fn drop_deferred(&mut self) {
        let n = self.deferred.len();
        if n > 0 {
            warn!(
                "posthog-rs: shutdown timeout reached; dropping {} deferred event(s)",
                n
            );
            self.deferred.clear();
            dec_len(&self.pipeline.len, n);
        }
    }
```

### Issue 2 of 3
src/client/transport.rs:728-731
**`pending_events` leak if all senders panic**

If every sender thread panics (terminates without sending an outcome), `outcome_rx` closes while `in_flight > 0`. The disconnected-drain path's `Err` arm sets `done = true` and exits the loop without decrementing `len` for the in-flight batches — `pending_events()` would then report stale counts indefinitely. This is an exceptional path and the PR already acknowledges a future panic-hook; noting it here so the follow-up work has a concrete surface area to address (`dec_len` the still-in-flight count, or add a panic-hook on sender spawn).

### Issue 3 of 3
src/client/transport.rs:1057-1163
**Tests could be parameterised per the repo convention**

The three new regression tests (`flush_waits_for_every_batch_across_the_pool`, `sender_abandons_queued_jobs_without_posting`, `sender_accounts_retry_when_dispatcher_is_gone`) each exercise one specific scenario. The repo prefers parameterised tests where applicable. While these scenarios have meaningfully different setups, a parameterised approach for, for example, the abandon-flag test (varying event counts, or `abandon = true/false`) and the dispatcher-gone test (varying `max_capture_attempts`) would make it easier to extend coverage later without adding boilerplate.

_{Reviews (1): Last reviewed commit: "docs: note the concurrent-flush barrier ..." | Re-trigger Greptile}

[greptile-apps]

drop_deferred silently discards events without any log message. drop_buffer logs a warning in the same codepath (zero-grace shutdown), so a user debugging missing events on the drop_buffer path would see a message, but an equivalent loss from the deferred queue would be invisible. Given the deferred queue can hold events that arrived during an active flush (a common concurrent-capture scenario), this asymmetry makes the zero-grace shutdown harder to reason about.

Suggested change

	/// Drop held captures (zero-grace shutdown), accounting for them in `len`.
	fn drop_deferred(&mut self) {
	let n = self.deferred.len();
	if n > 0 {
	self.deferred.clear();
	dec_len(&self.pipeline.len, n);
	}
	}
	/// Drop held captures (zero-grace shutdown), accounting for them in `len`.
	fn drop_deferred(&mut self) {
	let n = self.deferred.len();
	if n > 0 {
	warn!(
	"posthog-rs: shutdown timeout reached; dropping {} deferred event(s)",
	n
	);
	self.deferred.clear();
	dec_len(&self.pipeline.len, n);
	}
	}

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/client/transport.rs
Line: 510-517

Comment:
`drop_deferred` silently discards events without any log message. `drop_buffer` logs a warning in the same codepath (zero-grace shutdown), so a user debugging missing events on the `drop_buffer` path would see a message, but an equivalent loss from the deferred queue would be invisible. Given the deferred queue can hold events that arrived during an active flush (a common concurrent-capture scenario), this asymmetry makes the zero-grace shutdown harder to reason about.

```suggestion
    /// Drop held captures (zero-grace shutdown), accounting for them in `len`.
    fn drop_deferred(&mut self) {
        let n = self.deferred.len();
        if n > 0 {
            warn!(
                "posthog-rs: shutdown timeout reached; dropping {} deferred event(s)",
                n
            );
            self.deferred.clear();
            dec_len(&self.pipeline.len, n);
        }
    }
```

How can I resolve this? If you propose a fix, please make it concise.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[greptile-apps]

pending_events leak if all senders panic

If every sender thread panics (terminates without sending an outcome), outcome_rx closes while in_flight > 0. The disconnected-drain path's Err arm sets done = true and exits the loop without decrementing len for the in-flight batches — pending_events() would then report stale counts indefinitely. This is an exceptional path and the PR already acknowledges a future panic-hook; noting it here so the follow-up work has a concrete surface area to address (dec_len the still-in-flight count, or add a panic-hook on sender spawn).

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/client/transport.rs
Line: 728-731

Comment:
**`pending_events` leak if all senders panic**

If every sender thread panics (terminates without sending an outcome), `outcome_rx` closes while `in_flight > 0`. The disconnected-drain path's `Err` arm sets `done = true` and exits the loop without decrementing `len` for the in-flight batches — `pending_events()` would then report stale counts indefinitely. This is an exceptional path and the PR already acknowledges a future panic-hook; noting it here so the follow-up work has a concrete surface area to address (`dec_len` the still-in-flight count, or add a panic-hook on sender spawn).

How can I resolve this? If you propose a fix, please make it concise.