create: overlap pack store and build of next pack, fixes #9988

[ThomasWaldmann]

Fixes #9988.

PackWriter now hands a full pack to a background store-thread (at most one in flight): the pack bytes are joined, sha256-hashed (the pack_id) and stored in the store-thread, while the caller goes on assembling the next pack. hashlib and the store I/O release the GIL, so there is real overlap even on CPython. Throughput becomes max(assembly, store) instead of assembly + store — measured up to 1.50x on a real borg create, see the benchmark comment below.

The ChunkIndex is only ever touched by the calling thread: the store-thread's results (or error) are applied when it is joined, at the next pack boundary or flush(). Consequences:

• put()/add() return the previous pack's results while the current pack's store is in flight; update_pack_info() keys by chunk_id, so callers do not care which pack the results belong to.
• flush() is a barrier: it joins an in-flight store and writes the current buffer synchronously, so afterwards nothing is F_PENDING anymore (needed by the periodic chunk index persist (borg2: does borg persist UNKNOWN pack ids in periodic index writes? #9900) and by close()).
• a store error (e.g. ENOSPC) surfaces one pack later, from whichever add()/flush() call joins the store-thread — still before anything gets finalized, since the final flush is a barrier (no create: do not wrap repository writes in backup_io("read") (silent data loss on ENOSPC) #9853-style regression). The failed pack's index entries are dropped and the buffered pieces die with the aborting command, so the close()-time index persist stays clean.
• get()/get_many() of a chunk whose pack store is still in flight join the store-thread first (read barrier), then read normally.
• close() joins a still-in-flight store (normally a no-op, flush() ran before): a stored pack gets recorded, a failed one rolled back (not raising, to not mask the error being unwound).

Sharing the Store between the store-thread and the main thread (lock refresh, reads of already stored packs) requires borgstore >= 0.6.0, which serializes all Store operations internally (borgbackup/borgstore#206 / borgbackup/borgstore#207). That dependency bump is already on master, so this PR's diff is just repository.py and its tests.

Two env vars, both off/undocumented like the other BORG_PACK_* tuning knobs:

• BORG_PACK_ASYNC=no disables the store-thread (debugging aid, and the sync side of the benchmarks).

• BORG_PACK_TRACE=yes prints the store-thread lifecycle to stderr — < started, H hashing, S storing, > finished — which makes the overlap directly visible:

  <HS><HS><HS><HS>
  

Tests: the pre-existing synchronous-contract unit tests run with async_store=False; new tests cover deferred results, the combined flush barrier, deferred error surfacing with rollback, and the get() read barrier. Full local suite green (repository/cache/archiver), plus end-to-end create/check/extract/compact runs in both modes with tiny packs.

Note (pre-existing, not addressed here): on master, an abort with chunks still buffered makes close()'s "PackWriter has unflushed chunks" assert fire during unwind, masking the original exception — that happens with and without this PR and deserves its own fix.

(implemented by Claude, reviewed by TW)

🤖 Generated with Claude Code

[ThomasWaldmann]

Real-world benchmark of the store-thread overlap: borg create of the same data in sync (BORG_PACK_ASYNC=no) vs async mode, one run at a time on an otherwise idle machine (M-series Mac, local repo), slow storage emulated via borgstore's BORGSTORE_LATENCY / BORGSTORE_BANDWIDTH (borgstore 0.6.0 holds its op lock during the emulated delay, so it behaves like a genuinely slow backend toward the store-thread).

Data: unique random bytes interleaved 1:1 with zeros (≈2:1 lz4-compressible, no dedup effects), fresh repo per run, aes256-ocb, default 50 MB packs. "Assembly A" is the wall time of the identical create with no emulation (chunking, id-hashing, compressing, encrypting, item processing).

input data	stored	emulated store	assembly A (no emulation)	sync	async	speedup
1 GiB, 262,144 x 4 KiB files	577 MiB	3 MB/s + 100 ms	45-49 s	257.6 s	220.6 s	1.17x
10 GiB, 160 x 64 MiB files	5143 MiB	100 MB/s + 30 ms	25.2 s	95.9 s	64.0 s	1.50x

(Numbers re-measured after rebasing onto current master, i.e. including the nogil work of #10014. They reproduce the pre-rebase measurements almost exactly -- async 220.6 s vs 220.9 s and 63.9 s vs 64.0 s -- which is expected: the store-thread is I/O-bound here, so there is no GIL contention for #10014 to remove. An A/B of both builds on identical input showed the assembly baseline is unchanged within run-to-run scatter as well.)

Why these numbers: the pipeline turns assembly + store into max(assembly, store) -- async wall time lands on whichever side is larger, hiding the smaller one.

• Row 1 (many tiny files, slow uplink): store dominates -- S = 577 MiB / 3 MB/s + per-op latency, roughly 205 s. Sync pays A + S (measured 258 s); async pays max(A, S) plus a small remainder (measured 221 s), i.e. it hid essentially all of the per-file assembly overhead behind the upload. The remainder is main-thread store touches (periodic lock refresh, chunks-index write) queueing behind an in-flight pack store that holds the store for ~17 s at this bandwidth -- realistic for any genuinely slow backend.
• Row 2 (bulk data, fast storage): the two sides are close to balanced -- S is about 5143 MiB / 100 MB/s + ~104 pack stores x 30 ms, roughly 60 s, vs A = 25.2 s. Async lands almost exactly on the store floor (64 s): assembly is hidden completely. That the measured speedup (1.50x) even exceeds the naive (A+S)/max(A,S) = 1.42x is because the sync run pays some extra interleaving overhead per pack boundary that the pipelined run avoids.

Upper bound for reference: a synthetic micro-benchmark with perfectly balanced simulated stages (0.2 s store per 1 MB pack vs matching per-chunk assembly cost) measures 1.75x, close to the theoretical 2x for one pack in flight.

Summary: the gain ranges from ~nothing (very slow uplink + borg's fast assembly path, where there is simply little to hide) up to ~1.5x when assembly and store are comparable -- and async never measured slower than sync in any run.

You can watch the overlap directly with the new BORG_PACK_TRACE=yes (< store-thread started, H hashing, S storing, > finished):

<HS><HS><HS><HS>

(benchmarks by Claude, reviewed by TW)