fix(agentic): dispatch all profiling trajectories concurrently at startup

[ajcasagrande]

A quick glossary first, since the table leans on a few terms:

• AIPerf replays recorded AI-agent conversations against an inference server to measure its performance under load. Each replayed conversation slot is a lane; running at "concurrency 256" means 256 lanes are having conversations with the server at once.
• A credit is the internal "go ahead, send the next message" token. When a conversation finishes, its slot is recycled: the finished conversation goes back into a pool and a fresh one is drawn so the load level stays constant.
• A run has two phases: warmup (get the server's caches into a realistic steady state — not measured) and profiling (the actual measurement). Some conversations continue straight from warmup into profiling.
• A cache-bust marker is a small unique tag injected into each conversation's text so the server can't cheat by reusing cached work from an identical earlier conversation. The same conversation must keep the same tag for its whole life.
• Some replayed conversations spawn sub-agents (helper conversations) and then wait at a gate until all helpers finish before continuing — like a manager who can't proceed until every report comes back.

#	Bug	What went wrong	Why it matters
1	Slow-motion start (ea1b4e9, the original PR #9 fix)	At the start of measurement, the code launched the 256 lanes one at a time, fully waiting for each lane's first message to be accepted before even starting the next — like boarding a plane through a single door, one passenger at a time.	The benchmark claims to measure the server under 256 simultaneous conversations, but for the first ~54 seconds the load slowly trickled up from 1. Part of the measurement window described a load that wasn't actually there, skewing the results.
2	Same conversation played twice at once (5e2d20b)	Each lane is supposed to stamp "I'm using conversation X" into a ledger so no other lane picks X up. But lanes only stamped the ledger as they launched — and with the now-simultaneous start, a fast lane could finish its conversation and draw a replacement from the pool before a slower lane had stamped its claim. The fast lane could draw a conversation another lane was already about to run.	Two lanes replaying the same conversation simultaneously is something the system promises never happens. It distorts the load (the server sees duplicate traffic patterns) and corrupts the cache-related bookkeeping for that conversation. Fixed by stamping every lane's claim into the ledger up front, before anything launches.
3	The anti-cheat tag changed mid-conversation (3dfa3b4)	A conversation continuing from warmup into profiling must keep its exact cache-bust tag, so the cache the server warmed up stays usable. The code didn't store the tag across the boundary — it recomputed all tags from scratch in the new phase and relied on both phases computing them in the exact same order. But the two phases process slightly different lists, so in some setups the ordering shifted by one and a continuing conversation silently got a different tag.	The whole point of warmup is that the server enters measurement with realistic warm caches. A changed tag makes the conversation look brand-new to the server, so the warmed cache is never reused — and the run quietly under-reports the server's cache performance, with no error or warning. Fixed by remembering each conversation's tag rather than recomputing it.
4	Waiting forever for a report that already came in (9479d72, the worst one)	Benchmarks can resume from a mid-run snapshot. A manager-conversation waiting on two helper groups gets a checklist: "group A done? group B done?" The snapshot only records helpers still running at the snapshot moment — so a group that had already finished before the snapshot left no record, and nothing could ever check its box. The checklist could never complete.	That manager lane silently freezes for the entire run — no error, no crash. A 256-lane benchmark quietly becomes a 255-lane benchmark (or worse, with several frozen lanes), and at the end the run stalls for extra minutes waiting on the frozen lanes before force-quitting. Fixed by reasoning it out at resume time: if a group's start point is in the past and none of its helpers are alive, it must have finished — check its box.
5	Two departments sending the same message (04318d6)	When all of a manager's helpers fail to even start, someone must un-stick the manager. The gate-keeping code did it itself — it sent the manager's next message — but then also told the normal message-sending machinery "I didn't handle this," so the normal path sent the very same message a second time.	The server receives a duplicate request (wasted work, inflated request counts), and the duplication snowballs: each duplicated reply triggers two next-messages, and so on, until the run crashes on an internal "this conversation finished twice" safety check. Fixed by giving the dispatch a single owner: the gatekeeper just removes the dead gate and lets the normal machinery send the message once.

The connecting theme: bugs 2–5 were all hiding in code that resumes or coordinates work "mid-story" — they only show themselves under particular timing (a snapshot taken between two events, lanes starting simultaneously, all helpers failing at once), which is why they survived until a careful review rather than failing in everyday runs.

🤖 Generated with Claude Code

[github-actions]

Try out this PR

Quick install:

pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@92026e5028800ccb33ae3819440cbe549478e335

Recommended with virtual environment (using uv):

uv venv --python 3.12 && source .venv/bin/activate
uv pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@92026e5028800ccb33ae3819440cbe549478e335

---

Last updated for commit: 92026e5 • Browse code