Up to 9% decrease in output tokens per second between v0.3 and v0.4

[sjmonson]

Describe the bug

The measured output token throughput in GuideLLM v0.4.0 and later is significantly lower despite all other metrics being near identical.

Expected behavior

Output token throughput should not differ from v0.3.x when all else is equal.

Environment

Include all relevant environment information:

1. OS [e.g. Ubuntu 20.04]: Fedora 43 container on OCP 4.19
2. Python version [e.g. 3.12.2]: 3.13.9
3. GuideLLM version: 0.4.0 and 0.5.0

To Reproduce

Run the following GuideLLM test on v0.3.0 and v0.4.0, then compare metrics in output JSON:

# Ensure we use the same endpoint across versions
export GUIDELLM_REQUEST_TYPE=text_completions
export GUIDELLM_TARGET=http://localhost:8080

guidellm benchmark run \
            --target="${GUIDELLM_TARGET}" \
            --model=RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic \
            --processor=RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic \
            --rate-type=concurrent \
            --data=prompt_tokens=1000,output_tokens=1000 \
            --max-seconds=600 \
            --rate=1,50,100,200,300,500,650

Additional context

Total Output Tokens per second over intended concurrency

Percent Change from v0.3.0 baseline

[sjmonson]

Another place where the a regression is observable is measured request concurrency. P10 and lower suffer pretty heavily at 500 concurrency then improves at 650. This matches output token results.

Request Concurrency

		mean	p001	p01	p05	p10
version	concurrency
0.3.0	1	0.999876	1	1	1	1
0.4.0	1	0.999952	1	1	1	1
0.3.0	50	49.91858	16	50	50	50
0.4.0	50	48.928146	1	1	50	50
0.3.0	100	99.61601	16	100	100	100
0.4.0	100	97.151576	0	2	100	100
0.3.0	200	198.241756	14	118	200	200
0.4.0	200	196.219505	1	8	200	200
0.3.0	300	291.075135	16	118	239	239
0.4.0	300	293.583736	0	87	280	280
0.3.0	500	487.173261	10	95	485	500
0.4.0	500	440.305875	4	4	4	42
0.3.0	650	551.758179	15	82	82	82
0.4.0	650	563.084963	23	113	164	164

Request Concurrency % Change

		mean	p001	p01	p05	p10
version	concurrency
0.3.0	1	nan%	nan%	nan%	nan%	nan%
0.4.0	1	0.01%	0.00%	0.00%	0.00%	0.00%
0.3.0	50	nan%	nan%	nan%	nan%	nan%
0.4.0	50	-1.98%	-93.75%	-98.00%	0.00%	0.00%
0.3.0	100	nan%	nan%	nan%	nan%	nan%
0.4.0	100	-2.47%	-100.00%	-98.00%	0.00%	0.00%
0.3.0	200	nan%	nan%	nan%	nan%	nan%
0.4.0	200	-1.02%	-92.86%	-93.22%	0.00%	0.00%
0.3.0	300	nan%	nan%	nan%	nan%	nan%
0.4.0	300	0.86%	-100.00%	-26.27%	17.15%	17.15%
0.3.0	500	nan%	nan%	nan%	nan%	nan%
0.4.0	500	-9.62%	-60.00%	-95.79%	-99.18%	-91.60%
0.3.0	650	nan%	nan%	nan%	nan%	nan%
0.4.0	650	2.05%	53.33%	37.80%	100.00%	100.00%

[sjmonson]

For Request Concurrency it seems that v0.3.0 is the inaccurate one. Recalculating concurrency from request start and end times bring the v0.3.0 percentiles much closer to v0.4.0:

Request Concurrency

		count	mean	std	min	0.1%	1%	5%	10%	25%	50%	max
concurrency	version
1	0.3.0	120	0.5	0.502096	0	0	0	0	0	0	0.5	1
1	0.4.0	122	0.5	0.502062	0	0	0	0	0	0	0.5	1
5	0.3.0	3388	45.597403	4.849639	0	2	18	42	43	44	46	50
5	0.4.0	3396	46.004122	4.848149	0	2	17	43	44	45	47	50
100	0.3.0	4790	94.302088	11.010311	0	2.789	24.89	92	93	94	96	100
100	0.4.0	4798	94.668404	11.206696	0	2.797	24	92	93	95	97	100
200	0.3.0	5986	189.998496	28.175696	0	3	30	153.25	193	194	196	200
200	0.4.0	5992	190.29506	28.362782	0	3	30	151	193	195	197	200
300	0.3.0	5856	281.922643	51.564007	0	3	30.55	150.75	292	294	296	300
300	0.4.0	5946	282.198621	52.044279	0	3	30	150	292	295	297	300
500	0.3.0	6947	461.149417	101.111302	0	3.946	36	178	356	493	496	500
500	0.4.0	7003	461.991004	102.287375	0	4	35.02	175.1	350.2	495	497	500
650	0.3.0	6631	584.103755	150.971465	0	3.63	34.3	170.5	338	642	646	650
650	0.4.0	6823	585.782354	150.609707	0	3.822	34.22	171.1	342	644	647	650

Request Concurrency % Change

		count	mean	std	min	0.1%	1%	5%	10%	25%	50%	max
concurrency	version
1	0.3.0	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%
0.4.0	1.67%	0.00%	-0.01%	nan%	nan%	nan%	nan%	nan%	nan%	0.00%	0.00%
50	0.3.0	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%
0.4.0	0.24%	0.89%	-0.03%	nan%	0.00%	-5.56%	2.38%	2.33%	2.27%	2.17%	0.00%
100	0.3.0	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%
0.4.0	0.17%	0.39%	1.78%	nan%	0.29%	-3.58%	0.00%	0.00%	1.06%	1.04%	0.00%
200	0.3.0	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%
0.4.0	0.10%	0.16%	0.66%	nan%	0.00%	0.00%	-1.47%	0.00%	0.52%	0.51%	0.00%
300	0.3.0	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%
0.4.0	1.54%	0.10%	0.93%	nan%	0.00%	-1.80%	-0.50%	0.00%	0.34%	0.34%	0.00%
500	0.3.0	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%
0.4.0	0.81%	0.18%	1.16%	nan%	1.37%	-2.72%	-1.63%	-1.63%	0.41%	0.20%	0.00%
650	0.3.0	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%	nan%
0.4.0	2.90%	0.29%	-0.24%	nan%	5.29%	-0.23%	0.35%	1.18%	0.31%	0.15%	0.00%