Share: vLLM Support for Fun-ASR (Fun-ASR-vllm)

[yuekaizhang]

Hi,
Thank you for your excellent work on this project!
I wanted to share that I have implemented vLLM support for Fun-ASR inference, which currently achieves an approximately 50% speed increase. I created a repository for this implementation here:

https://github.com/yuekaizhang/Fun-ASR-vllm

Benchmark Details
Dataset: SPEECHIO_ASR_ZH00007 (approx. 1 hour of audio)
Hardware: Single NVIDIA H20 GPU

Mode	Decoding Time	RTF	RTFx	CER	Note
Huggingface PyTorch	218.2 Secs	0.06	16.5	7.02%	batch_size=1
vLLM (Qwen3-0.6B)	145.6 Secs	0.04	24.7	6.99%	batch_size=1

Any feedback or attention would be greatly appreciated!

[yuekaizhang]

Update: Batch inference is supported for both vllm and PyTorch.

Mode	Decoding Time	RTF	RTFx	CER	Note
Huggingface PyTorch	218.2 Secs	0.06	16.5	7.02%	batch_size=1
Huggingface PyTorch	45.4 Secs	0.013	79.3	8.53%	batch_size=16
vLLM (Qwen3-0.6B)	145.6 Secs	0.04	24.7	6.99%	batch_size=1
vLLM (Qwen3-0.6B)	26.3 Secs	0.007	136.9	7.03%	batch_size=16

[yuekaizhang]

Update:

For production deployment with high concurrency, we provide integration with NVIDIA Triton Inference Server.

For detailed setup instructions, see triton_server/README.md.

Quick Start

cd triton_server

# Using Docker Compose (recommended)
docker compose up

# Or pull and run the pre-built image
docker pull soar97/triton-fun-asr:25.06

Triton Performance

• Dataset: SPEECHIO_ASR_ZH00007 (approx. 1 hour of audio)
• Hardware: Single NVIDIA H20 GPU

Concurrency	CER	Processing Time	P50 Latency	RTF
8	7.04%	44.56s	450.99ms	0.0126
16	7.00%	27.96s	533.36ms	0.0079
32	7.07%	24.51s	952.93ms	0.0069