You’re deploying a model on a system. It starts up, prompts are getting responses. Now the hard question: Is this fast?
Your instincts might lead you to send curl commands, hand-roll an asyncio script, or vibe code yet another one-off load generator. All of these paths have the same problem: single-process performance limits, Python’s GIL capping concurrency, or numbers measured against a reference you built yourself. Either way, you end up with results you can’t fully trust, attached to tooling you’ll have to rewrite the moment requirements change.
What you need is a load client that can saturate a real server without becoming the bottleneck, produce output you can act on, and take five minutes to configure, not five hours. That’s NVIDIA AIPerf.
What AIPerf does differently AIPerf is the designated successor to GenAI-Perf and is a ground-up rewrite. The design choices reflect hard lessons from running LLM benchmarks at scale:
A clean break from the old architecture. AIPerf doesn’t run on top of Perf Analyzer the way GenAI-Perf did. It’s a clean architectural break and the reason AIPerf can scale the way it does. If you’re porting an existing workflow, the migration guide covers the key deltas. The client shouldn’t be the bottleneck. Most benchmarkers, GenAI-Perf included, use a single-process architecture that becomes GIL-bound under real concurrency or request rate. AIPerf is a multiprocessed system: worker processes generate load, separate record-processor services handle results, and everything is coordinated over ZMQ.. This structure allows for more accurate server benchmarking by preventing AIPerf from becoming a client-side bottleneck. Workload breadth that matches what you actually run. AIPerf supports 15+ endpoint types: chat, responses, NIM rankings, image generation, and more — along with public datasets like ShareGPT and trace replay formats from Mooncake, Baseten, WEKA (AgentX), and others. Whether you’re running a quick synthetic smoke test or replaying captured production traffic, you don’t need a different tool. Load shape you actually control. AIPerf supports constant, Poisson, and gamma arrival patterns with tunable burstiness, gradual ramping for concurrency and request rate, and synthetic distributions including vLLM/SGLang range-ratio for variable ISL/OSL. You control the shape of the load, not just the volume. Your maiden benchmark: Synthetic ISL/OSL on vLLM For this walkthrough we’ll use Qwen3-0.6B served through vLLM. The model choice is deliberate; it’s small enough to run on a single GPU and fast enough to iterate on without waiting. The point isn’t to benchmark Qwen3-0.6B specifically; it’s to establish the measurement loop. Once you have that, swapping in a different model or endpoint is a one-flag change.
Start the Server
Pull and start vLLM with the reasoning parser enabled:
docker pull vllm/vllm-openai:latest
docker run --gpus all -p 8000:8000 -e HF_TOKEN vllm/vllm-openai:latest
--model Qwen/Qwen3-0.6B
--reasoning-parser qwen3
--host 0.0.0.0 --port 8000
Install AIPerf
We can use uv to install a central copy:
uv tool install aiperf
Or for a virtual environment:
uv venv venv source venv/bin/activate uv pip install aiperf One platform note: on aarch64, the crick dependency ships as source-only and requires a C toolchain (build-essential on Debian/Ubuntu, Development Tools on RHEL). If the install stalls on that package, that’s why.
Running the benchmark With the server up and AIPerf installed, we can now run our first profile:
aiperf profile
--model Qwen/Qwen3-0.6B
--endpoint-type chat
--streaming
--url localhost:8000
--synthetic-input-tokens-mean 128
--synthetic-input-tokens-stddev 0
--output-tokens-mean 128
--output-tokens-stddev 0
--extra-inputs min_tokens:128
--extra-inputs ignore_eos:true
A few flags here are doing more work than they look like:
--synthetic-input-tokens-stddev 0 and --output-tokens-stddev 0 pin the workload to exactly 128 input and 128 output tokens per request. This reproduces a commonly used static benchmark that holds request and output lengths constant.
--extra-inputs min_tokens:128 and --extra-inputs ignore_eos:true tell the model to actually emit 128 tokens rather than stopping early. Without these, the output token count is a suggestion. The model stops whenever it naturally finishes, which can be well short of your target OSL. Throughput numbers end up lower than they should be, and they’re not reproducible across runs.
--streaming is not optional if you want to measure TTFT and ITL. Without streaming, the server batches the full response before sending it, and there are no first- or decode-token events to measure.
