Load Testing LLM Endpoints with GPU Observability

What you'll build

By the end of this guide you will have a load test that hammers a local LLM server (Ollama or vLLM) with streaming chat completions, measures time-to-first-token (TTFT) and tokens/sec per request, and records a GPU timeseries (utilization, VRAM) on the same timeline — so you can tell whether the GPU or the server is the bottleneck. SLO gates turn the whole thing into a pass/fail check you can wire into CI.

You'll need:

• An NVIDIA GPU with nvidia-smi on PATH (comes with the driver), or a dcgm-exporter endpoint
• A perfscale binary (OSS build is enough)
• A running model server — Ollama or vLLM (below)

Step 1 — Start the model server

Ollama exposes an OpenAI-compatible endpoint on http://127.0.0.1:11434:

ollama pull llama3.2:3b
ollama serve   # usually already running as a service

vLLM serves the OpenAI API on http://127.0.0.1:8000. With Docker (needs the NVIDIA Container Toolkit):

docker run --gpus all -p 8000:8000 \
  vllm/vllm-openai --model meta-llama/Llama-3.2-3B-Instruct

or with pip: pip install vllm && vllm serve meta-llama/Llama-3.2-3B-Instruct. Gated Hugging Face models also need HF_TOKEN.

No GPU tooling at all? The runs still work — GPU collection is best-effort, the GPU columns just come out empty.

Step 2 — The basic std/llm@v1 scenario

Create llm-test.yaml. One step = one completion request; streaming is the default for OpenAI-compatible endpoints, and TTFT is the arrival time of the first chunk that carries content:

vus: 2
duration: 1m

steps:
  - name: chat completion
    use: std/llm@v1
    with:
      url: http://127.0.0.1:11434/v1/chat/completions   # vLLM: http://127.0.0.1:8000/v1/chat/completions
      model: llama3.2:3b
      prompt: Summarize the plot of Hamlet in three sentences.
      max_tokens: 256

Run it:

perfscale run -f llm-test.yaml

Useful with: parameters for real scenarios:

Parameter	Default	Notes
endpoint	openai	openai (also covers Ollama, vLLM, LM Studio), anthropic, or generic
prompt / messages	—	prompt is sugar for a single user message; messages takes [{ role, content }] — mutually exclusive
max_tokens	256	Completion token cap
stream	true (openai/anthropic)	Read the response as SSE; required for TTFT
params	—	Passthrough into the request body (temperature, …)
api_key, headers	—	Bearer key and extra headers for gated endpoints
timeout_ms	120000	Whole-request timeout (connect → last chunk)

Every request logs a line like:

LLM openai llama3.2:3b → 200, 34 chunks, ttft 120.31ms, 131.5 tok/s (850.02ms)

Step 3 — Add a load profile and GPU sampling

One-shot probes tell you nothing about saturation. Add a stepped ramping-VU profile (closed model) so each plateau is a measurement point, and switch on the gpu: run config so SM utilization and VRAM are sampled on the same timeline:

vus: 2
duration: 1m

stages:
  - { duration: 30s, target: 2 }  # warm-up ramp
  - { duration: 1m, target: 2 }   # plateau @ 2 concurrent requests
  - { duration: 30s, target: 8 }
  - { duration: 1m, target: 8 }   # plateau @ 8
  - { duration: 30s, target: 16 }
  - { duration: 1m, target: 16 }  # plateau @ 16
  - { duration: 30s, target: 0 }  # graceful ramp-down

gpu:
  enabled: true
  interval_ms: 1000
  source: nvidia-smi    # or: dcgm with GPU_SOURCE/DCGM_URL

steps:
  - name: chat completion
    use: std/llm@v1
    with:
      url: http://127.0.0.1:11434/v1/chat/completions
      model: llama3.2:3b
      prompt: Summarize the plot of Hamlet in three sentences.
      max_tokens: 256

Prefer an open model (find the request rate the server actually sustains) with the arrival: section instead of stages: — new requests keep arriving on schedule even when the server falls behind:

arrival:
  max_vus: 64
  pre_allocated_vus: 2   # pool grows lazily to max_vus
  stages:
    - { duration: 1m, rate: 1 }   # 1 completion/sec
    - { duration: 1m, rate: 2 }
    - { duration: 1m, rate: 4 }
    - { duration: 1m, rate: 8 }
    - { duration: 1m, rate: 16 }
    - { duration: 30s, rate: 0 }  # drain in-flight requests

A permit that arrives while all max_vus workers are busy is dropped and counted in the dropped_iterations summary metric — the first rate where it grows past 0 is the server's practical capacity for this model/prompt.

Step 4 — Read the metrics

The k6-style end-of-run summary aggregates per request:

• llm_ttft_ms — time to first content chunk (streamed requests only)
• llm_tokens_per_sec — completion tokens per second of generation time
• llm_prompt_tokens / llm_completion_tokens / llm_chunks — token and chunk counters
• dropped_iterations — arrival-model drops (emitted even at 0)

During the run, periodic [stats] lines share a timeline with the GPU timeseries, so each plateau maps to its own tok/s + TTFT + utilization window — read plateaus, not just run-level aggregates. Add --summary-export results.json to persist the run metadata, request summary, and the full GPU timeseries with per-device aggregates under gpu.

How to correlate:

• tok/s flat across plateaus while GPU utilization sits at ~100% → the GPU is the bottleneck (smaller model, quantization, more VRAM — or accept the rate)
• Utilization well below 100% with rising TTFT p95 → the server is the bottleneck (queueing, context/batch limits) — tune the server, not the GPU
• VRAM creeping toward the card's total → explains mid-run evictions/OOMs
• Longer max_tokens shifts the bottleneck from prefill to decode — the same config can look GPU-bound on short answers and server-bound on long ones

Step 5 — Set SLO gates

End the config with std/thresholds@v1 steps in after: — a violated fail gate fails the run and sets a non-zero exit code:

after:
  - name: SLO gate
    use: std/thresholds@v1
    with:
      llm_ttft_ms: ["p50<500", "p95<2000"]
      llm_tokens_per_sec: ["avg>20"]
      llm_ttft_ms_failed: ["rate<0.01"]
    message: "LLM SLO (stages)"

For the arrival profile, split gates by intent — it deliberately probes past the capacity knee, so degradation at the top rates should be reported, not failed:

after:
  - name: correctness gate
    use: std/thresholds@v1
    with:
      llm_ttft_ms_failed: ["rate<0.01"]
      dropped_iterations: ["count==0"]
    message: "LLM correctness (arrival)"
  - name: capacity advisory
    use: std/thresholds@v1
    with:
      llm_ttft_ms: ["p95<3000"]
    severity: warn
    message: "capacity knee"

The numbers above are starting points calibrated for a mid-range GPU — measure your card's baseline first, then tighten.

Step 6 — CI integration

A violated fail gate exits non-zero, so any pipeline can gate on it directly:

# .github/workflows/llm-bench.yml
- name: LLM bench with GPU SLOs
  run: perfscale run -f llm-test.yaml --summary-export results.json

The catch: CI runners usually have no GPU — self-host a runner on the GPU host (or trigger the run on a dedicated bench machine) and keep the GPU bench out of the default PR workflow. The perfscale repo does exactly this: bench/gpu/ is local-only by design and deliberately not wired into a workflow. Gate PRs on the non-GPU checks; run the GPU suite nightly or on-demand.

Going further

TTFT and tokens/sec are the OSS layer. On paid agents, the same std/llm@v1 stream gains per-request ITL/TPOT percentiles (pro_llm_itl_p95_ms, pro_llm_tpot_ms), per-request cost from your own price list (pro_llm_cost_usd), and the nvidia-smi-pro GPU source (SM/memory clocks, throttle reasons, per-process VRAM) — see LLM & GPU metrics.

Troubleshooting

• GPU columns are empty — nvidia-smi not on PATH, or running under WSL/older drivers; GPU collection is best-effort, everything else still works.
• vLLM model errors — the model: in your YAML must match the --model the server was started with; gated HF models need HF_TOKEN on the server side.
• Everything fails with HTTP 404/connection refused — wrong port or endpoint path: Ollama is :11434, vLLM is :8000, and the step expects the full /v1/chat/completions path in url.
• dropped_iterations climbs immediately — max_vus is too low for the arrival rate; raise the pool or lower the rate ladder.