Are LLM Performance Benchmarks Reliable? — Ashok Chandrasekar & Jason Kramberger, Google
Summary
This talk addresses the critical issue of unreliable LLM performance benchmarks, arguing that often the failure point is the benchmark harness itself, not the inference server. The speakers introduce `Inference Perf`, a CNCF project, which provides a standardized, high-fidelity solution. This tool solves common pitfalls—such as Python's Global Interpreter Lock (GIL) limitations, client-side latency inflation, and non-reproducible configurations—by offering declarative configuration, robust load generation across multiple processes, and combined client/server telemetry for accurate production-scale measurement.
Key takeaways
-
Benchmark Harness vs. System Under Test
Many performance discrepancies are incorrectly attributed to the server when the fault lies with the benchmark harness. Issues include GIL limitations in Python, client-side latency inflation (e.g., up to 58 seconds), and non-deterministic settings (e.g., model temperature set to zero).
-
Need for High Metric Fidelity
Production-scale benchmarking requires observing both the system under test and the client's behavior simultaneously. The proposed solution, `Inference Perf`, provides client-side telemetry alongside server metrics to ensure accurate diagnosis.
-
Standardization of Workloads
The `LLMD` project provides a published workload catalog defining state-of-the-art inference tasks (e.g., agentic generation, tree of thought, batch summarization) in a standardized, declarative format, improving reproducibility across different tools.
Technical details
-
Inference Perf Architecture
0s
`Inference Perf` is a CNCF project that uses a main process to schedule requests (Poisson, constant rate, or fixed concurrency) and fans them across multiple worker processes. This architecture solves the GIL problem in Python and ensures accurate load generation up to 5,000 QPS.
-
LLM Performance Metrics
0s
Reliable benchmarking requires measuring not just single QPS, but a suite of metrics including time to first token, time per output token, and adherence to Service Level Objectives (SLOs) like P90.
-
Workload Configuration
0s
The system supports complex, reproducible configurations, including defining input/output length distributions for multi-turn conversation replay, which is crucial for simulating real-world production workloads.
Mentioned resources
- Inference Perf
- LLMD
- Prism
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.