Are LLM Performance Benchmarks Reliable? — Ashok Chandrasekar & Jason Kramberger, Google
This talk addresses the critical issue of unreliable LLM performance benchmarks, arguing that often the failure point is the benchmark harness itself, not the inference server. The speakers introduce `Inference Perf`, a CNCF project, which provides a standardized, high-fidelity solution. This tool solves common pitfalls—such as Python's Global Interpreter Lock (GIL) limitations, client-side latency inflation, and non-reproducible configurations—by offering declarative configuration, robust load generation across multiple processes, and combined client/server telemetry for accurate production-scale measurement.
Key takeaways
-
Benchmark Harness vs. System Under Test
Many performance discrepancies are incorrectly attributed to the server when the fault lies with the benchmark harness. Issues include GIL limitations in Python, client-side latency inflation (e.g., up to 58 seconds), and non-deterministic settings (e.g., model temperature set to zero).
-
Need for High Metric Fidelity
Production-scale benchmarking requires observing both the system under test and the client's behavior simultaneously. The proposed solution, `Inference Perf`, provides client-side telemetry alongside server metrics to ensure accurate diagnosis.
-
Standardization of Workloads
The `LLMD` project provides a published workload catalog defining state-of-the-art inference tasks (e.g., agentic generation, tree of thought, batch summarization) in a standardized, declarative format, improving reproducibility across different tools.