Two Bugs That Hid in Plain Sight: A vLLM Debugging Detective Story — Asaf Gardin & Yuval Belfer
This talk details two critical, non-crashing bugs found in stateful inference systems (specifically vLLM/Mamba) that manifest as 'gibberish' or logprob spikes. Both issues were surfaced by memory pressure and required advanced debugging techniques, including logprob comparison against baseline frameworks and kernel inspection. The core finding is that stateful inference systems do not fail loudly; they can lie with high confidence.
Key takeaways
-
Use Logprob Comparison for Quality Assurance
0:05
When debugging inference issues, compare the logprobs generated by the system under test (e.g., vLLM) against a plain reference implementation (e.g., Hugging Face transformers) to detect subtle divergence in token distributions, even when no explicit error occurs. (5:18)
-
Stress Test with Resource Constraints
0:03
To reproduce rare bugs, manipulate system constraints. Starving the GPU by reducing memory utilization (e.g., from 90% to 20%) or increasing the scale of rollouts per prompt can force latent bugs to surface. (3:43)
-
Thread Identity Through the Forward Pass
0:10
In complex kernels, identity can be lost when data is reduced to tensors. Propagating a unique request ID through a `forward context` allows engineers to set breakpoints and trace which specific request is executing faulty logic. (10:30)