Two Bugs That Hid in Plain Sight: A vLLM Debugging Detective Story — Asaf Gardin & Yuval Belfer
Summary
This talk details two critical, non-crashing bugs found in stateful inference systems (specifically vLLM/Mamba) that manifest as 'gibberish' or logprob spikes. Both issues were surfaced by memory pressure and required advanced debugging techniques, including logprob comparison against baseline frameworks and kernel inspection. The core finding is that stateful inference systems do not fail loudly; they can lie with high confidence.
Key takeaways
-
Use Logprob Comparison for Quality Assurance
0:05
When debugging inference issues, compare the logprobs generated by the system under test (e.g., vLLM) against a plain reference implementation (e.g., Hugging Face transformers) to detect subtle divergence in token distributions, even when no explicit error occurs. (5:18)
-
Stress Test with Resource Constraints
0:03
To reproduce rare bugs, manipulate system constraints. Starving the GPU by reducing memory utilization (e.g., from 90% to 20%) or increasing the scale of rollouts per prompt can force latent bugs to surface. (3:43)
-
Thread Identity Through the Forward Pass
0:10
In complex kernels, identity can be lost when data is reduced to tensors. Propagating a unique request ID through a `forward context` allows engineers to set breakpoints and trace which specific request is executing faulty logic. (10:30)
Technical details
-
Imposter Request (Prefill/Decode Timing)
11s
The first bug occurred when the scheduler incorrectly initiated the decode kernel before the prefill kernel for certain requests. For Mamba, reading the state first (decode) using stale data from previous requests generated gibberish. The fix required ensuring the scheduler correctly classified requests that had never been computed, forcing them through prefill first. (11:40)
-
Index Overflow in Mamba State Cache
19s
The second bug was a silent failure caused by a 32-bit index (`uint32`) wrapping around past four billion. This was triggered by memory pressure. The fix was changing the data type from `uint32` to `size_t` (unsigned 64-bit) to prevent the overflow. (18:50)
Mentioned resources
- AI21 Blog Post
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.