The Algorithm That Nearly Killed Me: When Testing Isn't Enough - Andrew Paul - NDC Oslo 2026
Summary
The talk uses a personal narrative—the failure of an automated insulin pump algorithm—to warn the industry that passing tests do not guarantee user safety, especially in AI/ML-driven systems. The speaker argues that traditional testing methods are insufficient when systems learn and adapt. He proposes ten practical strategies for build engineers to test for 'impact' and 'edge cases' rather than just 'coverage,' emphasizing the need for explicit documentation of 'must never' invariants and robust system-level validation.
Key takeaways
-
Testing must focus on Impact, not just Coverage
24:13
Passing unit and integration tests only prove what we know or assume; they do not highlight bigger, real-world problems. The focus must shift to testing the potential impact on the user.
-
The Danger of Automated Systems (The Insulin Pump Case)
27:10
The speaker's experience with an automated insulin pump demonstrated that even when the system appeared to be working (the algorithm was 'learning'), it could deliver dangerous doses or fail to respond correctly, leading to life-threatening outcomes (e.g., diabetic ketoacidosis).
-
The Importance of Edge Case Documentation
34:10
Engineers must create an 'interaction matrix' to document and discuss how multiple edge cases (e.g., fraud flags interacting with retry logic and database failover) interact. This documentation provides a traceable record of anticipated failures.
-
Prioritize 'Must Never' Invariants
37:30
Explicitly document and test 'must never' invariants (e.g., 'A customer must never be charged twice'). These critical rules must be run as standalone tests and trigger high-priority alarms (e.g., an SMTP email to the project manager) if violated.
-
Build Observability into Business Logic
40:00
Beyond infrastructure monitoring, build code that watches for and triggers metrics when specific edge cases occur. This allows for proactive identification of systemic issues before they impact users.
Technical details
-
Diabetic Control Measurement
360s
The HBA1C measurement indicates the average blood sugar residue over the last 90 days. High levels (above 97) indicate poor control and significant damage risk.
-
Automated Insulin Delivery System
1630s
The system uses a sensor to measure blood sugar and an algorithm (running on a phone app, not the pump) to automatically deliver insulin doses. The algorithm's failure to account for complex physiological states led to dangerous outcomes.
-
Property-Based Testing
2150s
A testing method where constraints are tested across a wide range of generated inputs, rather than just predefined test cases. The speaker advises embedding these properties into every single scenario.
-
Feature Flag Controls
2650s
Instead of rolling out full features, use feature flags to roll out parts (A and B) to different subgroups, allowing for controlled testing of feature interactions and safe rollbacks if observability metrics flag an issue.
-
Threat Modeling
2800s
A living document that identifies valuable assets and the biggest defense lines. When applied to testing, it helps identify where safety is most likely to fail due to potential crossovers or unaddressed edge cases.
Mentioned resources
- NICE guidelines
- The Gilded Rose
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.