# The Algorithm That Nearly Killed Me: When Testing Isn't Enough - Andrew Paul - NDC Oslo 2026

## Executive summary

The talk uses a personal narrative—the failure of an automated insulin pump algorithm—to warn the industry that passing tests do not guarantee user safety, especially in AI/ML-driven systems. The speaker argues that traditional testing methods are insufficient when systems learn and adapt. He proposes ten practical strategies for build engineers to test for 'impact' and 'edge cases' rather than just 'coverage,' emphasizing the need for explicit documentation of 'must never' invariants and robust system-level validation.

## Key takeaways

- Testing must focus on Impact, not just Coverage: Passing unit and integration tests only prove what we know or assume; they do not highlight bigger, real-world problems. The focus must shift to testing the potential impact on the user.
- The Danger of Automated Systems (The Insulin Pump Case): The speaker's experience with an automated insulin pump demonstrated that even when the system appeared to be working (the algorithm was 'learning'), it could deliver dangerous doses or fail to respond correctly, leading to life-threatening outcomes (e.g., diabetic ketoacidosis).
- The Importance of Edge Case Documentation: Engineers must create an 'interaction matrix' to document and discuss how multiple edge cases (e.g., fraud flags interacting with retry logic and database failover) interact. This documentation provides a traceable record of anticipated failures.
- Prioritize 'Must Never' Invariants: Explicitly document and test 'must never' invariants (e.g., 'A customer must never be charged twice'). These critical rules must be run as standalone tests and trigger high-priority alarms (e.g., an SMTP email to the project manager) if violated.
- Build Observability into Business Logic: Beyond infrastructure monitoring, build code that watches for and triggers metrics when specific edge cases occur. This allows for proactive identification of systemic issues before they impact users.

## Technical details

- Diabetic Control Measurement: The HBA1C measurement indicates the average blood sugar residue over the last 90 days. High levels (above 97) indicate poor control and significant damage risk.
- Automated Insulin Delivery System: The system uses a sensor to measure blood sugar and an algorithm (running on a phone app, not the pump) to automatically deliver insulin doses. The algorithm's failure to account for complex physiological states led to dangerous outcomes.
- Property-Based Testing: A testing method where constraints are tested across a wide range of generated inputs, rather than just predefined test cases. The speaker advises embedding these properties into every single scenario.
- Feature Flag Controls: Instead of rolling out full features, use feature flags to roll out parts (A and B) to different subgroups, allowing for controlled testing of feature interactions and safe rollbacks if observability metrics flag an issue.
- Threat Modeling: A living document that identifies valuable assets and the biggest defense lines. When applied to testing, it helps identify where safety is most likely to fail due to potential crossovers or unaddressed edge cases.

## Practical implications

- Implement an interaction matrix to map and document all overlaps between edge cases.
- Build standalone, high-priority tests for 'must never' invariants, ensuring failure triggers immediate, high-visibility alerts (e.g., SMTP email).
- Use property-based testing to ensure critical constraints (like 'no insulin when blood sugar < 4') are enforced universally, not just in specific scenarios.
- Integrate observability metrics into business logic to track the simultaneous occurrence of multiple edge cases.
- Adopt a 'worst-case scenario' mindset when designing systems, asking: 'Would I trust this system with my life/wealth?'

## Topics

Software Testing, AI/ML Safety, Edge Case Management, System Reliability, Build Engineering Practices, NICE guidelines, The Gilded Rose

Source: https://www.youtube.com/watch?v=TJefrFDNXmI