Safe Systems in a Chaotic World - Bryan Hunter - NDC Oslo 2026
Summary
The talk argues that in an era of systemic chaos (AI, geopolitical instability, etc.), software development must shift its focus from preventing failures to enforcing immutable system invariants and constraints. Drawing on safety-critical engineering principles (STAMP/STPA) and manufacturing concepts (Pokeyoke), the speaker advocates for designing systems that define what they *must* refuse to do, thereby shrinking the 'blast radius' of potential errors, whether caused by human oversight, bad actors, or complex interactions in distributed cloud environments.
Key takeaways
-
Focus on Invariants, Not Failure Prevention
19:00
Instead of worrying about handling every possible failure case, the goal is to identify and enforce invariants—properties that must remain true at every observable moment, regardless of environmental changes. Constraints are the rules that keep these invariants true.
-
System Failures are Model Divergence
22:00
Accidents are not typically component failures, but rather failures in the system's control model. A loss occurs when a controller's process model (its internal belief of reality) diverges from the actual process state. This applies to human decisions (e.g., a CEO replacing developers with AI) as well as software.
-
Mistake-Proofing (Pokeyoke) is Superior to Diligence
27:00
Relying on human diligence is unreliable. The principle of Pokeyoke (mistake-proofing) dictates that the physical or digital system should be designed to make errors impossible or immediately visible, thereby respecting the intelligence of the worker.
-
Adopt Safety Engineering for Modern Threats
29:10
Safety engineering frameworks like STAMP and STPA are crucial for analyzing complex systems, especially those involving LLMs and AI. They help identify hazards and define safety controls by modeling the control structure, making the approach 'adversary agnostic' (meaning it doesn't care if the action is accidental or malicious).
-
Build for Continuous Availability and Resilience
43:20
To achieve continuous availability on commodity cloud infrastructure, systems must be designed to be cloud-agnostic and geographically distributed. Using tools like open Tofu/Terraform and robust platforms like the Erlang VM (The Beam) allows for building 'asteroid proof' systems.
Technical details
-
System Theoretic Accident Model and Processes (STAMP/STPA)
1750s
STAMP shifts the focus from root cause analysis (which only fixes symptoms) to analyzing the control structure. It views accidents as resulting from the interaction among components that violate system safety constraints. STPA is the method used to apply this theory, identifying unsafe control actions (e.g., provided too early, too late, or never delivered).
-
Pokeyoke (Mistake-Proofing)
1620s
A concept from Japanese manufacturing, Pokeyoke involves designing physical or digital mechanisms (like guide pins or type systems) to prevent mistakes from becoming defects. In software, this translates to making illegal states unrepresentable, often through exhaustive pattern matching or compile-time checks.
-
Fault-Tolerant Programming (Erlang/Elixir/Gleam)
2200s
The Erlang VM (The Beam) is highlighted for its ability to handle massive concurrency and fault tolerance by isolating processes into separate heaps. Communication occurs only through message passing, ensuring that a corrupted process model cannot spill chaos onto a neighbor. Gleam is noted as a strongly typed language running on the Beam, adding compile-time constraints and preventing illegal states.
-
Property-Based Testing
2400s
This testing method, exemplified by John Hughes' QuickCheck, is used to verify properties across a vast range of inputs (e.g., 100,000 iterations) rather than just known test cases. It is a powerful way to find unexpected edge cases and ensure constraints hold.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.