# Safe Systems in a Chaotic World - Bryan Hunter - NDC Oslo 2026

## Executive summary

The talk argues that in an era of systemic chaos (AI, geopolitical instability, etc.), software development must shift its focus from preventing failures to enforcing immutable system invariants and constraints. Drawing on safety-critical engineering principles (STAMP/STPA) and manufacturing concepts (Pokeyoke), the speaker advocates for designing systems that define what they *must* refuse to do, thereby shrinking the 'blast radius' of potential errors, whether caused by human oversight, bad actors, or complex interactions in distributed cloud environments.

## Key takeaways

- Focus on Invariants, Not Failure Prevention: Instead of worrying about handling every possible failure case, the goal is to identify and enforce invariants—properties that must remain true at every observable moment, regardless of environmental changes. Constraints are the rules that keep these invariants true.
- System Failures are Model Divergence: Accidents are not typically component failures, but rather failures in the system's control model. A loss occurs when a controller's process model (its internal belief of reality) diverges from the actual process state. This applies to human decisions (e.g., a CEO replacing developers with AI) as well as software.
- Mistake-Proofing (Pokeyoke) is Superior to Diligence: Relying on human diligence is unreliable. The principle of Pokeyoke (mistake-proofing) dictates that the physical or digital system should be designed to make errors impossible or immediately visible, thereby respecting the intelligence of the worker.
- Adopt Safety Engineering for Modern Threats: Safety engineering frameworks like STAMP and STPA are crucial for analyzing complex systems, especially those involving LLMs and AI. They help identify hazards and define safety controls by modeling the control structure, making the approach 'adversary agnostic' (meaning it doesn't care if the action is accidental or malicious).
- Build for Continuous Availability and Resilience: To achieve continuous availability on commodity cloud infrastructure, systems must be designed to be cloud-agnostic and geographically distributed. Using tools like open Tofu/Terraform and robust platforms like the Erlang VM (The Beam) allows for building 'asteroid proof' systems.

## Technical details

- System Theoretic Accident Model and Processes (STAMP/STPA): STAMP shifts the focus from root cause analysis (which only fixes symptoms) to analyzing the control structure. It views accidents as resulting from the interaction among components that violate system safety constraints. STPA is the method used to apply this theory, identifying unsafe control actions (e.g., provided too early, too late, or never delivered).
- Pokeyoke (Mistake-Proofing): A concept from Japanese manufacturing, Pokeyoke involves designing physical or digital mechanisms (like guide pins or type systems) to prevent mistakes from becoming defects. In software, this translates to making illegal states unrepresentable, often through exhaustive pattern matching or compile-time checks.
- Fault-Tolerant Programming (Erlang/Elixir/Gleam): The Erlang VM (The Beam) is highlighted for its ability to handle massive concurrency and fault tolerance by isolating processes into separate heaps. Communication occurs only through message passing, ensuring that a corrupted process model cannot spill chaos onto a neighbor. Gleam is noted as a strongly typed language running on the Beam, adding compile-time constraints and preventing illegal states.
- Property-Based Testing: This testing method, exemplified by John Hughes' QuickCheck, is used to verify properties across a vast range of inputs (e.g., 100,000 iterations) rather than just known test cases. It is a powerful way to find unexpected edge cases and ensure constraints hold.

## Practical implications

- When designing software, prioritize defining and enforcing system invariants (constraints) over attempting to write exhaustive error handling for every possible failure.
- Apply mistake-proofing principles (Pokeyoke) to both physical and digital processes to eliminate the need for constant human vigilance.
- Utilize formal methods and advanced testing (like property-based testing) to prove that constraints hold across all possible inputs, especially when integrating complex components like LLMs.
- For distributed systems, adopt fault-tolerant architectures (e.g., using the Erlang VM/Beam) that isolate failures and ensure continuous operation even when components fail or are compromised.
- When dealing with modern AI/LLMs, treat them as potential vulnerabilities and model their interactions as part of the system's control structure, rather than trusting their output implicitly.

## Topics

System Safety Engineering, Constraints, Invariants, Fault Tolerance, LLM Security, Distributed Systems, Process Modeling, Safety Engineering (STAMP/STPA), Engineering a Safer World, The Toyota Way, The Checklist Manifesto, The Art of Computer Programming (TLA+)

Source: https://www.youtube.com/watch?v=UeLdsKfgGaA