Training Frontier Models to Out-Think Hackers — Uri Rolls, Arithmetic & Thom Wolf, Hugging Face
Summary
This talk addresses the rapidly shifting economics of cyber security due to increasingly powerful AI models. While frontier models can perform extensive reconnaissance and find vulnerabilities, they often fail at the critical 'logical leap' required for exploitation—the ability to reason across complex system states. Arithmetic proposes a new defense paradigm: creating specialized benchmarks focused on access control by having human researchers discover real-world zero days in open-source software. The goal is to train models to replicate the deep reasoning and multi-step logic of skilled attackers, thereby giving defenders a lasting edge.
Key takeaways
-
AI Models Lack World Modeling for Exploitation
17:01
Current large language models (LLMs) struggle with building dynamic world models. The benchmark developed by Arithmetic shows that even advanced models like GPT 5.5 and Opus fail to make the necessary logical leap required to exploit a system, despite successfully reaching the vulnerability check.
-
Open Source Models are Key to Cyber Defense
5:21
The solution to modern cyber challenges requires open-source models. The speaker argues that relying solely on a few large companies is insufficient, and collaboration using open source LLMs will be crucial for building the next generation of defensive systems.
-
Focusing on Access Control Vulnerabilities
11:21
Arithmetic's benchmark focuses specifically on access control, as this is the primary entry point for most attacks. The vulnerabilities tested are 'logic based,' meaning they exploit discrepancies between how different parts of a system check permissions (e.g., checking by name vs. checking by ID).
Technical details
-
Benchmark Methodology and Grading
840s
The benchmark operates in a blackbox environment where the model does not see the underlying code or know about the zero day. The process involves: 1) Finding real-world vulnerabilities (zero days) using human researchers. 2) Creating chained, multi-application environments (e.g., Keycloak, Vault, and a broker). 3) Using deterministic grading that verifies every single step of discovery and exploitation to measure how deep the model can penetrate the system.
-
The Keycloak/Vault Exploit Demonstration
1021s
A real-world task involves escalating privileges from a low-privileged user to production code via a chain linking Keycloak, Vault, and a broker. The specific vulnerability exploited is that one system check validates an admin by name, while another checks by ID, allowing a user to rename themselves to the admin's identity to inherit elevated privileges.
Mentioned resources
- Keycloak
- Vault
- Hugging Face
- Arithmetic
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.