# Training Frontier Models to Out-Think Hackers — Uri Rolls, Arithmetic & Thom Wolf, Hugging Face

## Executive summary

This talk addresses the rapidly shifting economics of cyber security due to increasingly powerful AI models. While frontier models can perform extensive reconnaissance and find vulnerabilities, they often fail at the critical 'logical leap' required for exploitation—the ability to reason across complex system states. Arithmetic proposes a new defense paradigm: creating specialized benchmarks focused on access control by having human researchers discover real-world zero days in open-source software. The goal is to train models to replicate the deep reasoning and multi-step logic of skilled attackers, thereby giving defenders a lasting edge.

## Key takeaways

- AI Models Lack World Modeling for Exploitation: Current large language models (LLMs) struggle with building dynamic world models. The benchmark developed by Arithmetic shows that even advanced models like GPT 5.5 and Opus fail to make the necessary logical leap required to exploit a system, despite successfully reaching the vulnerability check.
- Open Source Models are Key to Cyber Defense: The solution to modern cyber challenges requires open-source models. The speaker argues that relying solely on a few large companies is insufficient, and collaboration using open source LLMs will be crucial for building the next generation of defensive systems.
- Focusing on Access Control Vulnerabilities: Arithmetic's benchmark focuses specifically on access control, as this is the primary entry point for most attacks. The vulnerabilities tested are 'logic based,' meaning they exploit discrepancies between how different parts of a system check permissions (e.g., checking by name vs. checking by ID).

## Technical details

- Benchmark Methodology and Grading: The benchmark operates in a blackbox environment where the model does not see the underlying code or know about the zero day. The process involves: 1) Finding real-world vulnerabilities (zero days) using human researchers. 2) Creating chained, multi-application environments (e.g., Keycloak, Vault, and a broker). 3) Using deterministic grading that verifies every single step of discovery and exploitation to measure how deep the model can penetrate the system.
- The Keycloak/Vault Exploit Demonstration: A real-world task involves escalating privileges from a low-privileged user to production code via a chain linking Keycloak, Vault, and a broker. The specific vulnerability exploited is that one system check validates an admin by name, while another checks by ID, allowing a user to rename themselves to the admin's identity to inherit elevated privileges.

## Practical implications

- Defensive systems must evolve to incorporate LLM-based reasoning and logic leaps rather than relying solely on traditional bug patching.
- The development of standardized, open-source benchmarks for cyber security is critical for accelerating defensive AI capabilities.
- Future secure architectures will likely require models capable of understanding complex state changes across chained microservices (e.g., Keycloak $\rightarrow$ Vault $\rightarrow$ Broker).

## Topics

Cyber Security, AI Ethics and Safety, Large Language Models (LLMs), Access Control, Zero-Day Exploitation, Open Source AI, Keycloak, Vault, Hugging Face, Arithmetic

Source: https://www.youtube.com/watch?v=O-CBZ3JtRvo