Codex, Behind the Harness — Dominik Kundel, OpenAI
Summary
The Codex harness provides a comprehensive framework for building advanced, agentic AI systems. The system relies on two open protocols—the App Server (for UI-to-harness communication) and the Responses API (for harness-to-inference)—allowing developers to build custom agents regardless of their interface. Key features include sophisticated context management (using deferred tools and capping available skills), robust action capabilities (async tasks, code execution for computer use, and file system interaction via an 'apply patch tool'), and mandatory sandboxing layers (e.g., Seatbelt on macOS, Bubblewrap on Linux). Furthermore, the architecture addresses real-world enterprise concerns like security (Auto Review subagents) and performance (WebSocket mode and Auto Compaction), making it suitable for complex, long-running build processes.
Key takeaways
-
Open Ecosystem Protocols
4:00
The Codex harness is built on open standards: the App Server protocol (UI to harness) and the Responses API (harness to inference). These protocols allow developers to build custom UIs or integrate with different model providers, ensuring flexibility.
-
Advanced Context Management
6:45
To manage context size and maintain cacheability, the system uses 'deferred tools,' which are not added directly to the context window but are surfaced via tool search. The available skills list is capped at 2% of the total context window.
-
Secure and Robust Actions
10:20
Actions are handled through specialized tools: file edits use an 'apply patch tool,' while general navigation uses a shell tool (often defaulting to `ripgrep`). All interactions occur within mandatory sandboxes (e.g., Seatbelt on macOS, Bubblewrap on Linux).
-
Mitigating Approval Fatigue
15:20
An 'auto review subagent' is implemented to automatically judge high-risk actions against user authorization and the transcript context, reducing reliance on manual full-access approvals.
-
Optimized Communication
17:15
The system transitioned from Server-Sent Events (SSE) over HTTP to WebSocket mode. This persistent connection saves network overhead and provides stateful context, ensuring only changed data is transmitted.
Technical details
-
Protocols & Architecture
240s
The Codex harness uses two primary protocols: the App Server (for UI interactions) and the Responses API. The Responses API was redesigned to support complex agentic capabilities like web search and image generation, and it is designed as an open schema for partners (e.g., Ollama, LM Studio).
-
Context Construction & Efficiency
360s
To balance context size, flexibility, and cacheability, the system implements deferred tools (available only via tool search) and limits the available skills list to a maximum of 2% of the total context window. This is possible since GPT-5.4.
-
Agent Actions & Execution
620s
Agents perform actions through specialized tools: Async tasks (e.g., sub-agents via `spawn agent` tool), Computer Use (now flexible enough to use code execution in JavaScript or Python, demonstrated with Playwright for browser interaction), and File System interactions (using an 'apply patch tool' for diffs).
-
Security & Sandboxing
750s
All file system interactions are routed through a sandbox layer. Specific sandboxes include Seatbelt (macOS), Bubblewrap (Linux), and a custom open-source solution built by the team for Windows.
-
Long-Horizon Goal Setting
1100s
For complex, multi-step goals, the system injects a continuation prompt until the model explicitly calls an `update goal tool`, requiring concrete and verifiable objectives rather than general essays.
Mentioned resources
- Codex harness
- App Server Protocol
- Responses API
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.