How a Trading Firm Ships New AI Models the Same Day - Noyan Tokgozoglu, Optiver
Summary
Optiver, a market-making trading firm, operates in an environment where AI decisions have immediate, measurable impacts on Profit & Loss (P&L) within hours. To scale AI rapidly and safely, the firm built a highly modular, on-premise AI platform. This platform utilizes an Inference Gateway for model routing and budget control, a 'meta harness' for portability, and an Agent Gateway to manage complex multi-agent orchestrations, ensuring that new frontier models can be benchmarked and deployed on the same day.
Key takeaways
-
Immediate Feedback Loop
2:00
Unlike traditional tech companies where value is measured over weeks or quarters, Optiver sees the P&L impact of AI changes within hours, necessitating extremely fast, measurable, and high-stakes decision-making (1:00, 2:53).
-
Platform Principles
2:53
The platform must be modular to accommodate rapidly changing AI paradigms (e.g., loops, swarms, graphs) and must be built to measure performance against internal 'golden sets' rather than relying solely on external benchmarks (2:53, 7:06).
-
AI Governance and Spend Control
11:30
To manage escalating AI costs, Optiver implemented optimizations like MCP tool search and CLI wrappers, focusing on optimizing token usage and giving teams visibility and agency to self-manage spending (11:30, 12:30).
Technical details
-
AI Platform Architecture
330s
The core stack is on-premise due to regulatory requirements and IP value. Key components include: an Inference Gateway (for routing and budgeting across third-party and self-hosted models), an Agent Gateway (a single ledger for all company agents), and a Meta Harness (providing model- and harness-agnostic portability for teams) (4:30, 7:30).
-
Agent Orchestration and Development
1010s
Agents are designed to be scoped to specific tasks rather than being monolithic. The system supports multi-agent orchestration, allowing agents to communicate and operate through the Agent Gateway, which helps manage complexity and technical debt (16:50, 18:30).
-
Model Deployment and Benchmarking
253s
New frontier models can be benchmarked on internal golden sets immediately upon release. The platform ensures resilience by allowing optionality between different models and providers, and supporting workloads in a DMZ on-prem zone (8:31, 10:30).
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.