# How a Trading Firm Ships New AI Models the Same Day - Noyan Tokgozoglu, Optiver

## Executive summary

Optiver, a market-making trading firm, operates in an environment where AI decisions have immediate, measurable impacts on Profit & Loss (P&L) within hours. To scale AI rapidly and safely, the firm built a highly modular, on-premise AI platform. This platform utilizes an Inference Gateway for model routing and budget control, a 'meta harness' for portability, and an Agent Gateway to manage complex multi-agent orchestrations, ensuring that new frontier models can be benchmarked and deployed on the same day.

## Key takeaways

- Immediate Feedback Loop: Unlike traditional tech companies where value is measured over weeks or quarters, Optiver sees the P&L impact of AI changes within hours, necessitating extremely fast, measurable, and high-stakes decision-making (1:00, 2:53).
- Platform Principles: The platform must be modular to accommodate rapidly changing AI paradigms (e.g., loops, swarms, graphs) and must be built to measure performance against internal 'golden sets' rather than relying solely on external benchmarks (2:53, 7:06).
- AI Governance and Spend Control: To manage escalating AI costs, Optiver implemented optimizations like MCP tool search and CLI wrappers, focusing on optimizing token usage and giving teams visibility and agency to self-manage spending (11:30, 12:30).

## Technical details

- AI Platform Architecture: The core stack is on-premise due to regulatory requirements and IP value. Key components include: an Inference Gateway (for routing and budgeting across third-party and self-hosted models), an Agent Gateway (a single ledger for all company agents), and a Meta Harness (providing model- and harness-agnostic portability for teams) (4:30, 7:30).
- Agent Orchestration and Development: Agents are designed to be scoped to specific tasks rather than being monolithic. The system supports multi-agent orchestration, allowing agents to communicate and operate through the Agent Gateway, which helps manage complexity and technical debt (16:50, 18:30).
- Model Deployment and Benchmarking: New frontier models can be benchmarked on internal golden sets immediately upon release. The platform ensures resilience by allowing optionality between different models and providers, and supporting workloads in a DMZ on-prem zone (8:31, 10:30).

## Practical implications

- The necessity of building highly modular and abstracted platforms (like the Meta Harness) to handle rapid technological change (new models, new paradigms).
- Implementing robust governance layers (Inference Gateway, budget controls) is critical when scaling AI, especially in high-stakes, regulated environments.
- Adopting a 'federated' approach to agent development, where small, specialized agents communicate via a central ledger (Agent Gateway), rather than building single, overly complex agents.

## Topics

AI Engineering, MLOps, FinTech, High-Frequency Trading, Agent Systems, Platform Architecture, AIE CODE summit, Optiver

Source: https://www.youtube.com/watch?v=aGIUg8zKI8A