How Unify cut its AI agent costs 95% in two weeks
Unify's agent platform automates go-to-market tasks by giving sales representatives 'an engineer in their back pocket.' The discussion details how Unify achieved a 90-95% cost optimization within two weeks of launch. Key architectural insights include optimizing for prompt caching hit rates, designing subagents as simple function calls, and implementing robust data handling over tabular data using virtualized Pandas-like interfaces. Furthermore, the talk emphasizes advanced evaluation techniques (DQA sets) and stateful, durable cloud execution environments like Monty to ensure scalability and cost control.
Key takeaways
-
Cost Optimization via Prompt Caching
30:25
Achieving a high prompt cache hit rate is critical for cost-effectiveness. The limit is approximately 15 requests per second (RPS) within OpenAI's cache, and developers must implement strategies to maximize hits, as providers won't solve this distribution problem for you.
-
Subagents are Function Calls
25:50
Architecturally, Unify treats a subagent not as a complex process, but simply as an arbitrary function call. This simplifies the system and allows for efficient execution patterns like mapping over rows.
-
The Importance of Planning Steps
38:20
Adding an initial planning step (e.g., using GPT-4) to the agent workflow significantly improves quality and efficiency by forcing the model to scout multiple potential trajectories before executing.
-
Durable Cloud Execution
To run agents in a cloud environment without full VMs, using specialized tools like Monty (a Python REPL that suspends) is crucial for maintaining statefulness and strong tenancy.
-
Adversarial LLM Judging
When using an LLM as a judge, it must be from a different model family than the original agent model to avoid 'mode collapse' or groupthink, ensuring an adversarial evaluation.