How Unify cut its AI agent costs 95% in two weeks
Summary
Unify's agent platform automates go-to-market tasks by giving sales representatives 'an engineer in their back pocket.' The discussion details how Unify achieved a 90-95% cost optimization within two weeks of launch. Key architectural insights include optimizing for prompt caching hit rates, designing subagents as simple function calls, and implementing robust data handling over tabular data using virtualized Pandas-like interfaces. Furthermore, the talk emphasizes advanced evaluation techniques (DQA sets) and stateful, durable cloud execution environments like Monty to ensure scalability and cost control.
Key takeaways
-
Cost Optimization via Prompt Caching
30:25
Achieving a high prompt cache hit rate is critical for cost-effectiveness. The limit is approximately 15 requests per second (RPS) within OpenAI's cache, and developers must implement strategies to maximize hits, as providers won't solve this distribution problem for you.
-
Subagents are Function Calls
25:50
Architecturally, Unify treats a subagent not as a complex process, but simply as an arbitrary function call. This simplifies the system and allows for efficient execution patterns like mapping over rows.
-
The Importance of Planning Steps
38:20
Adding an initial planning step (e.g., using GPT-4) to the agent workflow significantly improves quality and efficiency by forcing the model to scout multiple potential trajectories before executing.
-
Durable Cloud Execution
To run agents in a cloud environment without full VMs, using specialized tools like Monty (a Python REPL that suspends) is crucial for maintaining statefulness and strong tenancy.
-
Adversarial LLM Judging
When using an LLM as a judge, it must be from a different model family than the original agent model to avoid 'mode collapse' or groupthink, ensuring an adversarial evaluation.
Technical details
-
Agent Architecture
1550s
The system is designed around the principle that a subagent is merely an arbitrary function call, allowing for scalable execution patterns like mapping over thousands of rows.
-
Data Handling and Pandas Emulation
2400s
To enable models to operate effectively on tabular data (e.g., CSVs), Unify implemented a class that wraps the database, reimplementing key Pandas-like functions (filtering, mapping, sorting) in TypeScript.
-
Prompt Caching Optimization
2080s
The system achieves high cache hit rates by hashing the User ID and combining it with a random number to distribute the load across multiple prompt cache keys, mitigating the 15 RPS limit.
-
Stateful Execution Environment
Instead of full VMs, Unify uses Monty (a Python REPL) to provide a stateful, durable execution environment that suspends and resumes code execution while maintaining strong tenancy.
-
Evaluation Methodology
Rigorous evaluation requires creating Dedicated QA (DQA) sets—specific distributions of usage cases (e.g., adversarial prompts, specific tool call failures)—and running multiple passes (Pass K) to ensure reliability.
Mentioned resources
- OpenAI API
- Monty
- Linksmith
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.