# How Unify cut its AI agent costs 95% in two weeks

## Executive summary

Unify's agent platform automates go-to-market tasks by giving sales representatives 'an engineer in their back pocket.' The discussion details how Unify achieved a 90-95% cost optimization within two weeks of launch. Key architectural insights include optimizing for prompt caching hit rates, designing subagents as simple function calls, and implementing robust data handling over tabular data using virtualized Pandas-like interfaces. Furthermore, the talk emphasizes advanced evaluation techniques (DQA sets) and stateful, durable cloud execution environments like Monty to ensure scalability and cost control.

## Key takeaways

- Cost Optimization via Prompt Caching: Achieving a high prompt cache hit rate is critical for cost-effectiveness. The limit is approximately 15 requests per second (RPS) within OpenAI's cache, and developers must implement strategies to maximize hits, as providers won't solve this distribution problem for you.
- Subagents are Function Calls: Architecturally, Unify treats a subagent not as a complex process, but simply as an arbitrary function call. This simplifies the system and allows for efficient execution patterns like mapping over rows.
- The Importance of Planning Steps: Adding an initial planning step (e.g., using GPT-4) to the agent workflow significantly improves quality and efficiency by forcing the model to scout multiple potential trajectories before executing.
- Durable Cloud Execution: To run agents in a cloud environment without full VMs, using specialized tools like Monty (a Python REPL that suspends) is crucial for maintaining statefulness and strong tenancy.
- Adversarial LLM Judging: When using an LLM as a judge, it must be from a different model family than the original agent model to avoid 'mode collapse' or groupthink, ensuring an adversarial evaluation.

## Technical details

- Agent Architecture: The system is designed around the principle that a subagent is merely an arbitrary function call, allowing for scalable execution patterns like mapping over thousands of rows.
- Data Handling and Pandas Emulation: To enable models to operate effectively on tabular data (e.g., CSVs), Unify implemented a class that wraps the database, reimplementing key Pandas-like functions (filtering, mapping, sorting) in TypeScript.
- Prompt Caching Optimization: The system achieves high cache hit rates by hashing the User ID and combining it with a random number to distribute the load across multiple prompt cache keys, mitigating the 15 RPS limit.
- Stateful Execution Environment: Instead of full VMs, Unify uses Monty (a Python REPL) to provide a stateful, durable execution environment that suspends and resumes code execution while maintaining strong tenancy.
- Evaluation Methodology: Rigorous evaluation requires creating Dedicated QA (DQA) sets—specific distributions of usage cases (e.g., adversarial prompts, specific tool call failures)—and running multiple passes (Pass K) to ensure reliability.

## Practical implications

- When designing agent workflows, prioritize robust planning steps at the beginning to guide model reasoning and improve outcome quality.
- For high-volume agents, focus engineering efforts on optimizing prompt caching strategies (e.g., key hashing) rather than relying solely on provider features.
- Treat subagents as modular function calls; this simplifies architecture and allows for efficient parallelization and cost management.
- When building data access layers for LLMs, virtualizing the underlying database with Pandas-like functions is a highly effective pattern for structured data manipulation.

## Topics

AI Agents, LLM Optimization, Go-to-Market Automation, Cloud Architecture, Prompt Engineering, OpenAI API, Monty, Linksmith

Source: https://www.youtube.com/watch?v=6898VdRtKDE