How to Build an AI Code Review Agent - Daksh Gupta, Greptile
Summary
Daksh Gupta, co-founder of Greptile, outlines the advanced steps required to build an effective AI code review agent, moving far beyond simple sandbox-plus-webhook setups. The presentation details six key improvements: optimizing prompts to focus on specific bug types and skip nitpicks; utilizing a dataset of historical PRs; implementing model inversion (using different LLMs for generation vs. review); integrating deep codebase context (including docs and tickets); performing autonomous, dependency-rich testing; and optimizing inference costs by routing subtasks to cheaper models. Implementing these fixes can significantly improve bug detection rates and reduce operational costs.
Key takeaways
-
Initial Setup Limitations (The Simple Version)
A basic setup (sandbox + agent + webhook) is insufficient because it will miss bugs, lack codebase context, be expensive ($3–$25 per PR), and generate excessive nitpicky comments. Bugs are described as 'fractally infinite,' meaning simple reviews will always miss critical issues.
-
Fix 1: Prompt Optimization
10:02
Prompts must be optimized to specifically look for known bug types (derived from internal data science on past outages) and explicitly instruct the agent to skip nitpicks. The prompt must guide the agent to focus on catastrophic problems, not minor stylistic complaints.
-
Fix 2: Historical Dataset and Model Inversion
12:24
To improve confidence, the agent must be run against a real-time updating dataset of historical PRs that caused outages. Furthermore, 'model inversion' is critical: if the code was written by Model A (e.g., Claude), it should be reviewed by Model B (e.g., GPT) to catch different types of bugs. Using different models can increase the catch rate by 10% in terms of incidents.
-
Fix 3: Codebase Context and Living Wiki
18:24
Relying solely on `grep` is imprecise and expensive. A superior method is building an internal knowledge base that indexes the entire codebase, including authentication flows and file relationships. The most effective approach is creating a 'living wiki' that disintermediates context from sources like Jira, Linear, Confluence, and Notion, providing a 'world model' of the company's operations.
-
Fix 4: Autonomous Testing and Cost Efficiency
For comprehensive testing, agents must operate in dependency-rich sandboxes (configurable virtual machines) and use computer vision to perform end-to-end QA, catching issues like UI elements hidden on specific devices (e.g., iPhones). To manage costs, the process must use a mix of models: frontier models for critical tasks (e.g., bug analysis) and smaller, cheaper models for the long tail of subtasks (e.g., file reads).
Technical details
-
Codebase Indexing
1104s
The goal is to move beyond imprecise tools like `grep` by creating an internal knowledge base that catalogs and indexes the entire codebase, providing flow diagrams and file relationships to improve context awareness.
-
Model Inversion
902s
To prevent the review agent from failing in the same ways the generating model does, use a different LLM for review than was used for code generation (e.g., use GPT to review code written by Claude).
-
Autonomous Testing Environment
Requires building dependency-rich sandboxes (configurable VMs) and documenting all application flows. The agent uses computer vision to test end-to-end, identifying issues like UI elements that are technically present but functionally hidden.
-
Cost Optimization
The review process should be segmented into 20+ subtasks, each assigned a specific model. Frontier models are reserved for the most critical tasks (e.g., bug analysis), while smaller, cheaper models handle the bulk of the work (e.g., finding context or performing file reads).
Mentioned resources
- Greptile
- GitHub/GitLab
- Firecracker
- OpenCode or Pi
- Jira, Linear, Confluence, Notion
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.