# AI Agents for Performance: Ship Faster, Pay Less — Rajat Shah, Netflix

## Executive summary

This talk outlines how Netflix leveraged AI agents to automate performance engineering, addressing the bottleneck of manually identifying and fixing inefficient code patterns in production services. The process involves feeding profiling data (call stacks, CPU usage) into an LLM agent, which identifies suboptimal algorithms (e.g., quadratic-time patterns). The agent then proposes a fix, validates it against a canary deployment using real production traffic, and can even scale the fix across multiple services by identifying common anti-patterns in a centralized catalog. This shifts performance optimization from a reactive, manual effort to a proactive, automated part of the SDLC.

## Key takeaways

- The Performance Bottleneck: Traditional performance engineering is highly manual and time-consuming (taking minutes just to identify hot paths), making it difficult to scale across large codebases. This bottleneck increases as coding agents write code faster, potentially introducing inefficiencies.
- AI Agent Workflow for Optimization: An LLM agent can automate the entire loop: reading profiling data (call stack/CPU time), identifying anti-patterns (like O(N²) loops), locating the code path in the Git repo, proposing a fix, and validating it via canary deployment.
- Scaling Fixes with Pattern Catalogs: Instead of fixing one instance, agents can search across multiple services using a centralized 'Pattern/Anti-pattern Catalog' (stored in a Git repo) to find and propose fixes for the same bad pattern repeatedly implemented elsewhere.
- Shifting Left: Proactive Performance: The goal is to move performance improvement left in the SDLC. The catalog can be used proactively—at code review or even during code authoring—to suggest optimal patterns before suboptimal code reaches production.

## Technical details

- Profiling Data Structure: Profilers (regardless of language like Java, Python, Go) output similar data: call stack, self CPU, and inclusive CPU time spent on methods. This structured format is suitable for LLM consumption.
- Anti-Pattern Identification: LLMs can identify common performance anti-patterns (e.g., O(N²) loops, repeated object allocation in loops) purely by analyzing the call stack data, not just reading the source code.
- Agent Workflow Steps: The agent must: 1) Identify a pattern from profiling data. 2) Search the Git repo for the method definition at the specific production commit. 3) Trace the entire call path. 4) Generate a code review/fix proposal.
- Validation and Guardrails: The process requires multiple layers of verification: Unit tests, integration tests (functional correctness), and mandatory canary deployments. The observability report from the canary run provides 'ground truth' (e.g., CPU reduction, latency change) for the agent to make a judgment call.
- Pattern Catalog Implementation: The catalog should be a central, ever-growing Git repository containing documented patterns and anti-patterns. Entries should include hints for LLM querying, confirmed services, and confidence levels.

## Practical implications

- Establish a centralized 'Pattern/Anti-pattern Catalog' (Git repo) to capture performance findings across all services.
- Build foundational automation hooks: automated profiling triggers, data download pipelines, and mandatory canary deployment validation logic.
- Implement the agent workflow in stages: Start with reactive fixes (profiling -> fix proposal), then move toward proactive checks (inline code review suggestions based on the catalog).
- Ensure human oversight remains critical for approving any optimized code change due to business context risk.

## Topics

AI Agents, Performance Engineering, Software Development Life Cycle (SDLC), LLMs, Canary Deployments, Code Optimization, Jeff Dean's Blog Post, PyTorch torch fix code repo

Source: https://www.youtube.com/watch?v=CgsWxRUY5Eo