# Always-on agents run production without the on-call tax — Justin Smith, Resolve AI

## Executive summary

The talk introduces the concept of 'always-on agents' designed to automate operational tasks in complex production environments, thereby reducing the burden of manual on-call work. While CI/CD handles baseline checks well, the biggest gap is monitoring non-alerted changes—such as feature flag rollouts or infrastructure updates—that require continuous context understanding. Background agents can run autonomously (on schedules, events, or messages) to perform deep analysis, root cause investigations, and proactive health checks across systems like Kafka pipelines.

## Key takeaways

- The Operational Bottleneck: A significant portion of an engineer's time (estimated at 70%) is spent running code in production—maintaining platforms, debugging incidents, and handling alerts—rather than writing it. This complexity increases with the velocity of change driven by AI.
- Background Agents vs. Incident Response: While on-call agents handle immediate alerts and incidents, background agents address the 'long tail' of operational work—such as routine health checks, summarizing handoffs, or watching for subtle performance drifts (e.g., P99 drift) that don't trigger an alert.
- The Importance of Context: Execution is easy; production context is hard. The value lies in building knowledge systems that can determine if a metric 'smells wrong' or understand the causal chain impact of a change, rather than just loading a dashboard.

## Technical details

- Agent Architecture: Agents operate in a cloud sandbox with their own file system, ensuring state persistence regardless of the user's local machine status. They utilize a learning loop to capture knowledge and understand how services interact as the environment evolves.
- Agent Triggers: Agents can be triggered in multiple ways: on a schedule (e.g., weekly reports), by event streams (e.g., CI/CD pipeline completion, Slack messages), or via direct message commands.
- Deployment Monitoring: Agents can analyze GitHub release tags to understand the specific changes (e.g., 'checkout replacing currency service'). They then build customized check plans, monitoring targeted KPIs like checkout latency and error rates, following causal chains into systems like Kafka.
- Agent Interaction: Agents can proactively watch channels (e.g., Slack) for engineering questions. If they lack confidence in an answer, they can DM the user to confirm before responding publicly.

## Practical implications

- Implement background agents to monitor non-alerted changes (e.g., feature flag rollouts, infra updates) that fall outside standard CI/CD coverage.
- Focus on building knowledge systems that understand the causal relationships between services and telemetry data, rather than just monitoring individual metrics.
- Use agent capabilities for operational handoffs or recurring health checks to formalize 'invisible' engineering work and reduce cognitive load.

## Topics

AI Agents, Observability, DevOps, On-Call Management, Production Monitoring, Build Automation, Resolve AI, Justin Smith

Source: https://www.youtube.com/watch?v=vSx5IULvBns