The Video Signal

Topic

Observability

All digests tagged Observability

· 20:36

From Signal to PR: Anatomy of a Self-Improving Agent — Jason Lopatecki, Arize

The future of observability is shifting from human-driven dashboards to machine-readable telemetry that powers autonomous AI agents. Arize's Signal automates the debugging process by pulling deep production traces and logs (sometimes ten megabytes) directly into the repository as files. This allows coding harnesses, like Claude Code, to understand the exact code path taken during an error, enabling agents to propose fixes and creating a continuous loop where systems can autonomously improve themselves. The human role is evolving from responder to reviewer.

Key takeaways

  1. Observability Shift (2.0) 2:16

    Observability is moving beyond UI clicks and graphs; it's becoming a 'smoke'—telemetry data that agents can read to debug software, allowing for continuous automated fixing.

  2. The Key Unlock: Traces on Filesystem 6:08

    The critical breakthrough is pulling relevant production traces and logs down as files into the repo. Coding agents are highly effective with file formats, giving them the exact code path rather than guessing among millions of branches.

  3. Autonomous Fixing Loop 6:54

    The goal is to build systems that autonomously fix themselves. The process involves an agent investigating first, gathering deep evidence (traces/logs), and proposing a fix before human intervention.

  4. Security and Deployment

    To ensure compliance for large enterprises (e.g., Uber, Booking), agents must run within the customer's Virtual Private Cloud (VPC) using sandboxes, preventing production systems from connecting directly to external models.

Watch on YouTube Full article

· 41:14

Inside the Agent Engine: A LangChain and Traversal Fireside Chat

The discussion details the challenges and architectural requirements for building AI Site Reliability Engineering (SRE) agents capable of handling petabyte-scale production incidents. Speakers emphasize that SRE troubleshooting is uniquely difficult due to the lack of labeled data, high stakes, and massive telemetry volumes. Successful agent design requires moving beyond simple RAG/vector search by implementing sophisticated 'agent harnesses' that manage context via file systems, build a comprehensive 'production world model,' and strategically balance offline vs. online computation.

Key takeaways

  1. SRE Agents Face Unique Data Challenges 3:23

    Troubleshooting is difficult because there is no good labeled data for LLMs to train on, human troubleshooting processes are complex, and the scale of telemetry (e.g., petabytes per day) makes traditional context window methods infeasible.

  2. Agent Architecture Requires a Core/Sub-Agent Harness 11:30

    Instead of monolithic agents, the recommended approach is building one core agent that orchestrates multiple specialized sub-agents. This requires a robust harness to manage context and file systems effectively.

  3. The Production World Model is Key to System Knowledge 7:50

    Learning system knowledge involves synthesizing large streams of non-telemetry data (e.g., code, Slack) with raw observability logs to build a 'production world model,' which acts as the system's deep wiki.

  4. Evaluation Must Focus on Hardest Tasks 30:50

    When evaluating agents, focus on the hardest tasks (like incident RCA) because success in these complex areas tends to generalize better than focusing on easier, less representative tasks.

Watch on YouTube Full article

· 45:17

The Agent Development Lifecycle 101 by Harrison Chase

The Agent Development Lifecycle outlines a systematic approach for moving AI agents from isolated demos to reliable production systems. The process is broken down into five stages: Build, Test, Deploy, Monitor, and Govern. Key focus areas include ensuring agent reliability at scale by implementing durable execution, managing complex state via virtual file systems, and using advanced observability tools like tracing and online evaluation (evals) to detect failures and drive continuous improvement.

Key takeaways

  1. Systematic Iteration is Key 3:50

    Successful teams treat agents not as one-off projects but as systems requiring systematic iteration across the entire lifecycle: build, test, deploy, monitor, and improve. The primary challenge in shipping agents reliably at scale is ensuring consistent behavior.

  2. Agent Development Components 5:50

    The core components are Build (frameworks/harnesses), Test (data sets, metrics, benchmarks like Terminal Bench 2), Deploy (durable execution, sandboxes), Monitor (tracing, online evals), and Govern (cost control, tool access management).

  3. The Role of Tracing and Observability 17:06

    Tracing is fundamental for debugging agents, allowing developers to see the inputs and outputs at every step (including tool calls) to understand why an LLM or agent failed. Online evals extend this by scoring production traces without needing ground truth.

  4. Self-Improving Agents 31:30

    Advanced platforms, like LangSmith Engine, are beginning to automate the improvement loop. They run in the background over existing traces, clustering issues and suggesting fixes (code or prompt changes), thereby drastically lowering the burden of operating agents at scale.

Watch on YouTube Full article

· 32:04

Justin Cormack - When Tests Lie: Using Observability to Keep AI Honest - AI Native DevCon June 2026

The talk explores the challenges of using AI to build large-scale, complex distributed systems, exemplified by building an AWS S3 compatible object storage system in Rust. While testing is crucial, relying solely on achieving 100% test coverage is insufficient for complex systems. The speaker emphasizes that observability, robust test articles (like external services), and a 'human-in-the-loop' approach are necessary to enforce correctness, discover edge cases, and manage issues like race conditions and flaky tests in AI-assisted development.

Key takeaways

  1. Observability is Critical for Large Systems 17:47

    For complex distributed systems, the public API often doesn't cover all background behaviors. Observability techniques (like tracing) are necessary to infer or observe invisible behaviors that standard APIs cannot expose.

  2. Test Articles Provide Grounding 21:00

    Using an existing system, such as AWS S3, as a 'test article' provides a crucial behavioral baseline. This is more valuable than relying on documentation, which may be inaccurate.

  3. Beyond 100% Test Coverage 13:44

    Achieving 100% test coverage can lead to writing trivial or unhelpful tests. The focus should instead be on expanding the scope of testing and thinking like a QA professional to find edge cases.

  4. Flaky Tests Must Be Fixed 22:00

    The speaker asserts that flaky tests must be fixed immediately, as AI models may incorrectly suggest ignoring them based on training data. Running repeated test suites helps identify these issues.

Watch on YouTube Full article

· 46:40

Logs Are All You Need: Rethinking Observability with AI Agents

Sherwood Callaway introduces Sazabi, an AI-native observability platform designed to disrupt traditional tools like Datadog by focusing on logs as the primary source of truth. The core philosophy is that in the age of coding agents, engineers should interact with production data via natural language chat rather than complex dashboards. Key technical innovations include using Git for persistent agent memory across multiple threads and implementing a read-only sandbox environment to safely execute investigative tasks.

Key takeaways

  1. Logs are Sufficient: Rethinking Observability 22:20

    The traditional 'three pillars' (metrics, logs, traces) are considered overkill for modern agentic workflows. By focusing solely on logs, instrumentation becomes significantly simpler, requiring only basic logging statements, as the platform can reconstruct metrics and traces from log data.

  2. AI Agents Generate Alerts, They Don't Evaluate Them 28:00

    Instead of using AI to triage noisy alerts (alert fatigue), Sazabi autonomously generates actionable alerts directly from logs and codebase analysis. The agent determines what is meaningful to the user at runtime.

  3. Agent Memory via Git for Shared State 34:05

    Sazabi maintains persistent, shared memory across multiple parallel sub-agents and threads by committing findings (e.g., issue lists, facts) to a dedicated Git branch within the sandbox environment. This allows agents to benefit from collective findings.

  4. Sandbox Isolation and Read-Only Access 30:30

    The platform operates in a read-only system with no public internet access, ensuring security. All actions are routed through an isolated sandbox environment (e.g., using `bash` tools) to prevent data exfiltration or unauthorized changes.

Watch on YouTube Full article