The Video Signal

Topic

System Reliability

All digests tagged System Reliability

I Monitored Crime Audio. Voice Agents Scare Me More. — Sumanyu Sharma, Hamming AI thumbnail

· 16:04

I Monitored Crime Audio. Voice Agents Scare Me More. — Sumanyu Sharma, Hamming AI

The presentation compares the monitoring of decentralized, hyper-local crime data (Hamming's initial work) with the rapidly scaling, centralized risks of conversational voice agents. While voice AI is advancing rapidly, reliability remains the primary blocker for large-scale deployment. The speaker emphasizes that because voice agents are centralized, a single prompt or architectural change can have a massive 'blast radius.' He advocates for a continuous monitoring loop—including deep manual analysis, frequency/severity prioritization, and adversarial red teaming—to mitigate risks like unauthorized actions, incorrect information provision, and the leakage of PHI/PII.

Key takeaways

  1. Voice Agents vs. Crime Monitoring 7:12

    Crime incidents are generally hyper-local and decreasing, while voice agent usage is centralized and rapidly increasing, potentially handling a trillion calls annually. This centralization means a single failure point can impact millions of users.

  2. The Scale of Risk 8:43

    If a 1% error rate is assumed across annual calls, this equates to 10 billion potential bad interactions. In practice, monitoring 10,000 agents shows an error rate closer to 10%, manifesting as skipping eligibility checks or providing incorrect information.

  3. The Continuous Improvement Loop 11:44

    Fixing voice agent reliability requires a structured loop: Identify problems, prioritize by frequency and severity, understand the fix, execute the change, verify it hasn't caused regressions, and continue monitoring in production.

Watch on YouTube Full article

GPT-6 Astra Doesn't Need Your Instructions Anymore. thumbnail

· 27:35

GPT-6 Astra Doesn't Need Your Instructions Anymore.

The emergence of super agents like GPT-6 Astra signals a shift from task-based AI prompting to autonomous, self-directed systems. These agents can operate without explicit instructions, building environments and solving complex problems by reasoning across diverse software tools (e.g., browsers, spreadsheets). For build engineers, this means moving away from defining discrete tasks toward managing continuous 'areas of concern' or standing jobs that require long-term persistence and cross-system coordination.

Key takeaways

  1. AGI is defined by autonomy, not benchmarks

    The key shift is the ability to operate without needing a specific method or recipe. Astra's capability—picking its own approach and building necessary tooling—is presented as evidence that we are past the need for explicit instructions.

  2. Super agents handle persistent, long-running jobs 17:18

    Agents can be entrusted with ongoing areas of concern (e.g., 'Keep me aware of things I'm likely to miss') rather than single tasks. This requires remembering past events and maintaining long-term intent without constant human prompting.

  3. The bottleneck is reliability, not intelligence 25:19

    As agents become more capable, the critical challenge shifts from raw intelligence to trustworthiness. The goal is achieving a level of reliability (the last 1-2 percent) that allows for full operational trust in enterprise settings.

  4. Management evolves from coordination to value driving

    Managers will shift from assigning tasks and checking progress (coordination) to defining what matters, identifying trade-offs, and owning the overall outcome of a team of super agents and humans.

Watch on YouTube Full article

Tribal Dungeons of Global Shipping: AI Agents at Global Scale — Dmitry Buykin, Maersk thumbnail

· 12:02

Tribal Dungeons of Global Shipping: AI Agents at Global Scale — Dmitry Buykin, Maersk

This report details the engineering challenges of operationalizing complex, global business processes using AI agents. The core challenge is translating 'tribal knowledge'—which exists as unstructured Standard Operating Procedures (SOPs) or screenshots—into executable agent workflows. The speaker emphasizes that the system is not the agent loop itself, but rather the surrounding 'refining loop.' This loop involves building robust mechanisms for validation, error handling, and systematic improvement, which requires capturing a vast SOP corpus (estimated 20:1 ratio to runtime).

Key takeaways

  1. The Long Tail Problem

    While happy paths are often automated, the most expensive part of global operations is the 'long tail'—the exceptions and variations that require expert orchestration across multiple incomplete systems.

  2. Agent vs. Expert Knowledge

    SOPs explain what a person sees and clicks (screenshots), but an agent SOP requires complex setup: preconditions, decisions, identifiers, backend calls, validation, recovery, and evidence of successful execution. Experts own the 'what,' and agents own the 'how.'

  3. The Refining Loop is the System 3:34

    The true system is not the agent loop but the surrounding refining loop. This process requires making corrections executable changes, establishing guardrails to prevent unsafe paths, and ensuring behavior is observable.

  4. Scaling Improvement

    Accuracy was not designed upfront; it was earned through systematic effort. The team logged over 100,000 corrections over nine months, using heat maps to prioritize the most beneficial work for experts and engineers.

  5. The Goal of AI Native Operation

    AI native operation is not just about agents in a workflow; it's an adaptive architecture that learns from successful scenarios, aggregates them into composite tools, and allows repeatable rollouts across hundreds of countries.

Watch on YouTube Full article

Bringing agents onto the world wide web — Paul Klein IV, Browserbase thumbnail

· 18:26

Bringing agents onto the world wide web — Paul Klein IV, Browserbase

The talk argues that the current bottleneck for building reliable web agents is not model capability (AI), but rather the missing engineering infrastructure—the 'harness' and scalable systems. Reliable browser agents must be multimodal, harness-engineered with memory/skills, and operate on consistent, robust infrastructure. The speaker emphasizes that solving this overhang is an engineering problem that companies can tackle today, accelerating AI diffusion into real-world enterprise use cases (e.g., logistics, banking).

Key takeaways

  1. The bottleneck is the harness, not the model. 7:12

    Model capabilities have advanced significantly; therefore, agents struggle due to missing scaffolding and systems around the LLM (the 'harness'). Building a custom harness optimized for a domain can improve results beyond baseline models. (4:32)

  2. Reliable browser agents require three components. 11:45

    They must be multimodal, harness-engineered (using skills and memory), and run on reliable infrastructure that guarantees consistent page rendering across all runs. (7:05)

  3. The web needs to adapt for agents. 17:04

    Key improvements required from the web include better accessibility trees, adoption of Chrome's Web MCP, solving agent login/authentication securely, and establishing a trust framework (a 'Verisign moment') for agents. (10:24)

  4. Infrastructure must be scalable and consistent. 15:09

    Production environments require infrastructure that can handle thousands of agents, maintain consistency (e.g., preventing mobile/desktop layout shifts), and ideally achieve SOC 2 compliance at scale. (9:09)

Watch on YouTube Full article

Why Agentic Systems Need Ontologies — Frank Coyle, UC Berkeley thumbnail

· 21:18

Why Agentic Systems Need Ontologies — Frank Coyle, UC Berkeley

The talk argues that while Large Language Models (LLMs) and agentic systems represent a powerful shift toward AI, their inherent probabilistic nature makes them unreliable for mission-critical tasks. To mitigate failures—such as incorrect order statuses or double refunds—the solution is 'Neurosymbolic AI': coupling the LLM's generative power with external, formal ontologies. An ontology acts as a logical guardrail, enforcing constraints (e.g., payment status must be one of three values) that are impossible to reliably enforce using prompt engineering alone.

Key takeaways

  1. The Problem with Probabilistic Agents

    LLMs reason probabilistically over domains they only half understand, leading to failures in brittle tools and fragile handoffs. These errors cannot be reliably stopped by mere instructions or prompt engineering.

  2. Neurosymbolic AI: Guardrails for LLMs 7:04

    This approach ties together neural networks (LLMs) with symbolic AI (rule-based systems and ontologies). The ontology provides the necessary guardrails to keep the probabilistic model constrained and honest.

  3. The Role of Ontologies 8:43

    An ontology is a formal specification of a shared conceptualization, defining typed entities, their relationships, and constraints. It allows systems to validate proposed actions (e.g., ensuring an order can only be refunded once).

Watch on YouTube Full article

Learned Execution Graphs for Anomaly Detection & Drift in APIs — Ritvik Pandya, JP Morgan Chase thumbnail

· 19:38

Learned Execution Graphs for Anomaly Detection & Drift in APIs — Ritvik Pandya, JP Morgan Chase

This talk details using short-lived execution graphs (DAGs) to monitor API request processing for advanced anomaly and drift detection. Instead of relying on traditional service level monitoring (SLM) that reports 'healthy' averages, this method models the entire request flow as a Directed Acyclic Graph (DAG). By comparing the actual execution graph against a learned baseline, it can pinpoint performance issues or skipped steps to specific nodes, drastically reducing root cause analysis time. The system differentiates between transient anomalies and persistent drift—categorizing drift into structural, volume, and covariate types—to ensure accurate alerting and proactive system re-baselining.

Key takeaways

  1. Execution Graphs for Monitoring

    Representing API request processing as a DAG allows holistic tracking of middleware steps, context passing, and execution order. This provides granular visibility beyond simple endpoint latency checks.

  2. Anomaly vs. Drift Detection 12:13

    The system distinguishes between sudden anomalies (e.g., traffic spikes) and slow drift—a gradual change in performance or pattern over time—which requires updating the baseline itself.

  3. Three Types of Drift 14:15

    Drift is categorized into structural (new/removed nodes), volume (expected request load changes), and covariate (changes in input data distribution, e.g., different currency types).

Watch on YouTube Full article