Topic

Distributed Systems

All digests tagged Distributed Systems

MCP Tasks (async): Why Aren't Any Agents Supporting Them? — Cornelia Davis, Temporal thumbnail

· 23:54

MCP Tasks (async): Why Aren't Any Agents Supporting Them? — Cornelia Davis, Temporal

Cornelia Davis discusses MCP Tasks, a specification designed to enable durable, long-running asynchronous interactions for tools and agents that cannot complete in a single request/response cycle. The core challenge is maintaining state and functionality across infrastructure failures (network blips, process crashes) or human delays. While the initial V1 protocol was complex and stateful, the evolution to V2 significantly improves scalability by moving toward a stateless core and structured extensions, making it more viable for large-scale distributed systems.

Key takeaways

  1. The Problem of Long-Running Tasks

    Traditional request/response models fail when work takes time. MCP tasks solve this by allowing an agent to invoke a tool, receive a handle, and interact with that handle asynchronously, surviving disconnections and crashes.

  2. Durability is Paramount 6:43

    For the task to be reliable, it must be durable—meaning its state survives client disconnects, server outages, or human delays. This requirement adds significant complexity.

  3. V2 Moves Toward Statelessness 17:15

    The major improvement in the MCP Tasks V2 specification is its move toward a stateless core and structured extensions, addressing the inherent difficulties of managing stateful protocols in large-scale distributed systems.

Watch on YouTube Full article

Emulated: The Data for Fully Autonomous Software Engineers and Companies — Joseph Wang thumbnail

· 16:33

Emulated: The Data for Fully Autonomous Software Engineers and Companies — Joseph Wang

Emulated focuses on creating high-fidelity training data environments that simulate entire companies and complex infrastructure operations, moving beyond simple code diffs or single-node sandboxes. The core argument is that for AI agents to achieve true autonomy in mission-critical systems (like cloud providers), they must be trained on long-horizon tasks involving distributed cluster failures, resource provisioning across VPCs/subnets, managing cost constraints, and reasoning through real-world operational incidents.

Key takeaways

  1. The Data Gap in AI Agents 3:30

    Current benchmarks (e.g., SweBench Pro, Terminal Bench) limit agents to operating within a codebase, failing to capture the complexity of real-world tasks like PM communication, performance testing, or owning underlying infrastructure over years.

  2. Complexity Requires Full Simulation 6:10

    Real infrastructure work is not a simple code diff; it involves managing failing nodes, stale deprecated components, live traffic serving, and operational blast radius across distributed clusters.

  3. Limitations of Single-Node Sandboxes 10:40

    Standard post-training pipelines often use homogeneous single-node sandboxes. However, real cloud services require simulating resource provisioning (EC2, Cloud Run), VPCs, subnets, and security groups, which necessitates a multi-node sandbox with access to real infrastructure.

Watch on YouTube Full article

Your Agent Didn't Fail. Your Harness Did. — Vinoth Govindarajan, OpenAI thumbnail

· 18:26

Your Agent Didn't Fail. Your Harness Did. — Vinoth Govindarajan, OpenAI

The talk argues that most production failures in AI agents are not due to model hallucination or poor reasoning, but rather 'harness failures'—flaws in the underlying system architecture. The core principle is establishing a clear boundary: while a model can only propose an action, the surrounding harness must own the state transition, enforce ordered commits, and provide verifiable proof (the receipt) that the action was successfully executed and observed by the user.

Key takeaways

  1. Model Proposes, Harness Commits, Receipt Proves 4:06

    The model generates capability; the harness provides control. The system must own state transitions, enforce ordered mutations, and use a receipt to prove that an action was executed and visible to the user.

  2. Own the State, Order the Mutation, Prove the Action 5:20

    These three principles are critical for reliable agent systems. A fact must have a single owner (system of record) and a clear replay path; mutable state requires one ordered commit path.

  3. The Run Receipt Audit

    To audit an agent system, ask five questions: What woke it up (trigger)? What state did it inherit? Which authority did it use? What executed? And what evidence survived?

Watch on YouTube Full article

Learned Execution Graphs for Anomaly Detection & Drift in APIs — Ritvik Pandya, JP Morgan Chase thumbnail

· 19:38

Learned Execution Graphs for Anomaly Detection & Drift in APIs — Ritvik Pandya, JP Morgan Chase

This talk details using short-lived execution graphs (DAGs) to monitor API request processing for advanced anomaly and drift detection. Instead of relying on traditional service level monitoring (SLM) that reports 'healthy' averages, this method models the entire request flow as a Directed Acyclic Graph (DAG). By comparing the actual execution graph against a learned baseline, it can pinpoint performance issues or skipped steps to specific nodes, drastically reducing root cause analysis time. The system differentiates between transient anomalies and persistent drift—categorizing drift into structural, volume, and covariate types—to ensure accurate alerting and proactive system re-baselining.

Key takeaways

  1. Execution Graphs for Monitoring

    Representing API request processing as a DAG allows holistic tracking of middleware steps, context passing, and execution order. This provides granular visibility beyond simple endpoint latency checks.

  2. Anomaly vs. Drift Detection 12:13

    The system distinguishes between sudden anomalies (e.g., traffic spikes) and slow drift—a gradual change in performance or pattern over time—which requires updating the baseline itself.

  3. Three Types of Drift 14:15

    Drift is categorized into structural (new/removed nodes), volume (expected request load changes), and covariate (changes in input data distribution, e.g., different currency types).

Watch on YouTube Full article

Every Harness Will Become A Claw — Sam Bhagwat, Mastra thumbnail

· 15:36

Every Harness Will Become A Claw — Sam Bhagwat, Mastra

The evolution of AI agents is moving from localized 'Harnesses'—tools used for coding and task execution—to persistent, always-on services called 'Claws.' This transition involves imbuing agents with initiative, external connectivity (like a heartbeat), and continual learning capabilities. The speaker proposes Steinberger's law: every harness will expand until it becomes a Claw, driven by the desire for powerful, integrated developer experiences.

Key takeaways

  1. Harnesses are evolving into Claws 1:42

    The next generation of agents moves beyond local execution to become always-on services that listen to external events (e.g., Slack, mobile apps) and maintain a persistent 'heartbeat.'

  2. Agentic Spectrum Advancement 0:49

    Agents are advancing through stages: Agent $ ightarrow$ Harness $ ightarrow$ Claw. Key technical advancements include durability, doggedness, planning mode, and parallel subagents.

  3. Cloud vs. Local Architecture 1:48

    The shift from local harnesses to cloud harnesses provides greater parallelism and resources but necessitates a different distributed system architecture.

Watch on YouTube Full article

Justin Cormack - When Tests Lie: Using Observability to Keep AI Honest - AI Native DevCon June 2026 thumbnail

· 32:04

Justin Cormack - When Tests Lie: Using Observability to Keep AI Honest - AI Native DevCon June 2026

The talk explores the challenges of using AI to build large-scale, complex distributed systems, exemplified by building an AWS S3 compatible object storage system in Rust. While testing is crucial, relying solely on achieving 100% test coverage is insufficient for complex systems. The speaker emphasizes that observability, robust test articles (like external services), and a 'human-in-the-loop' approach are necessary to enforce correctness, discover edge cases, and manage issues like race conditions and flaky tests in AI-assisted development.

Key takeaways

  1. Observability is Critical for Large Systems 17:47

    For complex distributed systems, the public API often doesn't cover all background behaviors. Observability techniques (like tracing) are necessary to infer or observe invisible behaviors that standard APIs cannot expose.

  2. Test Articles Provide Grounding 21:00

    Using an existing system, such as AWS S3, as a 'test article' provides a crucial behavioral baseline. This is more valuable than relying on documentation, which may be inaccurate.

  3. Beyond 100% Test Coverage 13:44

    Achieving 100% test coverage can lead to writing trivial or unhelpful tests. The focus should instead be on expanding the scope of testing and thinking like a QA professional to find edge cases.

  4. Flaky Tests Must Be Fixed 22:00

    The speaker asserts that flaky tests must be fixed immediately, as AI models may incorrectly suggest ignoring them based on training data. Running repeated test suites helps identify these issues.

Watch on YouTube Full article