# Long-Horizon Agents Need Experiments, Not Just Prompts — Erina Karati

## Executive summary

This talk outlines the challenges of building multi-agent AI systems that maintain state and social consistency over long time horizons. The speaker introduces Project Paradox, a modular, stateful framework for game agents. The core problem—the degradation of social consistency (e.g., losing the source of a rumor)—is addressed by implementing an 'Auto Research' layer. This layer functions as an experimental loop, running controlled scenarios, collecting structured traces, and proposing constrained policy changes only if a balanced scorecard improves, thereby moving agent improvement from simple prompt tuning to systematic, verifiable engineering.

## Key takeaways

- Long-Horizon Agents Need Experiments, Not Just Prompts: Improving multi-agent behavior requires running controlled scenarios and collecting structured traces, rather than relying solely on prompt engineering or single-run demos. The goal is to evaluate entire runs, not just single answers.
- Provenance and Source Attribution are Critical: Simple Retrieval-Augmented Generation (RAG) memory is insufficient. Agents must track where information came from (firsthand, secondhand, verified, uncertain) to maintain social consistency.
- Optimization Requires a Balanced Scorecard: Instead of optimizing for a single metric (e.g., 'agent quality'), a balanced scorecard must measure multiple dimensions like reach, source retention, uncertainty preservation, action consistency, and privacy containment.
- The Auto Research Loop: The proposed solution is a meta-system that defines a controlled scenario, runs the agents, scores the behavior, and proposes small, constrained changes to the agent protocol (the 'editable surface').

## Technical details

- Project Paradox Architecture: The framework is intentionally stateful, featuring per-agent memory (backed by RAG), emotion tracking (small vector), a trust matrix (belief scores), and an importance score for memories to prioritize critical context.
- Auto Research Layer: This layer operates outside the agent village, reading full traces of a run. It compares outcomes against scenario ground truth, scores the behavior, and proposes constrained changes to the agent protocol or cognitive policy. It ensures changes only improve the score and maintain guardrails.
- Controlled Scenarios: To evaluate social behavior, scenarios must be controlled (e.g., testing public fact diffusion, rumor uncertainty, or replanning due to blocked routes). This allows for objective measurement of system improvement.
- Policy Surface Control: The auto research layer must only edit a small, frozen policy surface (e.g., memory writing policy, communication prompt, belief rules) to prevent 'gaming' the evaluation and ensure controlled improvement.

## Practical implications

- The principles apply beyond games to complex stateful systems, including support agents (tracking policy updates provenance), personal assistants (managing commitments and corrections), research agents (requiring provenance citations), coding agents (maintaining long-running context), and workflow agents (handling access controls and replanning).

## Topics

Multi-Agent Systems, AI Architecture, State Management, Retrieval-Augmented Generation (RAG), Policy Optimization, Behavioral Scoring, Project Paradox, Supercell AI Innovation Lab, Microsoft

Source: https://www.youtube.com/watch?v=x4e5O9zN0TE