# Perception Agents — Antje Barth, Amazon AGI Lab

## Executive summary

Current AI agents excel at discrete tasks like clicking or calling APIs but fail in complex, end-to-end knowledge work because they lack reliable perception and verification capabilities. The talk introduces 'Perception Agents,' which close the architectural gap by enabling agents to perceive rendered UIs (not just underlying code), maintain shared context, and verify their own output against design specs or user flows, mimicking human collaboration.

## Key takeaways

- The Gap in Agent Capability: Current agents struggle with end-to-end workflows because the 'real work' lives within the seams of multiple applications. While they can perform individual steps, they cannot manage the full process reliability required for critical tasks (e.g., deleting a database).
- The Need for Reliability and Verification: Unlike code, which is verifiable through unit tests, most knowledge work is 'messy' and lacks easy verification methods. This lack of verifiability is the primary hurdle to building trust in agents.
- Perception Agents: Closing the Loop: A perception agent must perceive the screen (rendered layout, state) like a human, not just scrape code. They must complete the loop by observing results to confirm if actions succeeded, rather than simply firing off commands.
- Shared Context and Multimodal Perception: Perception is more than just visual input; it includes understanding context from sources like audio transcripts. The goal is to build agents that react in real-time, similar to human collaboration, without the back-and-forth of prompt/response cycles.

## Technical details

- Perception Agent Architecture: The agent must follow a Perceive -> Plan -> Act loop. Perception involves taking in the rendered screen state, layout, and changes, rather than relying solely on underlying code structure.
- Annotation Tool (Input): The open-source annotation tool allows users to select specific elements on a screen (e.g., selecting a heading) and provide precise, location-based feedback (e.g., 'double the font size'), providing a more accurate signal than long text descriptions.
- Verification Tool (Output Check): The verification component allows agents to check their own work against defined design rules (e.g., colors, components). It performs two types of checks: visual adherence and automated user flow testing.

## Practical implications

- Automation workflows must evolve beyond simple API calls and structured data inputs to handle the unpredictable nature of modern UI interactions.
- Build engineers should anticipate integrating visual verification steps into CI/CD pipelines, ensuring that generated code or deployed features adhere to complex design specifications (e.g., brand guidelines).
- The concept of 'shared context' suggests future automation tools will need multimodal inputs (visual + audio/text) to solve real-world business process gaps.

## Topics

AI Agents, Human-Computer Interaction (HCI), Workflow Automation, UI Perception, Reliability Engineering, Perception Agent Harness, Podcast on Human-Agent Interactions

Source: https://www.youtube.com/watch?v=2JX6JYyQG4Y