Perception Agents — Antje Barth, Amazon AGI Lab
Current AI agents excel at discrete tasks like clicking or calling APIs but fail in complex, end-to-end knowledge work because they lack reliable perception and verification capabilities. The talk introduces 'Perception Agents,' which close the architectural gap by enabling agents to perceive rendered UIs (not just underlying code), maintain shared context, and verify their own output against design specs or user flows, mimicking human collaboration.
Key takeaways
-
The Gap in Agent Capability
Current agents struggle with end-to-end workflows because the 'real work' lives within the seams of multiple applications. While they can perform individual steps, they cannot manage the full process reliability required for critical tasks (e.g., deleting a database).
-
The Need for Reliability and Verification
3:50
Unlike code, which is verifiable through unit tests, most knowledge work is 'messy' and lacks easy verification methods. This lack of verifiability is the primary hurdle to building trust in agents.
-
Perception Agents: Closing the Loop
7:40
A perception agent must perceive the screen (rendered layout, state) like a human, not just scrape code. They must complete the loop by observing results to confirm if actions succeeded, rather than simply firing off commands.
-
Shared Context and Multimodal Perception
10:40
Perception is more than just visual input; it includes understanding context from sources like audio transcripts. The goal is to build agents that react in real-time, similar to human collaboration, without the back-and-forth of prompt/response cycles.