Build with Perception Agents — Emile Baizel & Shruti Arora, Amazon AGI Lab
The presentation introduces 'Perception Agents,' a paradigm shift in automation that allows AI agents to understand and interact with visual screen environments (like web pages) rather than relying solely on APIs. The core workflow involves two primitives: **Visual Annotation** (allowing users to deterministically select elements and describe desired changes via JSON output) and **Visual Verification** (checking a rendered site against design specifications or user flows defined in Gherkin files). This system aims to close the development feedback loop, enabling agents to automatically apply design changes and run comprehensive verification reports, significantly accelerating the collaboration between design teams and engineers.
Key takeaways
-
Perception Agents for Visual Workflow Automation
20:00
Agents are moving beyond simple API calls to understand screen context, mimicking human visual understanding. This is crucial for automating knowledge work across sites lacking dedicated APIs.
-
Deterministic Feedback Loop
12:23
The process is structured: Visual Annotation captures feedback (purely deterministic JSON output) $\rightarrow$ Coding Agent applies changes $\rightarrow$ Visual Verification confirms adherence to design specs (Gherkin files) and user flows.
-
Amazon Nova Act and AWS Integration
16:40
Nova Act is a first-class AWS service enabling reliable, large-scale browser-based workflows (e.g., form filling, search/extract) using natural language.