Build with Perception Agents — Emile Baizel & Shruti Arora, Amazon AGI Lab
Summary
The presentation introduces 'Perception Agents,' a paradigm shift in automation that allows AI agents to understand and interact with visual screen environments (like web pages) rather than relying solely on APIs. The core workflow involves two primitives: **Visual Annotation** (allowing users to deterministically select elements and describe desired changes via JSON output) and **Visual Verification** (checking a rendered site against design specifications or user flows defined in Gherkin files). This system aims to close the development feedback loop, enabling agents to automatically apply design changes and run comprehensive verification reports, significantly accelerating the collaboration between design teams and engineers.
Key takeaways
-
Perception Agents for Visual Workflow Automation
20:00
Agents are moving beyond simple API calls to understand screen context, mimicking human visual understanding. This is crucial for automating knowledge work across sites lacking dedicated APIs.
-
Deterministic Feedback Loop
12:23
The process is structured: Visual Annotation captures feedback (purely deterministic JSON output) $\rightarrow$ Coding Agent applies changes $\rightarrow$ Visual Verification confirms adherence to design specs (Gherkin files) and user flows.
-
Amazon Nova Act and AWS Integration
16:40
Nova Act is a first-class AWS service enabling reliable, large-scale browser-based workflows (e.g., form filling, search/extract) using natural language.
Technical details
-
Visual Annotation Primitive
743s
Allows users to point, click, draw, or drop a pin on any webpage to select elements and describe changes. The output is a deterministic JSON file and a prompt for the coding agent. This process is explicitly stated to be independent of model inference.
-
Visual Verification Primitive
825s
A skill that runs against design artifacts (like CSS styles) and user flows defined in Gherkin syntax. It ensures the rendered site adheres to specified design expectations and user journeys.
-
Agent Integration and Workflow
1400s
The full loop requires an 'Agent Bridge' to automatically take the structured annotations and prompt the coding agent. The workshop demonstrated wiring this up by modifying files like `manifest.json` and `content.js` in the Chrome extension.
-
Backend Tools and Requirements
1400s
The system requires an AI coding agent (e.g., Claude, Kiro), a local bridge, and specific prerequisites like Node.js, NPM, Python, and a Chrome browser for testing.
Mentioned resources
- Amazon AGI Lab
- Amazon Nova Act
- Visual Annotator
- Visual Verification Skill
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.