# Build with Perception Agents — Emile Baizel & Shruti Arora, Amazon AGI Lab

## Executive summary

The presentation introduces 'Perception Agents,' a paradigm shift in automation that allows AI agents to understand and interact with visual screen environments (like web pages) rather than relying solely on APIs. The core workflow involves two primitives: **Visual Annotation** (allowing users to deterministically select elements and describe desired changes via JSON output) and **Visual Verification** (checking a rendered site against design specifications or user flows defined in Gherkin files). This system aims to close the development feedback loop, enabling agents to automatically apply design changes and run comprehensive verification reports, significantly accelerating the collaboration between design teams and engineers.

## Key takeaways

- Perception Agents for Visual Workflow Automation: Agents are moving beyond simple API calls to understand screen context, mimicking human visual understanding. This is crucial for automating knowledge work across sites lacking dedicated APIs.
- Deterministic Feedback Loop: The process is structured: Visual Annotation captures feedback (purely deterministic JSON output) $\rightarrow$ Coding Agent applies changes $\rightarrow$ Visual Verification confirms adherence to design specs (Gherkin files) and user flows.
- Amazon Nova Act and AWS Integration: Nova Act is a first-class AWS service enabling reliable, large-scale browser-based workflows (e.g., form filling, search/extract) using natural language.

## Technical details

- Visual Annotation Primitive: Allows users to point, click, draw, or drop a pin on any webpage to select elements and describe changes. The output is a deterministic JSON file and a prompt for the coding agent. This process is explicitly stated to be independent of model inference.
- Visual Verification Primitive: A skill that runs against design artifacts (like CSS styles) and user flows defined in Gherkin syntax. It ensures the rendered site adheres to specified design expectations and user journeys.
- Agent Integration and Workflow: The full loop requires an 'Agent Bridge' to automatically take the structured annotations and prompt the coding agent. The workshop demonstrated wiring this up by modifying files like `manifest.json` and `content.js` in the Chrome extension.
- Backend Tools and Requirements: The system requires an AI coding agent (e.g., Claude, Kiro), a local bridge, and specific prerequisites like Node.js, NPM, Python, and a Chrome browser for testing.

## Practical implications

- Accelerates the development cycle by automating the incorporation of design feedback directly into the codebase.
- Provides a robust, visual QA layer that verifies site adherence to design specs and user flows, reducing manual testing effort.
- Enables non-technical stakeholders (designers, product managers) to provide actionable, structured feedback to development agents.
- The concept can be extended to capture feedback from conversational sources, such as the B device demonstration.

## Topics

AI Agents, Web Development, Workflow Automation, Continuous Integration (CI), UX/UI Design Systems, Amazon AGI Lab, Amazon Nova Act, Visual Annotator, Visual Verification Skill

Source: https://www.youtube.com/watch?v=ZGzthw4BZiA