# The Autonomous Computer: Infrastructure for Computer Use Agents — Ang Li, Simular

## Executive summary

The presentation outlines the evolution and infrastructure requirements for autonomous computer use agents. The core argument is that achieving reliable, repeatable, and general-purpose agent behavior requires a 'neuro-symbolic' approach, combining the adaptability of Vision Language Models (VLM) with the determinism of symbolic code (coding agents). The infrastructure discussion moves from local desktop agents to scalable cloud platforms, addressing challenges like statefulness, environment variability, and the need for robust APIs for enterprise integration.

## Key takeaways

- Neuro-Symbolic AI is the Future of Reliability: Autonomous agents must combine the adaptability of neural networks (VLM) with the deterministic reliability of symbolic code (coding agents). This fusion addresses the 'stability and plasticity dilemma'—the need for systems to be both reliable (stable) and adaptable (plastic).
- The Shift to Cloud Infrastructure: The focus is moving from agents operating on a single personal desktop to fleets of cloud computers (e.g., using 'SideBox'), allowing for scalable, multi-machine automation and enterprise deployment.
- Reliability Requires Multiple Modalities: No single input modality is sufficient for reliable agent operation. The system must employ a 'whimo approach' (like self-driving cars), utilizing multiple signals such as screenshots, accessibility trees, and DOM trees to ensure robust understanding of the environment.
- The Goal is Deterministic, Repeatable Workflows: The ultimate goal is to achieve zero token consumption for repetitive tasks by generating code (using the coding agent) that can deterministically replay successful workflows, minimizing reliance on expensive, probabilistic LLM calls.

## Technical details

- Agent Architecture: The proposed architecture is 'neuro-symbolic,' consisting of a Vision Language Model (VLM) component (for adaptability) and a Coding Agent component (for deterministic, reliable execution).
- Input Modalities: Agents utilize multiple signals: 1) Screenshots (visual data), 2) Accessibility Tree (OS-level structure, preferred for reliability), and 3) DOM Tree (browser structure). The accessibility tree is noted as a critical, unique layer tied to the operating system.
- Agent Frameworks and Benchmarks: The speaker introduced the 'Agent-S' framework, a model-agnostic system. Benchmarks include the OS World benchmark (achieving human performance in December 2025) and the upcoming OS World 2.
- Coding Agent Protocol: The speaker open-sourced 'Simulan,' a language designed to create reliable, deterministic code for computer automation, similar to RPA protocols but intended for general browser automation. This reduces the computer use problem into a coding problem.
- Cloud Infrastructure: The system is designed to run on cloud-managed virtual machines (VMs) via 'SideBox,' providing scalability and API access (SAPI) for enterprise integration across different operating systems (Windows, macOS, Android).

## Practical implications

- Automating legacy systems and traditional industries (e.g., ERP, insurance, healthcare) that lack modern APIs.
- Enabling the creation of highly scalable AI organizations by hosting agents on cloud fleets of virtual machines.
- Developing 'guard rails' and verification loops (e.g., asking the model to verify the filled form) to improve accuracy and reliability in critical processes.
- Using the API (SAPI) to integrate agent capabilities into existing enterprise software and workflows.

## Topics

Autonomous Agents, AI Infrastructure, Neuro-Symbolic AI, Computer Vision, Automation, Cloud Computing, Simular, Agent-S, SideBox

Source: https://www.youtube.com/watch?v=tyFPqtyOK5w