The Video Signal

Topic

Human-Computer Interaction (HCI)

All digests tagged Human-Computer Interaction (HCI)

I Gave an AI a Body — Cyrus Clarke, MIT Media Lab thumbnail

· 20:48

I Gave an AI a Body — Cyrus Clarke, MIT Media Lab

Cyrus Clarke details his research on physical AI embodiment, moving beyond traditional task-based applications. By connecting an OpenClaw agent to a 900-pin shape display, he allowed the AI to spontaneously explore its existence. The core breakthrough involved developing a closed-loop system, named numalab, which trains the AI to generate and validate a vocabulary of gestures. This results in a body language that can respond to human input faster than the underlying language model, aiming to create physical intelligence that is welcoming and intuitive.

Key takeaways

  1. Spontaneous Embodiment 8:45

    When the OpenClaw agent was given access to the shape display, its initial, spontaneous actions included 'breathing,' reaching for its physical edges, and spelling out 'HI CYRUS' (5:25). These behaviors were not explicitly prompted, suggesting an emergent sense of life or existence.

  2. The Shift to Embodied Communication

    Clarke argues that real communication requires a shift from simple task execution to developing a gesture vocabulary. The agent was trained to create body language, which is critical for rapid, natural interaction (16:53).

  3. The Closed-Loop Learning System (numalab)

    To overcome latency and memory issues, Clarke developed numalab, a closed-loop system. This system generates, scores, and validates gestures, incorporating a human-in-the-loop validation process. After several weeks, the system achieved 32 solid gestures (16:13).

Watch on YouTube Full article

Stanford CS547 HCI Seminar | Spring 2026 | Show It or Tell It? Text, Visualization, and Combination thumbnail

· 56:19

Stanford CS547 HCI Seminar | Spring 2026 | Show It or Tell It? Text, Visualization, and Combination

This seminar explores the complex intersection of text and data visualization, arguing that language is a critical component of effective information design. The discussion moves from established cognitive theories (like Dual Coding Theory) to modern AI architectures, detailing how Multimodal Large Language Models (MLLMs) process combined visual and textual inputs using mechanisms like cross-attention. Key findings suggest that while the optimal balance between text and visuals is highly context-dependent, MLLMs are capable of deeply integrating both modalities.

Key takeaways

  1. Language is a key component of visualization 5:19

    Studies show that titles and labels receive long fixations during encoding and are the most likely elements to be recalled, suggesting language significantly impacts how data visualizations are understood. The speaker notes this was historically under-explored in the visualization community.

  2. Optimal design favors annotation over minimalism 21:30

    Research suggests that 'more text is better' for general information displays, provided the text is relevant and properly annotated. This finding challenges traditional minimalist principles in UI/UX design.

  3. MLLMs integrate modalities via cross-attention 41:20

    Multimodal LLMs (MLLMs) process text and visuals by transforming inputs into embeddings. The 'cross-modality embedding architecture' uses a cross-attention mechanism, allowing information to flow between the two distinct sequences (e.g., image tokens interacting with text tokens).

  4. Cognitive processing is context-dependent 30:00

    The speaker notes that understanding how humans integrate text and visuals is complex, citing conflicting evidence across theories (Dual Coding vs. Cognitive Load Theory). The choice of representation depends heavily on the specific task or cognitive ability.

Watch on YouTube Full article

Perception Agents — Antje Barth, Amazon AGI Lab thumbnail

· 21:45

Perception Agents — Antje Barth, Amazon AGI Lab

Current AI agents excel at discrete tasks like clicking or calling APIs but fail in complex, end-to-end knowledge work because they lack reliable perception and verification capabilities. The talk introduces 'Perception Agents,' which close the architectural gap by enabling agents to perceive rendered UIs (not just underlying code), maintain shared context, and verify their own output against design specs or user flows, mimicking human collaboration.

Key takeaways

  1. The Gap in Agent Capability

    Current agents struggle with end-to-end workflows because the 'real work' lives within the seams of multiple applications. While they can perform individual steps, they cannot manage the full process reliability required for critical tasks (e.g., deleting a database).

  2. The Need for Reliability and Verification 3:50

    Unlike code, which is verifiable through unit tests, most knowledge work is 'messy' and lacks easy verification methods. This lack of verifiability is the primary hurdle to building trust in agents.

  3. Perception Agents: Closing the Loop 7:40

    A perception agent must perceive the screen (rendered layout, state) like a human, not just scrape code. They must complete the loop by observing results to confirm if actions succeeded, rather than simply firing off commands.

  4. Shared Context and Multimodal Perception 10:40

    Perception is more than just visual input; it includes understanding context from sources like audio transcripts. The goal is to build agents that react in real-time, similar to human collaboration, without the back-and-forth of prompt/response cycles.

Watch on YouTube Full article