Wearing the Agent: From Group Chats to Glasses — Sai Krishna Rallabandi
The talk addresses the architectural shift required for agentic systems moving from single-user deployments to complex group chats and wearable devices. The speaker argues that these multi-user environments fundamentally break single-user assumptions, necessitating a complete redesign of memory management and security layers. Key challenges include preventing information leaks when multiple skills collide (security) and efficiently curating evolving conversations over long periods without context bloat (memory).
Key takeaways
-
Group Settings Break Single-User Assumptions
Agents designed for one user fail in group settings because the complexity of tracking multiple participants, maintaining privacy, and managing shared information requires unique architectural solutions.
-
Security Must Be Designed at the Action Surface
12:24
Instead of trying to guard all input (which is impossible due to vast surface areas like web pages, emails, and GitHub issues), security must focus on designing a deterministic guard layer that monitors actions—specifically when the agent reads or exports variables (e.g., bash variables) before taking action.
-
Memory Requires Atomic Information Extraction
Traditional memory methods are insufficient for groups. The system must move beyond storing raw conversation logs and instead extract high-value, atomic facts from the discussion. This requires designing sophisticated auto-raters to determine relevance, hierarchy, and temporal importance.
