Wearing the Agent: From Group Chats to Glasses — Sai Krishna Rallabandi
Summary
The talk addresses the architectural shift required for agentic systems moving from single-user deployments to complex group chats and wearable devices. The speaker argues that these multi-user environments fundamentally break single-user assumptions, necessitating a complete redesign of memory management and security layers. Key challenges include preventing information leaks when multiple skills collide (security) and efficiently curating evolving conversations over long periods without context bloat (memory).
Key takeaways
-
Group Settings Break Single-User Assumptions
Agents designed for one user fail in group settings because the complexity of tracking multiple participants, maintaining privacy, and managing shared information requires unique architectural solutions.
-
Security Must Be Designed at the Action Surface
12:24
Instead of trying to guard all input (which is impossible due to vast surface areas like web pages, emails, and GitHub issues), security must focus on designing a deterministic guard layer that monitors actions—specifically when the agent reads or exports variables (e.g., bash variables) before taking action.
-
Memory Requires Atomic Information Extraction
Traditional memory methods are insufficient for groups. The system must move beyond storing raw conversation logs and instead extract high-value, atomic facts from the discussion. This requires designing sophisticated auto-raters to determine relevance, hierarchy, and temporal importance.
Technical details
-
Security Layer Design (The Guard)
7s
To prevent information leaks when two benign skills collide at runtime, the speaker proposes a multi-layered defense: 1) A deterministic guard layer monitoring actions (e.g., variable export). 2) Using supervised fine-tuning on an SLM (like Lora) to predict malignant data channels and separate them from instruction channels. This approach can catch attacks that regex or static scans miss, such as text interspersed with dots.
-
Memory Management for Groups
12s
Effective group memory requires identifying and storing 'atomic bits' of information rather than raw conversation logs. Furthermore, the system must incorporate a continuous relevance scorer (trained via ML) to manage context growth, allowing it to intelligently decide what information is important enough to keep and what can be forgotten, thereby saving token costs.
-
Privacy and Contextual Adapters
For shared assistants, privacy is managed by baking permissions into the model itself. This can be achieved by training different Lora adapters for each user on a common shared memory layer, ensuring that information access is governed by machine learning rather than just code.
Mentioned resources
- Judith
- Engram
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.