# Wearing the Agent: From Group Chats to Glasses — Sai Krishna Rallabandi

## Executive summary

The talk addresses the architectural shift required for agentic systems moving from single-user deployments to complex group chats and wearable devices. The speaker argues that these multi-user environments fundamentally break single-user assumptions, necessitating a complete redesign of memory management and security layers. Key challenges include preventing information leaks when multiple skills collide (security) and efficiently curating evolving conversations over long periods without context bloat (memory).

## Key takeaways

- Group Settings Break Single-User Assumptions: Agents designed for one user fail in group settings because the complexity of tracking multiple participants, maintaining privacy, and managing shared information requires unique architectural solutions.
- Security Must Be Designed at the Action Surface: Instead of trying to guard all input (which is impossible due to vast surface areas like web pages, emails, and GitHub issues), security must focus on designing a deterministic guard layer that monitors actions—specifically when the agent reads or exports variables (e.g., bash variables) before taking action.
- Memory Requires Atomic Information Extraction: Traditional memory methods are insufficient for groups. The system must move beyond storing raw conversation logs and instead extract high-value, atomic facts from the discussion. This requires designing sophisticated auto-raters to determine relevance, hierarchy, and temporal importance.

## Technical details

- Security Layer Design (The Guard): To prevent information leaks when two benign skills collide at runtime, the speaker proposes a multi-layered defense: 1) A deterministic guard layer monitoring actions (e.g., variable export). 2) Using supervised fine-tuning on an SLM (like Lora) to predict malignant data channels and separate them from instruction channels. This approach can catch attacks that regex or static scans miss, such as text interspersed with dots.
- Memory Management for Groups: Effective group memory requires identifying and storing 'atomic bits' of information rather than raw conversation logs. Furthermore, the system must incorporate a continuous relevance scorer (trained via ML) to manage context growth, allowing it to intelligently decide what information is important enough to keep and what can be forgotten, thereby saving token costs.
- Privacy and Contextual Adapters: For shared assistants, privacy is managed by baking permissions into the model itself. This can be achieved by training different Lora adapters for each user on a common shared memory layer, ensuring that information access is governed by machine learning rather than just code.

## Practical implications

- Build engineers must design agentic systems with group-chat and wearable deployment scenarios in mind, treating them as fundamentally different from single-user applications.
- Security architecture should prioritize monitoring the 'action surface' (data export, variable reading) rather than attempting to filter all input data.
- Memory components need to evolve from simple embedding storage to complex graph or atomic fact extraction systems that continuously score relevance.

## Topics

Agentic Systems, LLM Security, Distributed Computing, Context Management, AI Architecture, Judith, Engram

Source: https://www.youtube.com/watch?v=s67bE2Ur3bY