How Bridgewater Turned 100 Million Documents Into an Edge - Sam Green & Aaron Linsky, Bridgewater
Summary
Bridgewater Associates built 'The Vault,' a data platform designed to transform over 100 million unstructured documents from 135 external sources and internal firm data into a structured, first-class asset for investment research. The system addresses the distinct needs of human analysts (requiring high precision and signal-to-noise ratio) and AI agents (requiring massive scale and recall). Key technical advancements include implementing advanced filtering techniques (like Turbo Puffer's native filtering) and developing proactive agent workflows, such as the Investor Workbench, which automatically captures synthesized learning and compounds knowledge back into the corpus.
Key takeaways
-
Human and Agent Data Needs Diverge
Humans require high precision and a low signal-to-noise ratio because they have limited reading capacity (approx. 300 words/minute). Agents, conversely, are less sensitive to precision, scale rapidly, and consume data much faster (approx. 30,000 words/minute). A successful platform must be built to serve both sets of needs separately.
-
The Greatest Impact is Unified Tooling
The most significant gains are achieved when a single tool can simultaneously meet the requirements of both human users and autonomous agents, rather than requiring separate systems.
-
The Goal is Proactive Agent Participation
The ultimate objective is moving beyond reactive tools (where humans query the system) to building foundations that enable agents to become proactive participants, guiding high-value professionals toward areas needing attention.
Technical details
-
Data Corpus and Governance
0s
The Vault indexes over 100 million documents from 135 external sources and internal memos. Crucially, all data is aligned to an internal ontology, enriched with metadata like countries, concepts, markets, and the specific time period discussed, ensuring systematic study over time.
-
Search Performance Benchmarking
819s
Initial search performance was poor (Precision at 10 was around 60%). After implementing 'Agentic Search,' the precision at 10 improved significantly to 90-95%, measured by hand-benchmarking 200 examples.
-
Advanced Filtering Techniques
The platform addresses filtering performance by comparing three methods: Post-filtering (fast, but low recall), Pre-filtering (high recall, but slow), and Turbo Puffer's native filtering. The latter provides fast performance similar to post-filtering while maintaining recall close to pre-filtering.
-
Scaling Requirements
The system must scale beyond human usage (e.g., 100 queries/hour) to handle agent swarms, which can easily exceed 100 queries per second.
Mentioned resources
- The Vault
- Turbo Puffer
- Pocket Analyst
- Investor Workbench
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.