LLM Knowledge Bases: a practical guide — Ben Holmes, Warp
The talk outlines a comprehensive pipeline for transforming disorganized raw notes into structured, navigable LLM knowledge bases. The process begins with maximizing raw data capture via voice dictation (suggested over typing) and progresses through automated enrichment steps using AI agents. Key stages include enriching notes by adding timestamps, fixed-list tags, web research backlinks, generating wikis of People/Concepts/Sources, and finally visualizing the entire network structure using graph views. The entire system is designed to run on an automated schedule in a cloud sandbox environment.
Key takeaways
-
Raw Data Capture Priority
5:25
The most critical step for building a knowledge base is generating high volume of raw material. Voice dictation (approx. 200 words per minute) is presented as the fastest method for capturing thoughts, even if they are 'scrappy' or rambling.
-
The Enrichment Pipeline
9:53
Notes can be enriched using an agent skill (e.g., 'enrich note') to add structure: timestamps, tags from a fixed reference list (to prevent agents from inventing new categories), web-researched sources, and backlinks found via key term search.
-
Automated Knowledge Generation
13:40
The system can generate a wiki structure (People, Concepts, Organizations) from raw notes. This process is highly automated and can be scheduled to run in the background (e.g., daily), ensuring the knowledge base remains fresh.
-
Automation and Scheduling
17:45
To maintain the knowledge base, the entire flow must run on a schedule using cloud sandboxes (e.g., Oz.dev). The process involves syncing markdown files down, running the agent skill, and syncing the updated notes back up.