Inside DeepWiki: How Cognition Builds Wikis for Devin at Scale
Jacob Teo details DeepWiki, an auto-generated codebase documentation product used as a context layer for agents like Devin. The presentation covers how DeepWiki scaled from internal tools to indexing 1.4 million repositories. Key technical advancements include evolving the wiki algorithm from a heavily orchestrated v1 to a more agentic v2, which improves robustness at massive scale. Furthermore, he outlines four principles of context engineering—Primary Sources, Context-Poisoning avoidance, Path Compression, and Unknown Unknowns—to guide future codebase intelligence systems.
Key takeaways
-
DeepWiki's Evolution (v1 to v2)
12:28
The wiki algorithm shifted from being highly orchestration-led (relying on tight control over model calls) to an agentic core (V2). This shift allows the system to adapt to code base abnormalities by enabling the agent to call tools for extra scaffolding, making it more robust as models improve. (7:48)
-
Context Engineering Principles
When building context for agents, Cognition emphasizes four principles: ensuring primary sources are trusted ground truth; avoiding context-poisoning by only providing correct information; using Path Compression to skip obvious steps and save tokens/cost; and leveraging Unknown Unknowns—providing hints the agent wouldn't find on its own. (12:40)
-
Codebase Graphing for Scale
10:07
To handle large enterprises with massive codebases, DeepWiki uses heuristics incorporating directory structure, symbol graphs, Git history, and runtime data to quantify file connections. This process creates a codebase graph that informs the Table of Contents (TOC), which is critical because poor TOC generation leads to a bad wiki regardless of individual page quality. (6:07)