Stop Chunking Like It's 2022 — Yuval Belfer, AI21 Labs
This talk challenges the fixed-chunking paradigm in Retrieval-Augmented Generation (RAG) systems. The core claim is that there is no single optimal chunk size; the correct size is query-dependent. The speaker proposes 'multiscale indexing,' which involves duplicating the corpus and chunking it at multiple window sizes. At retrieval time, all resulting indices are queried, and the results are merged using Reciprocal Rank Fusion (RRF), achieving significant recall improvements (20-40%) over fixed-size methods. This approach increases memory usage but maintains low latency overhead.
Key takeaways
-
Chunking is Query-Dependent, Not Fixed
0:07
The optimal chunk size varies drastically based on the query type. For example, a highly focused question (e.g., 'Jerry's favorite church') performs best with smaller chunks, while a broad, narrative question (e.g., 'who does Jerry describe as his nemesis') requires larger context windows to capture the answer spread across a scene. (7:07)
-
Multiscale Indexing is the Solution
0:10
Instead of committing to one chunk size, the system should index the corpus at multiple window sizes (multiscale indexing). At query time, all these indices are queried, and the resulting rankings are combined using Reciprocal Rank Fusion (RRF). (9:59)
-
Significant Performance Gains
0:10
The gap between the performance of the optimal 'oracle' chunk size and any single fixed choice (e.g., 512 tokens) can cost 20% to 40% of recall. The proposed method closes this gap. (9:59)