Optimizing AI Retrieval: Choosing the Best Chunking Strategy

Iris Coleman
Jun 18, 2025 17:01

Explore the best chunking strategies for AI systems to enhance retrieval accuracy. Discover insights from NVIDIA’s experiments on page-level, section-level, and token-based chunking.

In the realm of artificial intelligence, particularly in retrieval-augmented generation (RAG) systems, the method of breaking down large documents into smaller, manageable pieces—known as chunking—is crucial. According to a blog post by NVIDIA, poor chunking can lead to irrelevant results and inefficiency, thus impacting the business value and efficacy of AI responses.

The Importance of Chunking

Chunking plays a vital role in preprocessing for RAG pipelines, as it involves dividing documents into smaller pieces that can be efficiently indexed and retrieved. A well-implemented chunking strategy can significantly enhance the precision of retrieval and the coherence of contextual information, which are essential for generating accurate AI responses. For businesses, this can mean improved user satisfaction and reduced operational costs due to efficient resource utilization.

Experimentation with Chunking Strategies

NVIDIA’s research evaluated various chunking strategies, including token-based, page-level, and section-level chunking, across multiple datasets. The aim was to establish guidelines for selecting the most effective approach based on specific content and use cases. The experiments involved datasets such as DigitalCorpora767, FinanceBench, and others, with a focus on retrieval quality and response accuracy.

Findings from the Experiments

The experiments revealed that page-level chunking generally provided the highest average accuracy and the most consistent performance across different datasets. Token-based chunking, while also effective, showed varying results depending on chunk size and overlap. Section-level chunking, which uses document structure as a natural boundary, performed well but was often outperformed by page-level chunking.

Guidelines for Chunking Strategy Selection

Based on the findings, the following recommendations were made:

• Page-level chunking is suggested as the default strategy due to its consistent performance.
• For financial documents, consider token sizes of 512 or 1,024 for potential improvements.
• The nature of queries should guide chunk size selection; factoid queries benefit from smaller chunks, while complex queries may require larger chunks or page-level chunking.

Conclusion

The study underscores the importance of selecting an appropriate chunking strategy to optimize AI retrieval systems. While page-level chunking emerges as a robust default, the specific needs of the data and queries should guide final decisions. Testing with actual data is crucial to achieving optimal performance.

For more detailed insights, you can read the full blog post on NVIDIA’s blog.

Image source: Shutterstock