A New HOPE: Domain-agnostic Automatic Evaluation of Text Chunking

TL;DR

This paper introduces HOPE, a domain-agnostic automatic evaluation metric for document chunking that correlates with RAG performance, revealing the importance of semantic independence over traditional concept unity assumptions.

Contribution

The paper presents a novel, holistic evaluation framework for chunking that considers intrinsic, extrinsic, and coherence properties, advancing analysis of chunking effects on RAG systems.

Findings

HOPE correlates significantly with RAG performance indicators.

Semantic independence of passages greatly improves factual correctness.

Traditional concept unity assumptions have minimal impact on system performance.

Abstract

Document chunking fundamentally impacts Retrieval-Augmented Generation (RAG) by determining how source materials are segmented before indexing. Despite evidence that Large Language Models (LLMs) are sensitive to the layout and structure of retrieved data, there is currently no framework to analyze the impact of different chunking methods. In this paper, we introduce a novel methodology that defines essential characteristics of the chunking process at three levels: intrinsic passage properties, extrinsic passage properties, and passages-document coherence. We propose HOPE (Holistic Passage Evaluation), a domain-agnostic, automatic evaluation metric that quantifies and aggregates these characteristics. Our empirical evaluations across seven domains demonstrate that the HOPE metric correlates significantly (p > 0.13) with various RAG performance indicators, revealing contrasts between the importance of extrinsic and intrinsic properties of passages. Semantic independence between passages proves essential for system performance with a performance gain of up to 56.2% in factual correctness and 21.1% in answer correctness. On the contrary, traditional assumptions about maintaining concept unity within passages show minimal impact. These findings provide actionable insights for optimizing chunking strategies, thus improving RAG system design to produce more factually correct responses.