LumberChunker: Long-Form Narrative Document Segmentation

TL;DR

LumberChunker is a novel document segmentation method using LLMs to improve retrieval by dynamically identifying content shifts in long narratives, demonstrated on a new benchmark with significant performance gains.

Contribution

The paper introduces LumberChunker, an LLM-based dynamic segmentation technique for long documents, and presents GutenQA, a new benchmark for evaluating retrieval in narrative texts.

Findings

LumberChunker outperforms baseline methods by 7.37% in retrieval performance.

It is more effective than other chunking methods in RAG pipelines.

The approach demonstrates significant improvements on the GutenQA benchmark.

Abstract

Modern NLP tasks increasingly rely on dense retrieval methods to access up-to-date and relevant contextual information. We are motivated by the premise that retrieval benefits from segments that can vary in size such that a content's semantic independence is better captured. We propose LumberChunker, a method leveraging an LLM to dynamically segment documents, which iteratively prompts the LLM to identify the point within a group of sequential passages where the content begins to shift. To evaluate our method, we introduce GutenQA, a benchmark with 3000 "needle in a haystack" type of question-answer pairs derived from 100 public domain narrative books available on Project Gutenberg. Our experiments show that LumberChunker not only outperforms the most competitive baseline by 7.37% in retrieval performance (DCG@20) but also that, when integrated into a RAG pipeline, LumberChunker proves to be more effective than other chunking methods and competitive baselines, such as the Gemini 1.5M Pro. Our Code and Data are available at https://github.com/joaodsmarques/LumberChunker