When Does Divide and Conquer Work for Long Context LLM? A Noise Decomposition Framework

TL;DR

This paper introduces a theoretical noise decomposition framework to analyze when divide and conquer strategies are effective for long context tasks in Large Language Models, supported by experiments on various NLP tasks.

Contribution

It presents a novel noise decomposition framework that explains the failure modes of long context processing and guides effective chunking strategies for LLMs.

Findings

Multi-agent chunking is effective under certain noise conditions.

Chunk-based processing can outperform single-shot models like GPT-4 on large inputs.

The framework clarifies when dividing long texts improves LLM performance.

Abstract

We investigate the challenge of applying Large Language Models (LLMs) to long texts. We propose a theoretical framework that distinguishes the failure modes of long context tasks into three categories: cross-chunk dependence (task noise), confusion that grows with context size (model noise), and the imperfect integration of partial results (aggregator noise). Under this view, we analyze when it is effective to use multi-agent chunking, i.e., dividing a lengthy sequence into smaller chunks and aggregating the processed results of each chunk. Our experiments on tasks such as retrieval, question answering, and summarization confirm both the theoretical analysis and the conditions that favor multi-agent chunking. By exploring the accelerated decay of model fidelity with input length, we also explain why, for large inputs, a weaker model configured with chunk-based processing can surpass a more advanced model like GPT4o applied in a single shot. Overall, we present a principled understanding framework and our results highlight a direct pathway to handling long contexts in LLMs with carefully managed chunking and aggregator strategies.