SEGMENT+: Long Text Processing with Short-Context Language Models

TL;DR

SEGMENT+ is a framework that enhances long-text processing in language models by managing extended inputs efficiently through structured notes and filtering, improving performance on long-document tasks.

Contribution

It introduces a novel framework that enables language models to handle longer inputs effectively within limited context windows, with a focus on control and interpretability.

Findings

Improves long-document question-answering performance.

Effective in Needle-in-a-Haystack tasks.

Works across various model sizes.

Abstract

There is a growing interest in expanding the input capacity of language models (LMs) across various domains. However, simply increasing the context window does not guarantee robust performance across diverse long-input processing tasks, such as understanding extensive documents and extracting detailed information from lengthy and noisy data. In response, we introduce SEGMENT+, a general framework that enables LMs to handle extended inputs within limited context windows efficiently. SEGMENT+ utilizes structured notes and a filtering module to manage information flow, resulting in a system that is both controllable and interpretable. Our extensive experiments across various model sizes, focusing on long-document question-answering and Needle-in-a-Haystack tasks, demonstrate the effectiveness of SEGMENT+ in improving performance.