ACER: Automatic Language Model Context Extension via Retrieval

TL;DR

This paper introduces ACER, a method that enhances language models' long-context understanding by using retrieval-based data synthesis and self-tuning of short-context models, outperforming existing models in long-context tasks.

Contribution

ACER proposes an automatic data synthesis pipeline inspired by human retrieval, enabling task-specific long-context capabilities in language models without extensive task-specific data.

Findings

ACER outperforms generalist long-context models in retrieval tasks.

Synthetic data improves long-context reasoning abilities.

Self-tuning of short-context models enhances performance in complex tasks.

Abstract

Long-context modeling is one of the critical capabilities of language AI for digesting and reasoning over complex information pieces. In practice, long-context capabilities are typically built into a pre-trained language model~(LM) through a carefully designed context extension stage, with the goal of producing generalist long-context capabilities. In our preliminary experiments, however, we discovered that the current open-weight generalist long-context models are still lacking in practical long-context processing tasks. While this means perfectly effective long-context modeling demands task-specific data, the cost can be prohibitive. In this paper, we draw inspiration from how humans process a large body of information: a lossy \textbf{retrieval} stage ranks a large set of documents while the reader ends up reading deeply only the top candidates. We build an \textbf{automatic} data synthesis pipeline that mimics this process using short-context LMs. The short-context LMs are further tuned using these self-generated data to obtain task-specific long-context capabilities. Similar to how pre-training learns from imperfect data, we hypothesize and further demonstrate that the short-context model can bootstrap over the synthetic data, outperforming not only long-context generalist models but also the retrieval and read pipeline used to synthesize the training data in real-world tasks such as long-context retrieval augmented generation.