Efficient Long Context Language Model Retrieval with Compression

TL;DR

This paper introduces CoLoR, a passage compression method for Long Context Language Models that enhances retrieval performance by nearly 6% while reducing in-context passage size by almost half, making IR more efficient.

Contribution

The paper presents a novel passage compression approach tailored for LCLM retrieval, trained with synthetic data and preference optimization to improve efficiency and effectiveness.

Findings

Improves retrieval performance by 6% on 9 datasets.

Reduces in-context passage size by a factor of 1.91.

Demonstrates effectiveness of compression in LCLM retrieval.

Abstract

Long Context Language Models (LCLMs) have emerged as a new paradigm to perform Information Retrieval (IR), which enables the direct ingestion and retrieval of information by processing an entire corpus in their single context, showcasing the potential to surpass traditional sparse and dense retrieval methods. However, processing a large number of passages within in-context for retrieval is computationally expensive, and handling their representations during inference further exacerbates the processing time; thus, we aim to make LCLM retrieval more efficient and potentially more effective with passage compression. Specifically, we propose a new compression approach tailored for LCLM retrieval, which is trained to maximize the retrieval performance while minimizing the length of the compressed passages. To accomplish this, we generate the synthetic data, where compressed passages are automatically created and labeled as chosen or rejected according to their retrieval success for a given query, and we train the proposed Compression model for Long context Retrieval (CoLoR) with this data via preference optimization while adding the length regularization loss on top of it to enforce brevity. Through extensive experiments on 9 datasets, we show that CoLoR improves the retrieval performance by 6% while compressing the in-context size by a factor of 1.91. Our code is available at: https://github.com/going-doer/CoLoR.