Data Distribution Matters: A Data-Centric Perspective on Context Compression for Large Language Model

TL;DR

This paper investigates how data distribution affects context compression in large language models, revealing key factors that influence compression quality and providing practical guidelines for optimization.

Contribution

It introduces a data-centric approach to analyze the impact of data distribution on context compression, a perspective previously overlooked in LLM research.

Findings

Encoder input entropy negatively correlates with compression quality.

Decoder entropy shows no significant relationship with compression.

Intrinsic data gap between encoder and decoder reduces compression gains.

Abstract

The deployment of Large Language Models (LLMs) in long-context scenarios is hindered by computational inefficiency and significant information redundancy. Although recent advancements have widely adopted context compression to address these challenges, existing research only focus on model-side improvements, the impact of the data distribution itself on context compression remains largely unexplored. To bridge this gap, we are the first to adopt a data-centric perspective to systematically investigate how data distribution impacts compression quality, including two dimensions: input data and intrinsic data (i.e., the model's internal pretrained knowledge). We evaluate the semantic integrity of compressed representations using an autoencoder-based framework to systematically investigate it. Our experimental results reveal that: (1) encoder-measured input entropy negatively correlates with compression quality, while decoder-measured entropy shows no significant relationship under a frozen-decoder setting; and (2) the gap between intrinsic data of the encoder and decoder significantly diminishes compression gains, which is hard to mitigate. Based on these findings, we further present practical guidelines to optimize compression gains.