On Representation Redundancy in Large-Scale Instruction Tuning Data Selection

TL;DR

This paper introduces CRDS, a novel data selection framework for instruction tuning that reduces redundancy in semantic representations, leading to improved data quality and model performance with significantly less data.

Contribution

The paper proposes CRDS, a new method for data selection that mitigates semantic redundancy in large language models, outperforming existing methods and reducing data requirements.

Findings

CRDS-W achieves strong performance with only 3.5% of data.

Both CRDS variants outperform state-of-the-art methods.

CRDS improves data quality and model performance.

Abstract

Data quality is a crucial factor in large language models training. While prior work has shown that models trained on smaller, high-quality datasets can outperform those trained on much larger but noisy or low-quality corpora, systematic methods for industrial-scale data selection in instruction tuning remain underexplored. In this work, we study instruction-tuning data selection through the lens of semantic representation similarity and identify a key limitation of state-of-the-art LLM encoders: they produce highly redundant semantic embeddings. To mitigate this redundancy, we propose Compressed Representation Data Selection (CRDS), a novel framework with two variants. CRDS-R applies Rademacher random projection followed by concatenation of transformer hidden-layer representations, while CRDS-W employs whitening-based dimensionality reduction to improve representational quality. Experimental results demonstrate that both variants substantially enhance data quality and consistently outperform state-of-the-art representation-based selection methods. Notably, CRDS-W achieves strong performance using only 3.5% of the data, surpassing the full-data baseline by an average of 0.71% across four datasets. Our code is available at https://github.com/tdano1/CRDS.