LM-mixup: Text Data Augmentation via Language Model based Mixup

TL;DR

This paper introduces LM-Mixup, a novel data augmentation method that distills and enhances low-quality instruction data for large language models, improving their instruction-following capabilities efficiently.

Contribution

It presents a new dataset construction pipeline, MIXTURE, and a reinforcement learning-based augmentation technique, LM-Mixup, to effectively utilize low-quality data for instruction tuning.

Findings

LM-Mixup improves model performance using only 3% of data.

Distilled data surpasses full dataset training in benchmarks.

Low-quality data can be valuable when properly distilled and augmented.

Abstract

Instruction tuning is crucial for aligning Large Language Models (LLMs), yet the quality of instruction-following data varies significantly. While high-quality data is paramount, it is often scarce; conversely, abundant low-quality data is frequently discarded, leading to substantial information loss. Existing data augmentation methods struggle to augment this low-quality data effectively, and the evaluation of such techniques remains poorly defined. To address this, we formally define the task of Instruction Distillation: distilling multiple low-quality and redundant inputs into high-quality and coherent instruction-output pairs. Specifically, we introduce a comprehensive data construction pipeline to create MIXTURE, a 144K-sample dataset pairing low-quality or semantically redundant imperfect instruction clusters with their high-quality distillations. We then introduce LM-Mixup, by first performing supervised fine-tuning on MIXTURE and then optimizing it with reinforcement learning. This process uses three complementary reward signals: quality, semantic alignment, and format compliance, via Group Relative Policy Optimization (GRPO). We demonstrate that LM-Mixup effectively augments imperfect datasets: fine-tuning LLMs on its distilled data, which accounts for only about 3% of the entire dataset, not only surpasses full-dataset training but also competes with state-of-the-art high-quality data selection methods across multiple benchmarks. Our work establishes that low-quality data is a valuable resource when properly distilled and augmented with LM-Mixup, significantly enhancing the efficiency and performance of instruction-tuned LLMs.