Boosting LLM via Learning from Data Iteratively and Selectively

TL;DR

This paper introduces IterIT, an iterative data selection method for instruction tuning of large language models that dynamically updates sample scores based on complexity and diversity, leading to improved performance.

Contribution

The paper presents a novel iterative data selection approach that updates complexity scores during fine-tuning, enhancing data quality and model performance.

Findings

Consistent improvements over strong baselines.

Effective in domain-specific scenarios.

Generalizes across different models.

Abstract

Datasets nowadays are generally constructed from multiple sources and using different synthetic techniques, making data de-noising and de-duplication crucial before being used for post-training. In this work, we propose to perform instruction tuning by iterative data selection (\ApproachName{}). We measure the quality of a sample from complexity and diversity simultaneously. Instead of calculating the complexity score once for all before fine-tuning, we highlight the importance of updating this model-specific score during fine-tuning to accurately accommodate the dynamic changes of the model. On the other hand, the diversity score is defined on top of the samples' responses under the consideration of their informativeness. IterIT integrates the strengths of both worlds by iteratively updating the complexity score for the top-ranked samples and greedily selecting the ones with the highest complexity-diversity score. Experiments on multiple instruction-tuning data demonstrate consistent improvements of IterIT over strong baselines. Moreover, our approach also generalizes well to domain-specific scenarios and different backbone models. All resources will be available at https://github.com/JiaQiSJTU/IterIT.