Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives

TL;DR

This paper provides a comprehensive survey of data mixture strategies for large language models, introduces a new fine-grained categorization, and discusses challenges in optimizing domain weights under fixed training budgets.

Contribution

It offers a detailed taxonomy of existing data mixture methods, extending previous classifications, and clarifies their relationships and limitations.

Findings

Categorized offline methods into heuristic, algorithm, and function fitting-based approaches.

Grouped online methods into min-max optimization, mixing law, and others.

Discussed advantages, disadvantages, and key challenges of each method.

Abstract

Training large language models with data collected from various domains can improve their performance on downstream tasks. However, given a fixed training budget, the sampling proportions of these different domains significantly impact the model's performance. How can we determine the domain weights across different data domains to train the best-performing model within constrained computational resources? In this paper, we provide a comprehensive overview of existing data mixture methods. First, we propose a fine-grained categorization of existing methods, extending beyond the previous offline and online classification. Offline methods are further grouped into heuristic-based, algorithm-based, and function fitting-based methods. For online methods, we categorize them into three groups: online min-max optimization, online mixing law, and other approaches by drawing connections with the optimization frameworks underlying offline methods. Second, we summarize the problem formulations, representative algorithms for each subtype of offline and online methods, and clarify the relationships and distinctions among them. Finally, we discuss the advantages and disadvantages of each method and highlight key challenges in the field of data mixture.