A Survey on Efficient Large Language Model Training: From Data-centric Perspectives

TL;DR

This survey reviews data-centric methods for efficient large language model post-training, emphasizing data selection, quality, synthetic data, and self-evolving ecosystems to address high costs and diminishing returns.

Contribution

It provides the first systematic taxonomy of data-efficient LLM post-training methods and outlines future research directions from a data-centric perspective.

Findings

Taxonomy of data-efficient post-training methods

Summary of representative approaches in each category

Identification of open problems and future research avenues

Abstract

Post-training of Large Language Models (LLMs) is crucial for unlocking their task generalization potential and domain-specific capabilities. However, the current LLM post-training paradigm faces significant data challenges, including the high costs of manual annotation and diminishing marginal returns on data scales. Therefore, achieving data-efficient post-training has become a key research question. In this paper, we present the first systematic survey of data-efficient LLM post-training from a data-centric perspective. We propose a taxonomy of data-efficient LLM post-training methods, covering data selection, data quality enhancement, synthetic data generation, data distillation and compression, and self-evolving data ecosystems. We summarize representative approaches in each category and outline future research directions. By examining the challenges in data-efficient LLM post-training, we highlight open problems and propose potential research avenues. We hope our work inspires further exploration into maximizing the potential of data utilization in large-scale model training. Paper List: https://github.com/luo-junyu/Awesome-Data-Efficient-LLM