Towards Next-Generation LLM Training: From the Data-Centric Perspective

TL;DR

This paper advocates for a data-centric approach to LLM training, emphasizing automated data preparation and dynamic data management to improve efficiency and performance.

Contribution

It introduces the concept of agent-based data preparation systems and a unified data-model interaction training framework for LLMs.

Findings

Proposes automated, scalable data workflow construction.

Suggests dynamic data selection and reweighting during training.

Highlights challenges and future directions in data-centric LLM training.

Abstract

Large language models (LLMs) have demonstrated remarkable performance across a wide range of tasks and domains, with data playing a central role in enabling these advances. Despite this success, the preparation and effective utilization of the massive datasets required for LLM training remain major bottlenecks. In current practice, LLM training data is often constructed using ad hoc scripts, and there is still a lack of mature, agent-based data preparation systems that can automatically construct robust and reusable data workflows, thereby freeing data scientists from repetitive and error-prone engineering efforts. Moreover, once collected, datasets are often consumed largely in their entirety during training, without systematic mechanisms for data selection, mixture optimization, or reweighting. To address these limitations, we advocate two complementary research directions. First, we propose building a robust, agent-based automatic data preparation system that supports automated workflow construction and scalable data management. Second, we argue for a unified data-model interaction training system in which data is dynamically selected, mixed, and reweighted throughout the training process, enabling more efficient, adaptive, and performance-aware data utilization. Finally, we discuss the remaining challenges and outline promising directions for future research and system development.