The Chicken and Egg Dilemma: Co-optimizing Data and Model Configurations for LLMs

TL;DR

This paper introduces JoBS, a novel method that efficiently co-optimizes data and model configurations for training large language models by leveraging a performance predictor and Bayesian optimization, outperforming existing approaches.

Contribution

JoBS is the first approach to jointly optimize data and model configurations for LLMs using a scaling-law-inspired predictor within a Bayesian optimization framework.

Findings

JoBS outperforms existing multi-fidelity BO baselines.

JoBS achieves better optimization results across diverse LLM tasks.

The method effectively allocates budget between predictor learning and optimization.

Abstract

Co-optimizing data and model configurations for training LLMs presents a classic chicken-and-egg dilemma: The best training data configuration (e.g., data mixture) for a downstream task depends on the chosen model configuration (e.g., model architecture), and vice versa. However, jointly optimizing both data and model configurations is often deemed intractable, and existing methods focus on either data or model optimization without considering their interaction. We introduce JoBS, an approach that uses a scaling-law-inspired performance predictor to aid Bayesian optimization (BO) in jointly optimizing LLM training data and model configurations efficiently. JoBS allocates a portion of the optimization budget to learn an LLM performance predictor that predicts how promising a training configuration is from a small number of training steps. The remaining budget is used to perform BO entirely with the predictor, effectively amortizing the cost of running full-training runs. We study JoBS's average regret and devise the optimal budget allocation to minimize regret. JoBS outperforms existing multi-fidelity BO baselines, as well as data and model optimization approaches across diverse LLM tasks under the same optimization budget.