Commute Your Domains: Trajectory Optimality Criterion for Multi-Domain Learning

TL;DR

This paper introduces a theoretical framework using Lie brackets to analyze how the order of training data from multiple domains affects model performance, validated on toy and large language models.

Contribution

It presents a novel analysis of training order effects in multi-domain learning using Lie brackets, providing insights into optimal data mixing strategies.

Findings

Training order significantly impacts model performance.

Lie bracket analysis predicts beneficial data ordering.

Validated on toy and bilingual LLM pre-training.

Abstract

In multi-domain learning, a single model is trained on diverse data domains to leverage shared knowledge and improve generalization. The order in which the data from these domains is used for training can significantly affect the model's performance on each domain. However, this dependence is under-studied. In this paper, we investigate the influence of training order (or data mixing) in multi-domain learning using the concept of Lie bracket of gradient vector fields. By analyzing the infinitesimal effects of changing the training order, we identify regions in the parameter space where altering the order between two training domains can benefit the target loss. We validate the predictions of our theoretical framework on the influence of training order (or data mixing) both on a toy example and bilingual LLM pre-training.