DUET: Optimizing Training Data Mixtures via Feedback from Unseen Evaluation Tasks

TL;DR

DUET is a novel algorithm that optimizes training data mixtures for unseen evaluation tasks using feedback, combining influence functions and Bayesian optimization, and it converges to optimal data mixtures without prior task data.

Contribution

It introduces DUET, a new method that effectively optimizes training data for unseen tasks by leveraging feedback and theoretical analysis of convergence.

Findings

DUET outperforms existing methods in unseen-task settings.

Theoretical analysis shows DUET's convergence to optimal data mixtures.

Experiments across language tasks validate DUET's effectiveness.

Abstract

The performance of an LLM depends heavily on the relevance of its training data to the downstream evaluation task. However, in practice, the data involved in an unseen evaluation task is often unknown (e.g., conversations between an LLM and a user are end-to-end encrypted). Hence, it is unclear what data are relevant for fine-tuning the LLM to maximize its performance on the specific unseen evaluation task. Instead, one can only deploy the LLM on the unseen task to gather multiple rounds of feedback on how well the model performs (e.g., user ratings). This novel setting offers a refreshing perspective towards optimizing training data mixtures via feedback from an unseen evaluation task, which prior data mixing and selection works do not consider. Our paper presents DUET, a novel global-to-local algorithm that interleaves influence function as a data selection method with Bayesian optimization to optimize data mixture via feedback from a specific unseen evaluation task. By analyzing DUET's cumulative regret, we theoretically show that DUET converges to the optimal training data mixture for an unseen task even without any data knowledge of the task. Finally, our experiments across a variety of language tasks demonstrate that DUET outperforms existing data selection and mixing methods in the unseen-task setting.