Shift is Good: Mismatched Data Mixing Improves Test Performance

TL;DR

This paper demonstrates that intentionally mismatching training and testing data proportions across components can enhance test performance, revealing that distribution shift can sometimes be beneficial even without transfer between unrelated components.

Contribution

It provides a theoretical analysis showing how mismatched data mixing can improve test outcomes and identifies optimal training proportions in various scenarios.

Findings

Distribution shift can be beneficial for test performance.

Optimal training proportions depend on the scenario.

Mismatched data mixing applies to compositional skill distributions.

Abstract

We consider training and testing on mixture distributions with different training and test proportions. We show that in many settings, and in some sense generically, distribution shift can be beneficial, and test performance can improve due to mismatched training proportions, even if the components are unrelated and with no transfer between components. In a variety of scenarios, we identify the optimal training proportions and the extent to which such distribution shift can be beneficial. We show how the same analysis applies also to a compositional setting with differing distribution of component "skills'' at training and test.