On Sensitivity of Learning with Limited Labelled Data to the Effects of Randomness: Impact of Interactions and Systematic Choices

TL;DR

This paper systematically investigates how randomness factors affect learning with limited labeled data, revealing that interactions and systematic choices significantly influence performance variability across different learning methods.

Contribution

It introduces a method to accurately measure individual randomness effects by accounting for interactions, clarifying inconsistent prior findings and highlighting the importance of systematic factors.

Findings

Randomness interactions cause inconsistent effects attribution.

Sample order sensitivity depends on systematic choices.

Disregarding interactions leads to incorrect conclusions.

Abstract

While learning with limited labelled data can improve performance when the labels are lacking, it is also sensitive to the effects of uncontrolled randomness introduced by so-called randomness factors (e.g., varying order of data). We propose a method to systematically investigate the effects of randomness factors while taking the interactions between them into consideration. To measure the true effects of an individual randomness factor, our method mitigates the effects of other factors and observes how the performance varies across multiple runs. Applying our method to multiple randomness factors across in-context learning and fine-tuning approaches on 7 representative text classification tasks and meta-learning on 3 tasks, we show that: 1) disregarding interactions between randomness factors in existing works caused inconsistent findings due to incorrect attribution of the effects of randomness factors, such as disproving the consistent sensitivity of in-context learning to sample order even with random sample selection; and 2) besides mutual interactions, the effects of randomness factors, especially sample order, are also dependent on more systematic choices unexplored in existing works, such as number of classes, samples per class or choice of prompt format.