Fighting Randomness with Randomness: Mitigating Optimisation Instability of Fine-Tuning using Delayed Ensemble and Noisy Interpolation

TL;DR

This paper introduces DENI, a novel, computationally efficient mitigation strategy for fine-tuning language models that reduces performance instability caused by randomness, outperforming existing methods and benefiting from data augmentation.

Contribution

The paper proposes DENI, a new mitigation approach combining ensembling, noise regularisation, and interpolation, which is more efficient and effective than existing strategies.

Findings

DENI outperforms the best existing mitigation strategy with less computational cost.

Mitigation strategies, including DENI, improve parameter-efficient fine-tuning performance.

Combining DENI with data augmentation enhances instability mitigation.

Abstract

While fine-tuning of pre-trained language models generally helps to overcome the lack of labelled training samples, it also displays model performance instability. This instability mainly originates from randomness in initialisation or data shuffling. To address this, researchers either modify the training process or augment the available samples, which typically results in increased computational costs. We propose a new mitigation strategy, called Delayed Ensemble with Noisy Interpolation (DENI), that leverages the strengths of ensembling, noise regularisation and model interpolation, while retaining computational efficiency. We compare DENI with 9 representative mitigation strategies across 3 models, 4 tuning strategies and 7 text classification datasets. We show that: 1) DENI outperforms the best performing mitigation strategy (Ensemble), while using only a fraction of its cost; 2) the mitigation strategies are beneficial for parameter-efficient fine-tuning (PEFT) methods, outperforming full fine-tuning in specific cases; and 3) combining DENI with data augmentation often leads to even more effective instability mitigation.