MML: Maximal Multiverse Learning for Robust Fine-Tuning of Language Models

TL;DR

This paper introduces Maximal Multiverse Learning (MML), a novel fine-tuning approach for language models that employs multiple orthogonal classifier heads to enhance robustness and accuracy, especially on smaller datasets.

Contribution

The paper proposes MML, a new fine-tuning method using multiple orthogonal classifier heads that adaptively eliminate weaker ones, improving model robustness and performance.

Findings

Achieves up to +9% accuracy gain on certain datasets

Enhances robustness of BERT through parallel classifier heads

Effective especially for smaller datasets

Abstract

Recent state-of-the-art language models utilize a two-phase training procedure comprised of (i) unsupervised pre-training on unlabeled text, and (ii) fine-tuning for a specific supervised task. More recently, many studies have been focused on trying to improve these models by enhancing the pre-training phase, either via better choice of hyperparameters or by leveraging an improved formulation. However, the pre-training phase is computationally expensive and often done on private datasets. In this work, we present a method that leverages BERT's fine-tuning phase to its fullest, by applying an extensive number of parallel classifier heads, which are enforced to be orthogonal, while adaptively eliminating the weaker heads during training. Our method allows the model to converge to an optimal number of parallel classifiers, depending on the given dataset at hand. We conduct an extensive inter- and intra-dataset evaluations, showing that our method improves the robustness of BERT, sometimes leading to a +9\% gain in accuracy. These results highlight the importance of a proper fine-tuning procedure, especially for relatively smaller-sized datasets. Our code is attached as supplementary and our models will be made completely public.