Noise Stability Regularization for Improving BERT Fine-tuning

TL;DR

This paper introduces Layer-wise Noise Stability Regularization (LNSR), a novel method to enhance the stability and generalizability of BERT fine-tuning, especially with limited training data, by leveraging noise stability properties of deep networks.

Contribution

The paper proposes a new regularization technique, LNSR, grounded in noise stability theory, to improve BERT fine-tuning stability and outperform existing methods.

Findings

LNSR improves fine-tuning stability and generalizability.

Models with LNSR show lower sensitivity to noise.

LNSR outperforms state-of-the-art regularization methods.

Abstract

Fine-tuning pre-trained language models such as BERT has become a common practice dominating leaderboards across various NLP tasks. Despite its recent success and wide adoption, this process is unstable when there are only a small number of training samples available. The brittleness of this process is often reflected by the sensitivity to random seeds. In this paper, we propose to tackle this problem based on the noise stability property of deep nets, which is investigated in recent literature (Arora et al., 2018; Sanyal et al., 2020). Specifically, we introduce a novel and effective regularization method to improve fine-tuning on NLP tasks, referred to as Layer-wise Noise Stability Regularization (LNSR). We extend the theories about adding noise to the input and prove that our method gives a stabler regularization effect. We provide supportive evidence by experimentally confirming that well-performing models show a low sensitivity to noise and fine-tuning with LNSR exhibits clearly higher generalizability and stability. Furthermore, our method also demonstrates advantages over other state-of-the-art algorithms including L2-SP (Li et al., 2018), Mixout (Lee et al., 2020) and SMART (Jiang et al., 2020).