PATS: Sensitivity-aware Noisy Learning for Pretrained Language Models

TL;DR

PATS introduces a sensitivity-aware noisy training method that selectively adds noise to parameters based on their importance, improving fine-tuning performance of pretrained language models across various NLP tasks.

Contribution

The paper proposes PATS, a novel noise-based fine-tuning approach that considers parameter sensitivity to enhance model performance and parameter utilization.

Findings

PATS consistently improves fine-tuning results on the GLUE benchmark.

Parameters with higher sensitivity tend to have more concentrated distributions after PATS.

PATS effectively activates less sensitive parameters without disrupting sensitive ones.

Abstract

A wide range of NLP tasks benefit from the fine-tuning of pretrained language models (PLMs). However, a number of redundant parameters which contribute less to the downstream task are observed in a directly fine-tuned model. We consider the gap between pretraining and downstream tasks hinders the training of these redundant parameters, and results in a suboptimal performance of the overall model. In this paper, we present PATS (Perturbation According To Sensitivity), a noisy training mechanism which considers each parameter's importance in the downstream task to help fine-tune PLMs. The main idea of PATS is to add bigger noise to parameters with lower sensitivity and vice versa, in order to activate more parameters' contributions to downstream tasks without affecting the sensitive ones much. Extensive experiments conducted on different tasks of the GLUE benchmark show PATS can consistently empower the fine-tuning of different sizes of PLMs, and the parameters in the well-performing models always have more concentrated distributions of sensitivities, which experimentally proves the effectiveness of our method.