Representation Projection Invariance Mitigates Representation Collapse

TL;DR

This paper introduces REPINA, a regularization method that preserves information in language model representations during fine-tuning, reducing collapse and improving performance across various NLP tasks.

Contribution

The paper proposes a novel regularization technique, Representation Projection Invariance (REPINA), to mitigate representation collapse during fine-tuning of language models.

Findings

REPINA outperforms baselines on most GLUE tasks

It is effective in few-shot learning scenarios

It enhances robustness to label perturbations

Abstract

Fine-tuning contextualized representations learned by pre-trained language models remains a prevalent practice in NLP. However, fine-tuning can lead to representation degradation (also known as representation collapse), which may result in instability, sub-optimal performance, and weak generalization. In this paper, we propose Representation Projection Invariance (REPINA), a novel regularization method to maintain the information content of representation and reduce representation collapse during fine-tuning by discouraging undesirable changes in the representations. We study the empirical behavior of the proposed regularization in comparison to 5 comparable baselines across 13 language understanding tasks (GLUE benchmark and six additional datasets). When evaluating in-domain performance, REPINA consistently outperforms other baselines on most tasks (10 out of 13). We also demonstrate its effectiveness in few-shot settings and robustness to label perturbation. As a by-product, we extend previous studies of representation collapse and propose several metrics to quantify it. Our empirical findings show that our approach is significantly more effective at mitigating representation collapse.