Towards Robust Low-Resource Fine-Tuning with Multi-View Compressed Representations

TL;DR

This paper introduces a novel fine-tuning method for pretrained language models that uses autoencoders to create multi-view compressed representations, reducing overfitting in low-resource NLP tasks without increasing inference costs.

Contribution

The method inserts autoencoders between hidden layers during fine-tuning to improve generalization in low-resource settings, without adding extra parameters during inference.

Findings

Improves performance on low-resource NLP tasks

Does not increase inference computational cost

Effective across sequence- and token-level tasks

Abstract

Due to the huge amount of parameters, fine-tuning of pretrained language models (PLMs) is prone to overfitting in the low resource scenarios. In this work, we present a novel method that operates on the hidden representations of a PLM to reduce overfitting. During fine-tuning, our method inserts random autoencoders between the hidden layers of a PLM, which transform activations from the previous layers into multi-view compressed representations before feeding them into the upper layers. The autoencoders are plugged out after fine-tuning, so our method does not add extra parameters or increase computation cost during inference. Our method demonstrates promising performance improvement across a wide range of sequence- and token-level low-resource NLP tasks.