Information Guided Regularization for Fine-tuning Language Models

TL;DR

This paper introduces an information-theoretic approach to regularization in fine-tuning language models, proposing guided dropout that improves generalization without extra computational cost.

Contribution

It presents a novel, task-agnostic dropout method based on insights into the loss landscape, enhancing transfer learning in language models.

Findings

Guided dropout improves downstream performance across tasks.

The method is effective even with limited data.

No additional computational overhead is introduced.

Abstract

The pretraining-fine-tuning paradigm has been the de facto strategy for transfer learning in modern language modeling. With the understanding that task adaptation in LMs is often a function of parameters shared across tasks, we argue that a more surgical approach to regularization needs to exist for smoother transfer learning. Towards this end, we investigate how the pretraining loss landscape is affected by these task-sensitive parameters through an information-theoretic lens. We then leverage the findings from our investigations to devise a novel approach to dropout for improved model regularization and better downstream generalization. This approach, named guided dropout, is both task & architecture agnostic and adds no computational overhead to the fine-tuning process. Through empirical evaluations, we showcase that our approach to regularization yields consistently better performance, even in scenarios of data paucity, compared to standardized baselines.