Prototypical Fine-tuning: Towards Robust Performance Under Varying Data Sizes

TL;DR

This paper introduces prototypical fine-tuning, a novel method that enhances pretrained language models' robustness across different data sizes, especially benefiting low-resource scenarios by automatically adjusting model capacity.

Contribution

It proposes a new prototypical fine-tuning framework that dynamically adapts model capacity based on data size and introduces four principles for effective prototype tuning.

Findings

Significant performance gains in low-resource settings.

Comparable or better results in high-resource scenarios.

Automatic adjustment of model capacity improves robustness.

Abstract

In this paper, we move towards combining large parametric models with non-parametric prototypical networks. We propose prototypical fine-tuning, a novel prototypical framework for fine-tuning pretrained language models (LM), which automatically learns a bias to improve predictive performance for varying data sizes, especially low-resource settings. Our prototypical fine-tuning approach can automatically adjust the model capacity according to the number of data points and the model's inherent attributes. Moreover, we propose four principles for effective prototype fine-tuning towards the optimal solution. Experimental results across various datasets show that our work achieves significant performance improvements under various low-resource settings, as well as comparable and usually better performances in high-resource scenarios.