Revisiting Distance Metric Learning for Few-Shot Natural Language Classification

TL;DR

This paper explores the application of Distance Metric Learning (DML) to improve few-shot NLP classification by fine-tuning RoBERTa models, demonstrating that proxy-based DML losses enhance performance over traditional methods.

Contribution

It systematically evaluates DML loss functions, especially proxy-based losses, in few-shot NLP tasks and shows their effectiveness during both training and inference phases.

Findings

Proxy-based DML losses improve few-shot NLP classification performance.

Combining CCE with ProxyAnchor Loss yields the best results.

Models with combined loss outperform CCE-only models by up to 10.38 percentage points.

Abstract

Distance Metric Learning (DML) has attracted much attention in image processing in recent years. This paper analyzes its impact on supervised fine-tuning language models for Natural Language Processing (NLP) classification tasks under few-shot learning settings. We investigated several DML loss functions in training RoBERTa language models on known SentEval Transfer Tasks datasets. We also analyzed the possibility of using proxy-based DML losses during model inference. Our systematic experiments have shown that under few-shot learning settings, particularly proxy-based DML losses can positively affect the fine-tuning and inference of a supervised language model. Models tuned with a combination of CCE (categorical cross-entropy loss) and ProxyAnchor Loss have, on average, the best performance and outperform models with only CCE by about 3.27 percentage points -- up to 10.38 percentage points depending on the training dataset.