APAM: Adaptive Pre-training and Adaptive Meta Learning in Language Model for Noisy Labels and Long-tailed Learning

TL;DR

This paper introduces a flexible framework combining adaptive pre-training and meta-learning to improve NLP model robustness against noisy labels and long-tailed data distributions, outperforming existing methods.

Contribution

It proposes a novel contrastive learning-based adaptive framework with a learnable re-weighting module for handling noisy and long-tailed NLP data.

Findings

Consistently outperforms baseline methods in experiments.

Effectively mitigates negative impact of noisy labels.

Handles long-tailed data distribution robustly.

Abstract

Practical natural language processing (NLP) tasks are commonly long-tailed with noisy labels. Those problems challenge the generalization and robustness of complex models such as Deep Neural Networks (DNNs). Some commonly used resampling techniques, such as oversampling or undersampling, could easily lead to overfitting. It is growing popular to learn the data weights leveraging a small amount of metadata. Besides, recent studies have shown the advantages of self-supervised pre-training, particularly to the under-represented data. In this work, we propose a general framework to handle the problem of both long-tail and noisy labels. The model is adapted to the domain of problems in a contrastive learning manner. The re-weighting module is a feed-forward network that learns explicit weighting functions and adapts weights according to metadata. The framework further adapts weights of terms in the loss function through a combination of the polynomial expansion of cross-entropy loss and focal loss. Our extensive experiments show that the proposed framework consistently outperforms baseline methods. Lastly, our sensitive analysis emphasizes the capability of the proposed framework to handle the long-tailed problem and mitigate the negative impact of noisy labels.