Calibrating Pre-trained Language Classifiers on LLM-generated Noisy Labels via Iterative Refinement

TL;DR

This paper introduces SiDyP, a method to improve classifier robustness against noisy labels generated by large language models, by iteratively refining label predictions using a simplex diffusion approach, significantly enhancing NLP task performance.

Contribution

The paper presents SiDyP, a novel iterative refinement framework that calibrates classifier predictions to handle LLM-generated noisy labels, improving NLP classifier accuracy.

Findings

Increases BERT classifier performance by over 7% on noisy datasets.

Effectively refines noisy labels using neighborhood distribution and diffusion.

Demonstrates robustness across various LLMs and NLP tasks.

Abstract

The traditional process of creating labeled datasets is labor-intensive and expensive. Recent breakthroughs in open-source large language models (LLMs) have opened up a new avenue in generating labeled datasets automatically for various natural language processing (NLP) tasks, providing an alternative to such an expensive annotation process. However, the reliability of such auto-generated labels remains a significant concern due to inherent inaccuracies. When learning from noisy labels, the model's generalization is likely to be harmed as it is prone to overfit to those label noises. While previous studies in learning from noisy labels mainly focus on synthetic noise and real-world noise, LLM-generated label noise receives less attention. In this paper, we propose SiDyP: Simplex Label Diffusion with Dynamic Prior to calibrate the classifier's prediction, thus enhancing its robustness towards LLM-generated noisy labels. SiDyP retrieves potential true label candidates by neighborhood label distribution in text embedding space and iteratively refines noisy candidates using a simplex diffusion model. Our framework can increase the performance of the BERT classifier fine-tuned on both zero-shot and few-shot LLM-generated noisy label datasets by an average of 7.21% and 7.30% respectively. We demonstrate the effectiveness of SiDyP by conducting extensive benchmarking for different LLMs over a variety of NLP tasks. Our code is available on Github.