In-Context Learning with Noisy Labels

TL;DR

This paper introduces a new challenge in in-context learning where noisy labels in demonstrations can degrade performance, and proposes methods to mitigate this issue, improving robustness of large language models.

Contribution

It defines the task of in-context learning with noisy labels and proposes a novel method to address label noise, enhancing the robustness of LLMs in real-world scenarios.

Findings

Proposed method effectively mitigates performance loss due to noisy labels.

Demonstrated robustness of the method across various noisy label scenarios.

Established baseline approaches for in-context learning with noisy labels.

Abstract

In-context learning refers to the emerging ability of large language models (LLMs) to perform a target task without additional training, utilizing demonstrations of the task. Recent studies aim to enhance in-context learning performance by selecting more useful demonstrations. However, they overlook the presence of inevitable noisy labels in task demonstrations that arise during the labeling process in the real-world. In this paper, we propose a new task, in-context learning with noisy labels, which aims to solve real-world problems for in-context learning where labels in task demonstrations would be corrupted. Moreover, we propose a new method and baseline methods for the new task, inspired by studies in learning with noisy labels. Through experiments, we demonstrate that our proposed method can serve as a safeguard against performance degradation in in-context learning caused by noisy labels.