CleaR: Towards Robust and Generalized Parameter-Efficient Fine-Tuning for Noisy Label Learning

TL;DR

This paper introduces CleaR, a routing-based PEFT method that adaptively filters noisy labels during fine-tuning, significantly improving robustness and performance in noisy label environments.

Contribution

CleaR is a novel routing-based PEFT approach that selectively exposes modules to clean data, enhancing robustness against noisy labels.

Findings

CleaR outperforms existing PEFT methods in noisy environments.

PEFT's limited capacity naturally confers robustness to noisy labels.

CleaR effectively minimizes the influence of noisy labels during training.

Abstract

Parameter-efficient fine-tuning (PEFT) has enabled the efficient optimization of cumbersome language models in real-world settings. However, as datasets in such environments often contain noisy labels that adversely affect performance, PEFT methods are inevitably exposed to noisy labels. Despite this challenge, the adaptability of PEFT to noisy environments remains underexplored. To bridge this gap, we investigate various PEFT methods under noisy labels. Interestingly, our findings reveal that PEFT has difficulty in memorizing noisy labels due to its inherently limited capacity, resulting in robustness. However, we also find that such limited capacity simultaneously makes PEFT more vulnerable to interference of noisy labels, impeding the learning of clean samples. To address this issue, we propose Clean Routing (CleaR), a novel routing-based PEFT approach that adaptively activates PEFT modules. In CleaR, PEFT modules are preferentially exposed to clean data while bypassing the noisy ones, thereby minimizing the noisy influence. To verify the efficacy of CleaR, we perform extensive experiments on diverse configurations of noisy labels. The results convincingly demonstrate that CleaR leads to substantially improved performance in noisy environments.