Noise-Robustness Through Noise: A Framework combining Asymmetric LoRA with Poisoning MoE

TL;DR

This paper introduces LoPE, a novel framework that enhances language model robustness to noisy data by using asymmetric LoRA poisoning experts, eliminating the need for data cleaning.

Contribution

The paper proposes a new noise-robust adaptation method combining asymmetric LoRA with poisoning experts, improving noise handling without data pre-processing or complex architecture changes.

Findings

LoPE improves noise robustness in language models.

It achieves strong performance without data cleaning.

The method is cost-effective and easy to implement.

Abstract

Current parameter-efficient fine-tuning methods for adapting pre-trained language models to downstream tasks are susceptible to interference from noisy data. Conventional noise-handling approaches either rely on laborious data pre-processing or employ model architecture modifications prone to error accumulation. In contrast to existing noise-process paradigms, we propose a noise-robust adaptation method via asymmetric LoRA poisoning experts (LoPE), a novel framework that enhances model robustness to noise only with generated noisy data. Drawing inspiration from the mixture-of-experts architecture, LoPE strategically integrates a dedicated poisoning expert in an asymmetric LoRA configuration. Through a two-stage paradigm, LoPE performs noise injection on the poisoning expert during fine-tuning to enhance its noise discrimination and processing ability. During inference, we selectively mask the dedicated poisoning expert to leverage purified knowledge acquired by normal experts for noise-robust output. Extensive experiments demonstrate that LoPE achieves strong performance and robustness purely through the low-cost noise injection, which completely eliminates the requirement of data cleaning.