MANTRA: a Framework for Multi-stage Adaptive Noise TReAtment During Training

TL;DR

MANTRA is a multi-stage adaptive noise treatment framework that improves the robustness and accuracy of code language models by diagnosing and mitigating noise during fine-tuning, leading to better performance in software engineering tasks.

Contribution

It introduces a novel noise diagnosis and mitigation approach integrated into fine-tuning code language models, enhancing their robustness against noisy data in software engineering tasks.

Findings

MANTRA improves model performance on code summarization and commit intent classification.

Some LLMs are more sensitive to noise, but MANTRA enhances robustness across models.

The framework reduces data cleaning time and maximizes fine-tuning effectiveness.

Abstract

The reliable application of deep learning models to software engineering tasks hinges on high-quality training data. Yet, large-scale repositories inevitably introduce noisy or mislabeled examples that degrade both accuracy and robustness. While Noise Label Learning (NLL) has been extensively studied in other fields, there are a few works that investigate NLL in Software Engineering (SE) and Large Language Models (LLMs) for SE tasks. In this work, we propose MANTRA, a Multi-stage Adaptive Noise TReAtment framework that embeds noise diagnosis and mitigation directly into the fine-tuning process of code-Pretrained Language Models (PTM) and code-LLMs. We first investigate the effect of noise at varying levels on convergence and loss trajectories of the models. Then we apply an adaptive dropout strategy guided by per-sample loss dynamics and Gaussian Mixture Model clustering to exclude persistently noisy points while preserving clean data. Applying to code summarization and commit intent classification, our experiments reveal that some LLMs are more sensitive to noise than others. However, with MANTRA, the performance of all models in both tasks is improved. MANTRA enables researchers and practitioners to reduce the impact of errors introduced by the dataset in training, saves time in data cleaning and processing, while maximizing the effect of fine-tuning.