Fine tuning Pre trained Models for Robustness Under Noisy Labels

TL;DR

This paper introduces TURN, a novel fine-tuning algorithm for pre-trained models that enhances robustness against noisy labels by protecting feature extractors and reducing label noise, leading to improved performance.

Contribution

The paper proposes a new fine-tuning method, TURN, specifically designed for noisy datasets, combining classifier tuning and noise reduction for better robustness and efficiency.

Findings

TURN outperforms previous methods on benchmark datasets.

The approach effectively reduces the impact of noisy labels.

It maintains high generalization performance with lower training costs.

Abstract

The presence of noisy labels in a training dataset can significantly impact the performance of machine learning models. To tackle this issue, researchers have explored methods for Learning with Noisy Labels to identify clean samples and reduce the influence of noisy labels. However, constraining the influence of a certain portion of the training dataset can result in a reduction in overall generalization performance. To alleviate this, recent studies have considered the careful utilization of noisy labels by leveraging huge computational resources. Therefore, the increasing training cost necessitates a reevaluation of efficiency. In other areas of research, there has been a focus on developing fine-tuning techniques for large pre-trained models that aim to achieve both high generalization performance and efficiency. However, these methods have mainly concentrated on clean datasets, and there has been limited exploration of the noisy label scenario. In this research, our aim is to find an appropriate way to fine-tune pre-trained models for noisy labeled datasets. To achieve this goal, we investigate the characteristics of pre-trained models when they encounter noisy datasets. Through empirical analysis, we introduce a novel algorithm called TURN, which robustly and efficiently transfers the prior knowledge of pre-trained models. The algorithm consists of two main steps: (1) independently tuning the linear classifier to protect the feature extractor from being distorted by noisy labels, and (2) reducing the noisy label ratio and fine-tuning the entire model based on the noise-reduced dataset to adapt it to the target dataset. The proposed algorithm has been extensively tested and demonstrates efficient yet improved denoising performance on various benchmarks compared to previous methods.