Implicit to Explicit Entropy Regularization: Benchmarking ViT Fine-tuning under Noisy Labels

TL;DR

This paper benchmarks the robustness of Vision Transformers (ViTs) against noisy labels and explores how explicit entropy regularization can improve their performance, comparing with CNNs and analyzing various NLL methods across multiple datasets.

Contribution

It provides a comprehensive benchmark of ViT fine-tuning under noisy labels and demonstrates that explicit entropy regularization enhances ViT robustness and performance.

Findings

Entropy regularization improves ViT robustness to noisy labels.

ViTs are more vulnerable to noisy labels than CNNs.

Explicit entropy minimization enhances NLL methods for ViTs.

Abstract

Automatic annotation of large-scale datasets can introduce noisy training data labels, which adversely affect the learning process of deep neural networks (DNNs). Consequently, Noisy Labels Learning (NLL) has become a critical research field for Convolutional Neural Networks (CNNs), though it remains less explored for Vision Transformers (ViTs). In this study, we evaluate the vulnerability of ViT fine-tuning to noisy labels and compare its robustness with CNNs. We also investigate whether NLL methods developed for CNNs are equally effective for ViTs. Using linear probing and MLP-K fine-tuning, we benchmark two ViT backbones (ViT-B/16 and ViT-L/16) using three commonly used classification losses: Cross Entropy (CE), Focal Loss (FL), and Mean Absolute Error (MAE), alongside six robust NLL methods: GCE, SCE, NLNL, APL, NCE+AGCE, and ANL-CE. The evaluation is conducted across six datasets including MNIST, CIFAR-10/100, WebVision, Clothing1M, and Food-101N. Furthermore, we explore whether implicit prediction entropy minimization contributes to ViT robustness against noisy labels, noting a general trend of prediction entropy reduction across most NLL methods. Building on this observation, we examine whether explicit entropy minimization could enhance ViT resilience to noisy labels. Our findings indicate that incorporating entropy regularization enhances the performance of established loss functions such as CE and FL, as well as the robustness of the six studied NLL methods across both ViT backbones.