ImbSAM: A Closer Look at Sharpness-Aware Minimization in Class-Imbalanced Recognition

TL;DR

This paper introduces ImbSAM, a class-aware extension of SAM, designed to improve generalization for tail classes in imbalanced recognition tasks, demonstrating significant performance gains in long-tailed classification and semi-supervised anomaly detection.

Contribution

It proposes ImbSAM, a novel class-aware smoothness optimization algorithm that leverages class priors to enhance tail class generalization in imbalanced recognition.

Findings

ImbSAM outperforms standard SAM in tail class accuracy.

ImbSAM achieves significant improvements in long-tailed classification.

ImbSAM enhances anomaly detection performance in semi-supervised settings.

Abstract

Class imbalance is a common challenge in real-world recognition tasks, where the majority of classes have few samples, also known as tail classes. We address this challenge with the perspective of generalization and empirically find that the promising Sharpness-Aware Minimization (SAM) fails to address generalization issues under the class-imbalanced setting. Through investigating this specific type of task, we identify that its generalization bottleneck primarily lies in the severe overfitting for tail classes with limited training data. To overcome this bottleneck, we leverage class priors to restrict the generalization scope of the class-agnostic SAM and propose a class-aware smoothness optimization algorithm named Imbalanced-SAM (ImbSAM). With the guidance of class priors, our ImbSAM specifically improves generalization targeting tail classes. We also verify the efficacy of ImbSAM on two prototypical applications of class-imbalanced recognition: long-tailed classification and semi-supervised anomaly detection, where our ImbSAM demonstrates remarkable performance improvements for tail classes and anomaly. Our code implementation is available at https://github.com/cool-xuan/Imbalanced_SAM.