Robust Generalization against Photon-Limited Corruptions via Worst-Case Sharpness Minimization

TL;DR

This paper introduces SharpDRO, a novel method that minimizes worst-case sharpness to improve robust generalization against photon-limited corruptions, outperforming existing approaches on multiple datasets.

Contribution

SharpDRO is a new approach that penalizes sharpness of the worst-case distribution, leading to flatter loss landscapes and better robustness in corrupted data scenarios.

Findings

SharpDRO achieves superior robustness on CIFAR10/100 and ImageNet30 datasets.

It outperforms baseline methods with significant performance gains.

Theoretically guarantees convergence of the proposed optimization.

Abstract

Robust generalization aims to tackle the most challenging data distributions which are rare in the training set and contain severe noises, i.e., photon-limited corruptions. Common solutions such as distributionally robust optimization (DRO) focus on the worst-case empirical risk to ensure low training error on the uncommon noisy distributions. However, due to the over-parameterized model being optimized on scarce worst-case data, DRO fails to produce a smooth loss landscape, thus struggling on generalizing well to the test set. Therefore, instead of focusing on the worst-case risk minimization, we propose SharpDRO by penalizing the sharpness of the worst-case distribution, which measures the loss changes around the neighbor of learning parameters. Through worst-case sharpness minimization, the proposed method successfully produces a flat loss curve on the corrupted distributions, thus achieving robust generalization. Moreover, by considering whether the distribution annotation is available, we apply SharpDRO to two problem settings and design a worst-case selection process for robust generalization. Theoretically, we show that SharpDRO has a great convergence guarantee. Experimentally, we simulate photon-limited corruptions using CIFAR10/100 and ImageNet30 datasets and show that SharpDRO exhibits a strong generalization ability against severe corruptions and exceeds well-known baseline methods with large performance gains.