Multiplicative Reweighting for Robust Neural Network Optimization

TL;DR

This paper introduces a multiplicative reweighting method for neural network training that enhances robustness against noisy labels and improves accuracy and adversarial robustness, supported by theoretical and empirical evidence.

Contribution

It proposes a novel multiplicative weights reweighting approach for neural network optimization, with theoretical convergence guarantees and empirical validation on noisy and adversarial data.

Findings

MW improves accuracy on CIFAR-10, CIFAR-100, Clothing1M

Theoretical convergence established for gradient descent

Enhanced adversarial robustness observed

Abstract

Neural networks are widespread due to their powerful performance. Yet, they degrade in the presence of noisy labels at training time. Inspired by the setting of learning with expert advice, where multiplicative weights (MW) updates were recently shown to be robust to moderate data corruptions in expert advice, we propose to use MW for reweighting examples during neural networks optimization. We theoretically establish the convergence of our method when used with gradient descent and prove its advantages in 1d cases. We then validate empirically our findings for the general case by showing that MW improves neural networks' accuracy in the presence of label noise on CIFAR-10, CIFAR-100 and Clothing1M. We also show the impact of our approach on adversarial robustness.