Improving Adversarial Robustness for Free with Snapshot Ensemble

TL;DR

This paper introduces a simplified snapshot ensemble method that enhances adversarial robustness by leveraging the last few training iterations, achieving significant accuracy improvements with reduced complexity and resource requirements.

Contribution

The proposed snapshot ensemble focuses on recent training iterations rather than local minima, offering a simpler yet effective way to improve adversarial robustness.

Findings

Achieves 5% to 30% accuracy increase over traditional adversarial training.

Reduces computational and memory costs compared to standard ensemble methods.

Maintains high robustness without complex local minima optimization.

Abstract

Adversarial training, as one of the few certified defenses against adversarial attacks, can be quite complicated and time-consuming, while the results might not be robust enough. To address the issue of lack of robustness, ensemble methods were proposed, aiming to get the final output by weighting the selected results from repeatedly trained processes. It is proved to be very useful in achieving robust and accurate results, but the computational and memory costs are even higher. Snapshot ensemble, a new ensemble method that combines several local minima in a single training process to make the final prediction, was proposed recently, which reduces the time spent on training multiple networks and the memory to store the results. Based on the snapshot ensemble, we present a new method that is easier to implement: unlike original snapshot ensemble that seeks for local minima, our snapshot ensemble focuses on the last few iterations of a training and stores the sets of parameters from them. Our algorithm is much simpler but the results are no less accurate than the original ones: based on different hyperparameters and datasets, our snapshot ensemble has shown a 5% to 30% increase in accuracy when compared to the traditional adversarial training.