Improving Robustness by Enhancing Weak Subnets

TL;DR

This paper introduces a novel training method called EWS that identifies and enhances weak internal sub-networks in deep models, significantly improving robustness against various perturbations and adversarial attacks.

Contribution

We propose a new training procedure that explicitly strengthens weak subnets via knowledge distillation, leading to improved model robustness and accuracy.

Findings

EWS improves robustness against corrupted images.

EWS enhances accuracy on clean data.

EWS complements existing data augmentation and adversarial training methods.

Abstract

Despite their success, deep networks have been shown to be highly susceptible to perturbations, often causing significant drops in accuracy. In this paper, we investigate model robustness on perturbed inputs by studying the performance of internal sub-networks (subnets). Interestingly, we observe that most subnets show particularly poor robustness against perturbations. More importantly, these weak subnets are correlated with the overall lack of robustness. Tackling this phenomenon, we propose a new training procedure that identifies and enhances weak subnets (EWS) to improve robustness. Specifically, we develop a search algorithm to find particularly weak subnets and explicitly strengthen them via knowledge distillation from the full network. We show that EWS greatly improves both robustness against corrupted images as well as accuracy on clean data. Being complementary to popular data augmentation methods, EWS consistently improves robustness when combined with these approaches. To highlight the flexibility of our approach, we combine EWS also with popular adversarial training methods resulting in improved adversarial robustness.