Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference

TL;DR

This paper introduces Robust Dynamic Inference Networks (RDI-Nets), which adaptively select inference layers for each input, enabling simultaneous improvements in accuracy, robustness, and efficiency in deep networks.

Contribution

The paper proposes RDI-Nets, a novel multi-exit network architecture that co-optimizes accuracy, robustness, and efficiency through input-adaptive inference, addressing the accuracy-robustness trade-off.

Findings

RDI-Nets improve accuracy and robustness over baseline models.

Achieve over 30% computational savings with RDI-Nets.

Enable flexible adversarial defenses via multi-loss adaptivity.

Abstract

Deep networks were recently suggested to face the odds between accuracy (on clean natural images) and robustness (on adversarially perturbed images) (Tsipras et al., 2019). Such a dilemma is shown to be rooted in the inherently higher sample complexity (Schmidt et al., 2018) and/or model capacity (Nakkiran, 2019), for learning a high-accuracy and robust classifier. In view of that, give a classification task, growing the model capacity appears to help draw a win-win between accuracy and robustness, yet at the expense of model size and latency, therefore posing challenges for resource-constrained applications. Is it possible to co-design model accuracy, robustness and efficiency to achieve their triple wins? This paper studies multi-exit networks associated with input-adaptive efficient inference, showing their strong promise in achieving a "sweet point" in cooptimizing model accuracy, robustness and efficiency. Our proposed solution, dubbed Robust Dynamic Inference Networks (RDI-Nets), allows for each input (either clean or adversarial) to adaptively choose one of the multiple output layers (early branches or the final one) to output its prediction. That multi-loss adaptivity adds new variations and flexibility to adversarial attacks and defenses, on which we present a systematical investigation. We show experimentally that by equipping existing backbones with such robust adaptive inference, the resulting RDI-Nets can achieve better accuracy and robustness, yet with over 30% computational savings, compared to the defended original models.