Hyper-parameter Tuning for Adversarially Robust Models

TL;DR

This paper investigates hyper-parameter tuning challenges for adversarially robust models, demonstrating the benefits of different tuning strategies and proposing cost-effective methods to improve tuning efficiency using multi-fidelity optimization.

Contribution

It provides an extensive experimental analysis of hyper-parameter tuning complexities and introduces a novel approach leveraging cheap adversarial training and multi-fidelity optimization to enhance efficiency.

Findings

Independent tuning of HPs during standard and adversarial training reduces errors significantly.

Using inexpensive adversarial training methods correlates well with high-quality estimations.

Multi-fidelity optimizer (taKG) improves HPT efficiency by up to 2.1 times.

Abstract

This work focuses on the problem of hyper-parameter tuning (HPT) for robust (i.e., adversarially trained) models, shedding light on the new challenges and opportunities arising during the HPT process for robust models. To this end, we conduct an extensive experimental study based on 3 popular deep models, in which we explore exhaustively 9 (discretized) HPs, 2 fidelity dimensions, and 2 attack bounds, for a total of 19208 configurations (corresponding to 50 thousand GPU hours). Through this study, we show that the complexity of the HPT problem is further exacerbated in adversarial settings due to the need to independently tune the HPs used during standard and adversarial training: succeeding in doing so (i.e., adopting different HP settings in both phases) can lead to a reduction of up to 80% and 43% of the error for clean and adversarial inputs, respectively. On the other hand, we also identify new opportunities to reduce the cost of HPT for robust models. Specifically, we propose to leverage cheap adversarial training methods to obtain inexpensive, yet highly correlated, estimations of the quality achievable using state-of-the-art methods. We show that, by exploiting this novel idea in conjunction with a recent multi-fidelity optimizer (taKG), the efficiency of the HPT process can be enhanced by up to 2.1x.