A Novel Plug-and-Play Approach for Adversarially Robust Generalization

TL;DR

This paper introduces a comprehensive framework for adversarially robust machine learning, providing exact solutions for various models and deriving new generalization bounds, with practical experiments demonstrating efficiency.

Contribution

It offers a novel plug-and-play approach with exact solutions and generalization bounds for adversarial robustness across diverse ML problems.

Findings

Exact solutions for multiple loss functions under norm constraints.

New bounds on adversarial Rademacher complexity for various models.

Empirical validation on real-world datasets with low computational overhead.

Abstract

In this work, we propose a robust framework that employs adversarially robust training to safeguard the ML models against perturbed testing data. Our contributions can be seen from both computational and statistical perspectives. Firstly, from a computational/optimization point of view, we derive the ready-to-use exact solution for several widely used loss functions with a variety of norm constraints on adversarial perturbation for various supervised and unsupervised ML problems, including regression, classification, two-layer neural networks, graphical models, and matrix completion. The solutions are either in closed-form, or an easily tractable optimization problem such as 1-D convex optimization, semidefinite programming, difference of convex programming or a sorting-based algorithm. Secondly, from statistical/generalization viewpoint, using some of these results, we derive novel bounds of the adversarial Rademacher complexity for various problems, which entails new generalization bounds. Thirdly, we perform some sanity-check experiments on real-world datasets for supervised problems such as regression and classification, as well as for unsupervised problems such as matrix completion and learning graphical models, with very little computational overhead.