Practical Convex Formulation of Robust One-hidden-layer Neural Network Training

TL;DR

This paper introduces a convex optimization approach for training robust one-hidden-layer neural networks efficiently, providing improved adversarial robustness and performance over existing methods like FGSM and PGD.

Contribution

It develops a scalable convex formulation for robust neural network training and proposes a stochastic approximation method that significantly reduces computational complexity.

Findings

The method achieves better adversarial robustness than existing techniques.

It provides an efficient convex optimization framework for training neural networks.

Experimental results show improved performance in binary classification and regression tasks.

Abstract

Recent work has shown that the training of a one-hidden-layer, scalar-output fully-connected ReLU neural network can be reformulated as a finite-dimensional convex program. Unfortunately, the scale of such a convex program grows exponentially in data size. In this work, we prove that a stochastic procedure with a linear complexity well approximates the exact formulation. Moreover, we derive a convex optimization approach to efficiently solve the "adversarial training" problem, which trains neural networks that are robust to adversarial input perturbations. Our method can be applied to binary classification and regression, and provides an alternative to the current adversarial training methods, such as Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD). We demonstrate in experiments that the proposed method achieves a noticeably better adversarial robustness and performance than the existing methods.