Adversarial Training of Two-Layer Polynomial and ReLU Activation Networks via Convex Optimization

TL;DR

This paper introduces a convex optimization approach for adversarial training of two-layer neural networks with polynomial and ReLU activations, improving robustness and scalability for large datasets.

Contribution

It develops a convex SDP formulation for adversarial training that matches nonconvex solutions and demonstrates practical, scalable implementations for large-scale models.

Findings

Convex SDP achieves the same optimality as nonconvex training.

Robust test accuracy improves against $$ attacks.

Re-trained ResNet-18 layers show higher robustness than sharpness-aware methods.

Abstract

Training neural networks which are robust to adversarial attacks remains an important problem in deep learning, especially as heavily overparameterized models are adopted in safety-critical settings. Drawing from recent work which reformulates the training problems for two-layer ReLU and polynomial activation networks as convex programs, we devise a convex semidefinite program (SDP) for adversarial training of two-layer polynomial activation networks and prove that the convex SDP achieves the same globally optimal solution as its nonconvex counterpart. The convex SDP is observed to improve robust test accuracy against $\ell_\infty$ attacks relative to the original convex training formulation on multiple datasets. Additionally, we present scalable implementations of adversarial training for two-layer polynomial and ReLU networks which are compatible with standard machine learning libraries and GPU acceleration. Leveraging these implementations, we retrain the final two fully connected layers of a Pre-Activation ResNet-18 model on the CIFAR-10 dataset with both polynomial and ReLU activations. The two `robustified' models achieve significantly higher robust test accuracies against $\ell_\infty$ attacks than a Pre-Activation ResNet-18 model trained with sharpness-aware minimization, demonstrating the practical utility of convex adversarial training on large-scale problems.