Improving Neural Network Robustness via Persistency of Excitation

TL;DR

This paper introduces a novel training approach inspired by control theory, specifically persistency of excitation, to enhance neural network robustness against adversarial attacks while maintaining accuracy.

Contribution

It establishes a theoretical link between gradient descent in neural networks and adaptive control, proposing a new learning rate schedule based on PoE to improve adversarial robustness.

Findings

Networks trained with PoE-inspired schedule show increased adversarial robustness.

The method maintains comparable clean accuracy to state-of-the-art models.

A practical technique for estimating Lipschitz constants improves applicability.

Abstract

Improving adversarial robustness of neural networks remains a major challenge. Fundamentally, training a neural network via gradient descent is a parameter estimation problem. In adaptive control, maintaining persistency of excitation (PoE) is integral to ensuring convergence of parameter estimates in dynamical systems to their true values. We show that parameter estimation with gradient descent can be modeled as a sampling of an adaptive linear time-varying continuous system. Leveraging this model, and with inspiration from Model-Reference Adaptive Control (MRAC), we prove a sufficient condition to constrain gradient descent updates to reference persistently excited trajectories converging to the true parameters. The sufficient condition is achieved when the learning rate is less than the inverse of the Lipschitz constant of the gradient of loss function. We provide an efficient technique for estimating the corresponding Lipschitz constant in practice using extreme value theory. Our experimental results in both standard and adversarial training illustrate that networks trained with the PoE-motivated learning rate schedule have similar clean accuracy but are significantly more robust to adversarial attacks than models trained using current state-of-the-art heuristics.