A Complete Set of Quadratic Constraints for Repeated ReLU and Generalizations

TL;DR

This paper develops a comprehensive set of quadratic constraints for repeated ReLU activations, enabling tighter stability and performance bounds for ReLU neural networks through semidefinite programming.

Contribution

It introduces a complete set of quadratic constraints for repeated ReLU and related functions, improving the tightness of bounds compared to previous methods.

Findings

Complete QCs bound repeated ReLU as tightly as possible.

Incremental QCs can lead to less conservative Lipschitz bounds.

QCs improve stability analysis for recurrent neural networks.

Abstract

This paper derives a complete set of quadratic constraints (QCs) for the repeated ReLU. The complete set of QCs is described by a collection of matrix copositivity conditions. We also show that only two functions satisfy all QCs in our complete set: the repeated ReLU and flipped ReLU. Thus our complete set of QCs bounds the repeated ReLU as tight as possible up to the sign invariance inherent in quadratic forms. We derive a similar complete set of incremental QCs for repeated ReLU, which can potentially lead to less conservative Lipschitz bounds for ReLU networks than the standard LipSDP approach. The basic constructions are also used to derive the complete sets of QCs for other piecewise linear activation functions such as leaky ReLU, MaxMin, and HouseHolder. Finally, we illustrate the use of the complete set of QCs to assess stability and performance for recurrent neural networks with ReLU activation functions. We rely on a standard copositivity relaxation to formulate the stability/performance condition as a semidefinite program. Simple examples are provided to illustrate that the complete sets of QCs and incremental QCs can yield less conservative bounds than existing sets.