Integral Quadratic Constraints for Repeated ReLU

TL;DR

This paper introduces dynamic integral quadratic constraints (IQCs) for analyzing the stability and performance of recurrent neural networks with ReLU activations, offering less conservative bounds than previous static IQCs.

Contribution

The paper develops a new class of dynamic IQCs for repeated ReLU, expanding the analysis tools for RNN stability and performance, and demonstrating their advantages over existing IQCs.

Findings

Dynamic IQCs form a superset of static IQCs for ReLU.

The $\ell_2$-gain bounds are nonincreasing with the IQC horizon.

Proposed IQCs yield less conservative bounds in numerical examples.

Abstract

This paper presents a new dynamic integral quadratic constraint (IQC) for the repeated Rectified Linear Unit (ReLU). These dynamic IQCs can be used to analyze stability and induced $\ell_2$-gain performance of discrete-time, recurrent neural networks (RNNs) with ReLU activation functions. These analysis conditions can be incorporated into learning-based controller synthesis methods, which currently rely on static IQCs. We show that our proposed dynamic IQCs for repeated ReLU form a superset of the dynamic IQCs for repeated, slope-restricted nonlinearities. We also prove that the $\ell_2$-gain bounds are nonincreasing with respect to the horizon used in the dynamic IQC filter. A numerical example using a simple (academic) RNN shows that our proposed IQCs lead to less conservative bounds than existing IQCs.