Stability and Performance Analysis of Discrete-Time ReLU Recurrent Neural Networks

TL;DR

This paper develops new stability and performance conditions for ReLU-based recurrent neural networks by combining Lyapunov and dissipativity theories with quadratic constraints, providing a framework for analyzing RNN stability.

Contribution

It introduces a novel approach using quadratic constraints and lifted representations to analyze stability and performance of ReLU RNNs, expanding theoretical understanding.

Findings

Derived sufficient stability conditions for ReLU RNNs.

Showed the effectiveness of quadratic constraints in stability analysis.

Analyzed the impact of lifting horizon on stability and performance.

Abstract

This paper presents sufficient conditions for the stability and $\ell_2$-gain performance of recurrent neural networks (RNNs) with ReLU activation functions. These conditions are derived by combining Lyapunov/dissipativity theory with Quadratic Constraints (QCs) satisfied by repeated ReLUs. We write a general class of QCs for repeated RELUs using known properties for the scalar ReLU. Our stability and performance condition uses these QCs along with a "lifted" representation for the ReLU RNN. We show that the positive homogeneity property satisfied by a scalar ReLU does not expand the class of QCs for the repeated ReLU. We present examples to demonstrate the stability / performance condition and study the effect of the lifting horizon.