Dissipative Deep Neural Dynamical Systems

TL;DR

This paper establishes conditions for dissipativity and stability in neural network-based dynamical systems, using classical analysis methods to interpret neural networks as affine maps and examining their energy and stability properties.

Contribution

It introduces a framework to analyze neural networks as dynamical systems, providing tools to assess their dissipativity, stability, and behavior through local linear operator analysis.

Findings

Neural networks can be characterized as dissipative systems with energy bounds.

Eigenvalue spectra vary with network parameters, affecting stability.

Analysis reveals how weight and bias configurations influence dynamical behavior.

Abstract

In this paper, we provide sufficient conditions for dissipativity and local asymptotic stability of discrete-time dynamical systems parametrized by deep neural networks. We leverage the representation of neural networks as pointwise affine maps, thus exposing their local linear operators and making them accessible to classical system analytic and design methods. This allows us to "crack open the black box" of the neural dynamical system's behavior by evaluating their dissipativity, and estimating their stationary points and state-space partitioning. We relate the norms of these local linear operators to the energy stored in the dissipative system with supply rates represented by their aggregate bias terms. Empirically, we analyze the variance in dynamical behavior and eigenvalue spectra of these local linear operators with varying weight factorizations, activation functions, bias terms, and depths.