Contracting Neural Networks: Sharp LMI Conditions with Applications to Integral Control and Deep Learning

TL;DR

This paper derives precise LMI conditions for neural network contractivity, applying these to control design and enhancing implicit neural network expressivity for image classification.

Contribution

It introduces sharp LMI conditions for neural network contractivity and applies them to control design and neural network expressivity improvements.

Findings

Derived sharp LMI conditions for contractivity in neural networks.

Developed an LMI-based design for low-gain integral controllers.

Provided a parameterization of weight matrices guaranteeing contraction.

Abstract

This paper studies contractivity of firing-rate and Hopfield recurrent neural networks. We derive sharp LMI conditions on the synaptic matrices that characterize contractivity of both architectures, for activation functions that are either non-expansive or monotone non-expansive, in both continuous and discrete time. We establish structural relationships among these conditions, including connections to Schur diagonal stability and the recovery of optimal contraction rates for symmetric synaptic matrices. We demonstrate the utility of these results through two applications. First, we develop an LMI-based design procedure for low-gain integral controllers enabling reference tracking in contracting firing rate networks. Second, we provide an exact parameterization of weight matrices that guarantee contraction and use it to improve the expressivity of Implicit Neural Networks, achieving competitive performance on image classification benchmarks with fewer parameters.