Non-Euclidean Monotone Operator Theory with Applications to Recurrent Neural Networks

TL;DR

This paper extends monotone operator theory to non-Euclidean spaces like  and , enabling new algorithms for neural network equilibrium computation with improved convergence.

Contribution

It introduces a novel non-Euclidean monotone operator framework and applies it to recurrent neural networks, enhancing convergence of equilibrium computation methods.

Findings

Applicable classic algorithms in non-Euclidean settings

Improved convergence rates in neural network equilibrium computation

Framework extends monotone operator theory beyond Euclidean spaces

Abstract

We provide a novel transcription of monotone operator theory to the non-Euclidean finite-dimensional spaces $\ell_1$ and $\ell_{\infty}$. We first establish properties of mappings which are monotone with respect to the non-Euclidean norms $\ell_1$ or $\ell_{\infty}$. In analogy with their Euclidean counterparts, mappings which are monotone with respect to a non-Euclidean norm are amenable to numerous algorithms for computing their zeros. We demonstrate that several classic iterative methods for computing zeros of monotone operators are directly applicable in the non-Euclidean framework. We present a case-study in the equilibrium computation of recurrent neural networks and demonstrate that casting the computation as a suitable operator splitting problem improves convergence rates.