Stable Invariant Models via Koopman Spectra

TL;DR

This paper introduces Stable Invariant Models (SIMs), a new class of deep models based on Koopman spectra that generalize deep equilibrium models (DEQs) by allowing convergence to invariant sets, with promising empirical results.

Contribution

The paper proposes SIMs, extending DEQs by using Koopman operator spectra to model stable dynamics converging to invariant sets, and provides an implementable learning method.

Findings

SIMs achieve comparable or better performance than DEQs.

SIMs can be learned similarly to feedforward models.

Empirical results demonstrate the effectiveness of SIMs in various tasks.

Abstract

Weight-tied models have attracted attention in the modern development of neural networks. The deep equilibrium model (DEQ) represents infinitely deep neural networks with weight-tying, and recent studies have shown the potential of this type of approach. DEQs are needed to iteratively solve root-finding problems in training and are built on the assumption that the underlying dynamics determined by the models converge to a fixed point. In this paper, we present the stable invariant model (SIM), a new class of deep models that in principle approximates DEQs under stability and extends the dynamics to more general ones converging to an invariant set (not restricted in a fixed point). The key ingredient in deriving SIMs is a representation of the dynamics with the spectra of the Koopman and Perron--Frobenius operators. This perspective approximately reveals stable dynamics with DEQs and then derives two variants of SIMs. We also propose an implementation of SIMs that can be learned in the same way as feedforward models. We illustrate the empirical performance of SIMs with experiments and demonstrate that SIMs achieve comparative or superior performance against DEQs in several learning tasks.