Lipschitz Multiscale Deep Equilibrium Models: A Theoretically Guaranteed and Accelerated Approach

TL;DR

This paper introduces Lipschitz multiscale DEQ, a deep equilibrium model with guaranteed fixed-point convergence that accelerates training and inference while maintaining competitive accuracy.

Contribution

It proposes a novel architecture restructuring for DEQs to ensure convergence, significantly reducing computational time with theoretical guarantees.

Findings

Achieved up to 4.75× speed-up in CIFAR-10 experiments.

Guaranteed fixed-point convergence for both forward and backward passes.

Minor accuracy drop at increased speed.

Abstract

Deep equilibrium models (DEQs) achieve infinitely deep network representations without stacking layers by exploring fixed points of layer transformations in neural networks. Such models constitute an innovative approach that achieves performance comparable to state-of-the-art methods in many large-scale numerical experiments, despite requiring significantly less memory. However, DEQs face the challenge of requiring vastly more computational time for training and inference than conventional methods, as they repeatedly perform fixed-point iterations with no convergence guarantee upon each input. Therefore, this study explored an approach to improve fixed-point convergence and consequently reduce computational time by restructuring the model architecture to guarantee fixed-point convergence. Our proposed approach for image classification, Lipschitz multiscale DEQ, has theoretically guaranteed fixed-point convergence for both forward and backward passes by hyperparameter adjustment, achieving up to a 4.75$\times$ speed-up in numerical experiments on CIFAR-10 at the cost of a minor drop in accuracy.