Global Convergence Rate of Deep Equilibrium Models with General Activations

TL;DR

This paper extends the analysis of Deep Equilibrium Models (DEQs) to general activation functions with bounded derivatives, proving global convergence rates similar to ReLU-based DEQs.

Contribution

It introduces a novel approach using Hermite polynomial expansion to analyze DEQs with non-homogeneous activations, establishing their convergence properties.

Findings

Global convergence rate holds for general bounded activation functions.

New techniques for analyzing non-homogeneous activations.

Development of a novel population Gram matrix and dual activation.

Abstract

In a recent paper, Ling et al. investigated the over-parametrized Deep Equilibrium Model (DEQ) with ReLU activation. They proved that the gradient descent converges to a globally optimal solution at a linear convergence rate for the quadratic loss function. This paper shows that this fact still holds for DEQs with any general activation that has bounded first and second derivatives. Since the new activation function is generally non-homogeneous, bounding the least eigenvalue of the Gram matrix of the equilibrium point is particularly challenging. To accomplish this task, we need to create a novel population Gram matrix and develop a new form of dual activation with Hermite polynomial expansion.