Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspective

TL;DR

This paper analyzes the representation of Deep Equilibrium Models (DEQ) using Neural Collapse phenomena, revealing their geometric properties and advantages over explicit neural networks, especially in imbalanced data scenarios.

Contribution

It provides the first theoretical analysis of DEQ representations via Neural Collapse, demonstrating their geometric structure and robustness in imbalanced conditions.

Findings

Neural Collapse exists in DEQ under balanced conditions.

In imbalanced settings, DEQ shows advantages like feature convergence to simplex vertices.

DEQ demonstrates self-duality and superior handling of imbalanced datasets.

Abstract

Deep Equilibrium Model (DEQ), which serves as a typical implicit neural network, emphasizes their memory efficiency and competitive performance compared to explicit neural networks. However, there has been relatively limited theoretical analysis on the representation of DEQ. In this paper, we utilize the Neural Collapse ($\mathcal{NC}$) as a tool to systematically analyze the representation of DEQ under both balanced and imbalanced conditions. $\mathcal{NC}$ is an interesting phenomenon in the neural network training process that characterizes the geometry of class features and classifier weights. While extensively studied in traditional explicit neural networks, the $\mathcal{NC}$ phenomenon has not received substantial attention in the context of implicit neural networks. We theoretically show that $\mathcal{NC}$ exists in DEQ under balanced conditions. Moreover, in imbalanced settings, despite the presence of minority collapse, DEQ demonstrated advantages over explicit neural networks. These advantages include the convergence of extracted features to the vertices of a simplex equiangular tight frame and self-duality properties under mild conditions, highlighting DEQ's superiority in handling imbalanced datasets. Finally, we validate our theoretical analyses through experiments in both balanced and imbalanced scenarios.