Revisiting Implicit Models: Sparsity Trade-offs Capability in Weight-tied Model for Vision Tasks

TL;DR

This paper revisits implicit models, particularly weight-tied models, demonstrating their superior efficiency and stability over DEQs in vision tasks, and explores sparsity techniques to enhance their capacity.

Contribution

It reveals the effectiveness of weight-tied models for vision tasks, compares them with DEQs, and proposes sparsity-based methods to improve their capacity and performance.

Findings

Weight-tied models outperform DEQs in efficiency and stability.

Sparsity masks can enhance model capacity.

Guidelines for designing depth, width, and sparsity in weight-tied models.

Abstract

Implicit models such as Deep Equilibrium Models (DEQs) have garnered significant attention in the community for their ability to train infinite layer models with elegant solution-finding procedures and constant memory footprint. However, despite several attempts, these methods are heavily constrained by model inefficiency and optimization instability. Furthermore, fair benchmarking across relevant methods for vision tasks is missing. In this work, we revisit the line of implicit models and trace them back to the original weight-tied models. Surprisingly, we observe that weight-tied models are more effective, stable, as well as efficient on vision tasks, compared to the DEQ variants. Through the lens of these simple-yet-clean weight-tied models, we further study the fundamental limits in the model capacity of such models and propose the use of distinct sparse masks to improve the model capacity. Finally, for practitioners, we offer design guidelines regarding the depth, width, and sparsity selection for weight-tied models, and demonstrate the generalizability of our insights to other learning paradigms.