Consistency Deep Equilibrium Models

TL;DR

This paper introduces C-DEQ, a framework that accelerates Deep Equilibrium Models by using consistency distillation to enable few-step inference with maintained accuracy, significantly reducing inference latency.

Contribution

The paper proposes C-DEQ, a novel method that leverages consistency distillation and ODE-based interpretation to enable fast, accurate inference in DEQs with fewer steps.

Findings

Achieves 2-20× accuracy improvements over implicit DEQs with few inference steps.

Enables flexible trade-offs between computation and performance.

Demonstrates effectiveness across various domain tasks.

Abstract

Deep Equilibrium Models (DEQs) have emerged as a powerful paradigm in deep learning, offering the ability to model infinite-depth networks with constant memory usage. However, DEQs incur significant inference latency due to the iterative nature of fixed-point solvers. In this work, we introduce the Consistency Deep Equilibrium Model (C-DEQ), a novel framework that leverages consistency distillation to accelerate DEQ inference. We cast the DEQ iterative inference process as evolution along a fixed ODE trajectory toward the equilibrium. Along this trajectory, we train C-DEQs to consistently map intermediate states directly to the fixed point, enabling few-step inference while preserving the performance of the teacher DEQ. At the same time, it facilitates multi-step evaluation to flexibly trade computation for performance gains. Extensive experiments across various domain tasks demonstrate that C-DEQs achieves consistent 2-20$\times$ accuracy improvements over implicit DEQs under the same few-step inference budget.