Deep Equilibrium Optical Flow Estimation

TL;DR

This paper introduces deep equilibrium (DEQ) flow estimators for optical flow that directly solve for stable flow as a fixed point, reducing memory use and improving performance over recurrent models.

Contribution

The authors propose a novel DEQ-based approach for optical flow estimation that is model-agnostic, memory-efficient, and stabilizes the fixed point solution with a new correction scheme.

Findings

Achieves better accuracy than SOTA methods on Sintel and KITTI datasets.

Uses 4-6 times less training memory than recurrent models.

Faster computation with fixed-point reuse and inexact gradients.

Abstract

Many recent state-of-the-art (SOTA) optical flow models use finite-step recurrent update operations to emulate traditional algorithms by encouraging iterative refinements toward a stable flow estimation. However, these RNNs impose large computation and memory overheads, and are not directly trained to model such stable estimation. They can converge poorly and thereby suffer from performance degradation. To combat these drawbacks, we propose deep equilibrium (DEQ) flow estimators, an approach that directly solves for the flow as the infinite-level fixed point of an implicit layer (using any black-box solver), and differentiates through this fixed point analytically (thus requiring $O(1)$ training memory). This implicit-depth approach is not predicated on any specific model, and thus can be applied to a wide range of SOTA flow estimation model designs. The use of these DEQ flow estimators allows us to compute the flow faster using, e.g., fixed-point reuse and inexact gradients, consumes $4\sim6\times$ times less training memory than the recurrent counterpart, and achieves better results with the same computation budget. In addition, we propose a novel, sparse fixed-point correction scheme to stabilize our DEQ flow estimators, which addresses a longstanding challenge for DEQ models in general. We test our approach in various realistic settings and show that it improves SOTA methods on Sintel and KITTI datasets with substantially better computational and memory efficiency.