DenVisCoM: Dense Vision Correspondence Mamba for Efficient and Real-time Optical Flow and Stereo Estimation

TL;DR

DenVisCoM introduces a hybrid architecture with a novel Mamba block and Transformer attention for accurate, real-time optical flow and stereo disparity estimation, effectively balancing accuracy, speed, and memory usage.

Contribution

The paper presents a new hybrid architecture and Mamba block that jointly estimates optical flow and disparity in real time, improving efficiency and accuracy over existing methods.

Findings

Achieves accurate optical flow and disparity estimation in real time.

Balances accuracy, speed, and memory footprint effectively.

Provides extensive benchmark analysis demonstrating superior performance.

Abstract

In this work, we propose a novel Mamba block DenVisCoM, as well as a novel hybrid architecture specifically tailored for accurate and real-time estimation of optical flow and disparity estimation. Given that such multi-view geometry and motion tasks are fundamentally related, we propose a unified architecture to tackle them jointly. Specifically, the proposed hybrid architecture is based on DenVisCoM and a Transformer-based attention block that efficiently addresses real-time inference, memory footprint, and accuracy at the same time for joint estimation of motion and 3D dense perception tasks. We extensively analyze the benchmark trade-off of accuracy and real-time processing on a large number of datasets. Our experimental results and related analysis suggest that our proposed model can accurately estimate optical flow and disparity estimation in real time. All models and associated code are available at https://github.com/vimstereo/DenVisCoM.