DensePercept-NCSSD: Vision Mamba towards Real-time Dense Visual Perception with Non-Causal State Space Duality

TL;DR

DensePercept-NCSSD introduces a non-causal state space model that enables real-time, accurate dense visual perception including optical flow and disparity estimation, suitable for practical 3D perception applications.

Contribution

The paper presents a novel non-causal Mamba block-based model that improves inference speed and efficiency for dense perception tasks without sacrificing accuracy.

Findings

Achieves real-time dense optical flow and disparity estimation

Reduces GPU usage compared to existing methods

Validated in real-world scenarios for 3D perception

Abstract

In this work, we propose an accurate and real-time optical flow and disparity estimation model by fusing pairwise input images in the proposed non-causal selective state space for dense perception tasks. We propose a non-causal Mamba block-based model that is fast and efficient and aptly manages the constraints present in a real-time applications. Our proposed model reduces inference times while maintaining high accuracy and low GPU usage for optical flow and disparity map generation. The results and analysis, and validation in real-life scenario justify that our proposed model can be used for unified real-time and accurate 3D dense perception estimation tasks. The code, along with the models, can be found at https://github.com/vimstereo/DensePerceptNCSSD