Real-time self-adaptive deep stereo

TL;DR

This paper introduces MADNet, a lightweight deep stereo network, and MAD, an adaptation algorithm, enabling real-time self-adaptive disparity estimation across diverse environments without extensive retraining.

Contribution

The paper presents MADNet and MAD, the first real-time self-adaptive deep stereo system capable of maintaining accuracy across different domains.

Findings

Achieves real-time performance with self-adaptation.

Maintains high accuracy across heterogeneous datasets.

Introduces a modular training approach for efficiency.

Abstract

Deep convolutional neural networks trained end-to-end are the state-of-the-art methods to regress dense disparity maps from stereo pairs. These models, however, suffer from a notable decrease in accuracy when exposed to scenarios significantly different from the training set, e.g., real vs synthetic images, etc.). We argue that it is extremely unlikely to gather enough samples to achieve effective training/tuning in any target domain, thus making this setup impractical for many applications. Instead, we propose to perform unsupervised and continuous online adaptation of a deep stereo network, which allows for preserving its accuracy in any environment. However, this strategy is extremely computationally demanding and thus prevents real-time inference. We address this issue introducing a new lightweight, yet effective, deep stereo architecture, Modularly ADaptive Network (MADNet) and developing a Modular ADaptation (MAD) algorithm, which independently trains sub-portions of the network. By deploying MADNet together with MAD we introduce the first real-time self-adaptive deep stereo system enabling competitive performance on heterogeneous datasets.