Hybrid Transformer Based Feature Fusion for Self-Supervised Monocular Depth Estimation

TL;DR

This paper introduces a hybrid model combining convolutional encoders and a transformer encoder for self-supervised monocular depth estimation, leveraging global and local features to improve accuracy.

Contribution

The novel fusion of convolutional and transformer-based features at multiple scales enhances depth prediction without supervised data.

Findings

Achieves state-of-the-art performance on standard benchmarks.

Effectively combines local and global features for depth estimation.

Outperforms ResNet-based architectures in self-supervised settings.

Abstract

With an unprecedented increase in the number of agents and systems that aim to navigate the real world using visual cues and the rising impetus for 3D Vision Models, the importance of depth estimation is hard to understate. While supervised methods remain the gold standard in the domain, the copious amount of paired stereo data required to train such models makes them impractical. Most State of the Art (SOTA) works in the self-supervised and unsupervised domain employ a ResNet-based encoder architecture to predict disparity maps from a given input image which are eventually used alongside a camera pose estimator to predict depth without direct supervision. The fully convolutional nature of ResNets makes them susceptible to capturing per-pixel local information only, which is suboptimal for depth prediction. Our key insight for doing away with this bottleneck is to use Vision Transformers, which employ self-attention to capture the global contextual information present in an input image. Our model fuses per-pixel local information learned using two fully convolutional depth encoders with global contextual information learned by a transformer encoder at different scales. It does so using a mask-guided multi-stream convolution in the feature space to achieve state-of-the-art performance on most standard benchmarks.