HiMODE: A Hybrid Monocular Omnidirectional Depth Estimation Model

TL;DR

HiMODE is a novel hybrid CNN-Transformer model for monocular omnidirectional depth estimation, effectively capturing details and reducing distortion with state-of-the-art results on multiple datasets.

Contribution

The paper introduces HiMODE, a hybrid CNN-Transformer architecture with novel modules for improved 360° depth estimation from monocular images.

Findings

Achieves state-of-the-art performance on Stanford3D, Matterport3D, and SunCG datasets.

Effectively captures small object details and reduces distortion.

Demonstrates the importance of each module through ablation studies.

Abstract

Monocular omnidirectional depth estimation is receiving considerable research attention due to its broad applications for sensing 360{\deg} surroundings. Existing approaches in this field suffer from limitations in recovering small object details and data lost during the ground-truth depth map acquisition. In this paper, a novel monocular omnidirectional depth estimation model, namely HiMODE is proposed based on a hybrid CNN+Transformer (encoder-decoder) architecture whose modules are efficiently designed to mitigate distortion and computational cost, without performance degradation. Firstly, we design a feature pyramid network based on the HNet block to extract high-resolution features near the edges. The performance is further improved, benefiting from a self and cross attention layer and spatial/temporal patches in the Transformer encoder and decoder, respectively. Besides, a spatial residual block is employed to reduce the number of parameters. By jointly passing the deep features extracted from an input image at each backbone block, along with the raw depth maps predicted by the transformer encoder-decoder, through a context adjustment layer, our model can produce resulting depth maps with better visual quality than the ground-truth. Comprehensive ablation studies demonstrate the significance of each individual module. Extensive experiments conducted on three datasets; Stanford3D, Matterport3D, and SunCG, demonstrate that HiMODE can achieve state-of-the-art performance for 360{\deg} monocular depth estimation.