Semi-Supervised Semantic Depth Estimation using Symbiotic Transformer and NearFarMix Augmentation

TL;DR

This paper presents a semi-supervised approach for depth estimation that integrates semantics using a symbiotic transformer and introduces NearFarMix augmentation to improve robustness and generalization across indoor and outdoor datasets.

Contribution

It introduces the Depth Semantics Symbiosis module with a symbiotic transformer and a novel NearFarMix augmentation, enhancing depth-semantic mutual understanding and dataset invariance.

Findings

Outperforms existing methods on NYU-Depth-V2 and KITTI datasets.

Improves depth and semantic accuracy in diverse environments.

Reduces overfitting through NearFarMix augmentation.

Abstract

In computer vision, depth estimation is crucial for domains like robotics, autonomous vehicles, augmented reality, and virtual reality. Integrating semantics with depth enhances scene understanding through reciprocal information sharing. However, the scarcity of semantic information in datasets poses challenges. Existing convolutional approaches with limited local receptive fields hinder the full utilization of the symbiotic potential between depth and semantics. This paper introduces a dataset-invariant semi-supervised strategy to address the scarcity of semantic information. It proposes the Depth Semantics Symbiosis module, leveraging the Symbiotic Transformer for achieving comprehensive mutual awareness by information exchange within both local and global contexts. Additionally, a novel augmentation, NearFarMix is introduced to combat overfitting and compensate both depth-semantic tasks by strategically merging regions from two images, generating diverse and structurally consistent samples with enhanced control. Extensive experiments on NYU-Depth-V2 and KITTI datasets demonstrate the superiority of our proposed techniques in indoor and outdoor environments.