Semantics-Depth-Symbiosis: Deeply Coupled Semi-Supervised Learning of Semantics and Depth

TL;DR

This paper introduces a novel multi-task learning framework for semantic segmentation and depth estimation, utilizing a new attention module and data augmentation techniques to achieve state-of-the-art semi-supervised results on Cityscapes and ScanNet datasets.

Contribution

It proposes a Cross-Channel Attention Module for effective feature sharing and novel augmentation methods, enhancing semi-supervised joint learning of semantics and depth.

Findings

Achieved state-of-the-art results on Cityscapes and ScanNet datasets.

Demonstrated mutual performance improvements for semantic segmentation and depth estimation.

Introduced effective augmentation techniques for semi-supervised multi-task learning.

Abstract

Multi-task learning (MTL) paradigm focuses on jointly learning two or more tasks, aiming for significant improvement w.r.t model's generalizability, performance, and training/inference memory footprint. The aforementioned benefits become ever so indispensable in the case of joint training for vision-related {\bf dense} prediction tasks. In this work, we tackle the MTL problem of two dense tasks, i.e., semantic segmentation and depth estimation, and present a novel attention module called Cross-Channel Attention Module ({CCAM}), which facilitates effective feature sharing along each channel between the two tasks, leading to mutual performance gain with a negligible increase in trainable parameters. In a true symbiotic spirit, we then formulate a novel data augmentation for the semantic segmentation task using predicted depth called {AffineMix}, and a simple depth augmentation using predicted semantics called {ColorAug}. Finally, we validate the performance gain of the proposed method on the Cityscapes and ScanNet dataset, which helps us achieve state-of-the-art results for a semi-supervised joint model based on depth and semantic segmentation.