Learning to Relate Depth and Semantics for Unsupervised Domain Adaptation

TL;DR

This paper introduces a novel neural network architecture with a Cross-Task Relation Layer and an iterative self-learning scheme to enhance unsupervised domain adaptation for semantic segmentation and depth estimation by effectively encoding their relationships.

Contribution

It proposes a new Cross-Task Relation Layer and an iterative self-learning training scheme to improve multi-task learning in unsupervised domain adaptation.

Findings

Improved performance on semantic segmentation and depth estimation tasks.

The CTRL effectively encodes task dependencies, boosting accuracy.

ISL further enhances segmentation results in the target domain.

Abstract

We present an approach for encoding visual task relationships to improve model performance in an Unsupervised Domain Adaptation (UDA) setting. Semantic segmentation and monocular depth estimation are shown to be complementary tasks; in a multi-task learning setting, a proper encoding of their relationships can further improve performance on both tasks. Motivated by this observation, we propose a novel Cross-Task Relation Layer (CTRL), which encodes task dependencies between the semantic and depth predictions. To capture the cross-task relationships, we propose a neural network architecture that contains task-specific and cross-task refinement heads. Furthermore, we propose an Iterative Self-Learning (ISL) training scheme, which exploits semantic pseudo-labels to provide extra supervision on the target domain. We experimentally observe improvements in both tasks' performance because the complementary information present in these tasks is better captured. Specifically, we show that: (1) our approach improves performance on all tasks when they are complementary and mutually dependent; (2) the CTRL helps to improve both semantic segmentation and depth estimation tasks performance in the challenging UDA setting; (3) the proposed ISL training scheme further improves the semantic segmentation performance. The implementation is available at https://github.com/susaha/ctrl-uda.