Semi-supervised Multi-task Learning for Semantics and Depth

TL;DR

This paper introduces Semi-supervised Multi-task Learning (SemiMTL), a novel approach that leverages unlabeled data and domain-aware discriminators to improve semantic segmentation and depth estimation across datasets with partial annotations.

Contribution

The paper proposes a semi-supervised MTL framework with adversarial training and domain-aware discriminators to enable learning from datasets with incomplete annotations for multiple tasks.

Findings

Effective across street view and remote sensing benchmarks.

Improves performance with partial annotations.

Mitigates domain discrepancy among datasets.

Abstract

Multi-Task Learning (MTL) aims to enhance the model generalization by sharing representations between related tasks for better performance. Typical MTL methods are jointly trained with the complete multitude of ground-truths for all tasks simultaneously. However, one single dataset may not contain the annotations for each task of interest. To address this issue, we propose the Semi-supervised Multi-Task Learning (SemiMTL) method to leverage the available supervisory signals from different datasets, particularly for semantic segmentation and depth estimation tasks. To this end, we design an adversarial learning scheme in our semi-supervised training by leveraging unlabeled data to optimize all the task branches simultaneously and accomplish all tasks across datasets with partial annotations. We further present a domain-aware discriminator structure with various alignment formulations to mitigate the domain discrepancy issue among datasets. Finally, we demonstrate the effectiveness of the proposed method to learn across different datasets on challenging street view and remote sensing benchmarks.