3D-Aware Multi-Task Learning with Cross-View Correlations for Dense Scene Understanding

TL;DR

This paper introduces a 3D-aware multi-task learning framework that incorporates cross-view correlations via a lightweight module, enhancing dense scene understanding tasks like segmentation and depth estimation.

Contribution

It proposes a novel Cross-view Module (CvM) that captures 3D geometric consistency across views, improving multi-task learning performance.

Findings

Improved accuracy on NYUv2 and PASCAL-Context datasets.

Effective integration of geometric consistency into existing MTL methods.

Applicable to both single and multi-view data.

Abstract

This paper addresses the challenge of training a single network to jointly perform multiple dense prediction tasks, such as segmentation and depth estimation, i.e., multi-task learning (MTL). Current approaches mainly capture cross-task relations in the 2D image space, often leading to unstructured features lacking 3D-awareness. We argue that 3D-awareness is vital for modeling cross-task correlations essential for comprehensive scene understanding. We propose to address this problem by integrating correlations across views, i.e., cost volume, as geometric consistency in the MTL network. Specifically, we introduce a lightweight Cross-view Module (CvM), shared across tasks, to exchange information across views and capture cross-view correlations, integrated with a feature from MTL encoder for multi-task predictions. This module is architecture-agnostic and can be applied to both single and multi-view data. Extensive results on NYUv2 and PASCAL-Context demonstrate that our method effectively injects geometric consistency into existing MTL methods to improve performance.