Multi-Task Label Discovery via Hierarchical Task Tokens for Partially Annotated Dense Predictions

TL;DR

This paper introduces a hierarchical task token approach to improve multi-task dense prediction with partial labels, enabling direct pixel-wise supervision and outperforming existing methods on multiple datasets.

Contribution

It proposes a novel hierarchical task token framework for discovering pixel-wise supervision signals, reducing reliance on heavy mapping networks and enhancing multi-task learning.

Findings

Significant performance improvements on NYUD-v2, Cityscapes, and PASCAL Context datasets.

Effective discovery of pseudo dense labels for multi-task learning.

Enhanced cross-task feature interactions through hierarchical task tokens.

Abstract

In recent years, simultaneous learning of multiple dense prediction tasks with partially annotated label data has emerged as an important research area. Previous works primarily focus on leveraging cross-task relations or conducting adversarial training for extra regularization, which achieve promising performance improvements, while still suffering from the lack of direct pixel-wise supervision and extra training of heavy mapping networks. To effectively tackle this challenge, we propose a novel approach to optimize a set of compact learnable hierarchical task tokens, including global and fine-grained ones, to discover consistent pixel-wise supervision signals in both feature and prediction levels. Specifically, the global task tokens are designed for effective cross-task feature interactions in a global context. Then, a group of fine-grained task-specific spatial tokens for each task is learned from the corresponding global task tokens. It is embedded to have dense interactions with each task-specific feature map. The learned global and local fine-grained task tokens are further used to discover pseudo task-specific dense labels at different levels of granularity, and they can be utilized to directly supervise the learning of the multi-task dense prediction framework. Extensive experimental results on challenging NYUD-v2, Cityscapes, and PASCAL Context datasets demonstrate significant improvements over existing state-of-the-art methods for partially annotated multi-task dense prediction.