Exploring Relational Context for Multi-Task Dense Prediction

TL;DR

This paper introduces an adaptive context refinement module for multi-task dense prediction that automatically selects the best cross-task contexts, leading to state-of-the-art results with low computational cost.

Contribution

It proposes the ATRC module that uses neural architecture search to optimize cross-task context selection in multi-task dense prediction models.

Findings

Achieves state-of-the-art results on NYUD-v2 and PASCAL-Context benchmarks.

Different task pairs benefit from different types of contextual information.

The ATRC module is computationally efficient and easily integrable.

Abstract

The timeline of computer vision research is marked with advances in learning and utilizing efficient contextual representations. Most of them, however, are targeted at improving model performance on a single downstream task. We consider a multi-task environment for dense prediction tasks, represented by a common backbone and independent task-specific heads. Our goal is to find the most efficient way to refine each task prediction by capturing cross-task contexts dependent on tasks' relations. We explore various attention-based contexts, such as global and local, in the multi-task setting and analyze their behavior when applied to refine each task independently. Empirical findings confirm that different source-target task pairs benefit from different context types. To automate the selection process, we propose an Adaptive Task-Relational Context (ATRC) module, which samples the pool of all available contexts for each task pair using neural architecture search and outputs the optimal configuration for deployment. Our method achieves state-of-the-art performance on two important multi-task benchmarks, namely NYUD-v2 and PASCAL-Context. The proposed ATRC has a low computational toll and can be used as a drop-in refinement module for any supervised multi-task architecture.