On Steering Multi-Annotations per Sample for Multi-Task Learning

TL;DR

This paper introduces stochastic task allocation methods, STA and ISTA, to improve multi-task learning by randomly assigning tasks to samples, outperforming existing gradient modification techniques.

Contribution

The paper proposes novel stochastic task allocation mechanisms, STA and ISTA, for better multi-task learning without relying on subjective task relationship assumptions.

Findings

STA and ISTA outperform state-of-the-art methods on multiple datasets.

Both methods improve multi-task learning performance.

ISTA iteratively allocates all tasks, enhancing learning efficiency.

Abstract

The study of multi-task learning has drawn great attention from the community. Despite the remarkable progress, the challenge of optimally learning different tasks simultaneously remains to be explored. Previous works attempt to modify the gradients from different tasks. Yet these methods give a subjective assumption of the relationship between tasks, and the modified gradient may be less accurate. In this paper, we introduce Stochastic Task Allocation~(STA), a mechanism that addresses this issue by a task allocation approach, in which each sample is randomly allocated a subset of tasks. For further progress, we propose Interleaved Stochastic Task Allocation~(ISTA) to iteratively allocate all tasks to each example during several consecutive iterations. We evaluate STA and ISTA on various datasets and applications: NYUv2, Cityscapes, and COCO for scene understanding and instance segmentation. Our experiments show both STA and ISTA outperform current state-of-the-art methods. The code will be available.