Which Tasks Should Be Learned Together in Multi-task Learning?

TL;DR

This paper investigates how to optimally group tasks in multi-task learning to improve accuracy and efficiency, proposing a framework that balances task cooperation and competition for better performance.

Contribution

It introduces a novel framework for assigning tasks to neural networks based on their cooperation and competition, optimizing the trade-off between inference time and accuracy.

Findings

The framework improves accuracy over single large multi-task networks.

It reduces inference time compared to multiple single-task networks.

It effectively balances task cooperation and competition.

Abstract

Many computer vision applications require solving multiple tasks in real-time. A neural network can be trained to solve multiple tasks simultaneously using multi-task learning. This can save computation at inference time as only a single network needs to be evaluated. Unfortunately, this often leads to inferior overall performance as task objectives can compete, which consequently poses the question: which tasks should and should not be learned together in one network when employing multi-task learning? We study task cooperation and competition in several different learning settings and propose a framework for assigning tasks to a few neural networks such that cooperating tasks are computed by the same neural network, while competing tasks are computed by different networks. Our framework offers a time-accuracy trade-off and can produce better accuracy using less inference time than not only a single large multi-task neural network but also many single-task networks.