Improved optimization strategies for deep Multi-Task Networks

TL;DR

This paper proposes an alternative optimization strategy for multi-task learning that involves alternating independent gradient steps for each task, improving performance especially for tasks of different natures, with a trade-off in computational time.

Contribution

It introduces a novel alternating gradient descent method combined with random task grouping to enhance multi-task learning optimization.

Findings

Better performance on visual MTL datasets compared to traditional methods.

Effective handling of tasks with different natures.

Trade-off between optimization benefits and computational efficiency via random grouping.

Abstract

In Multi-Task Learning (MTL), it is a common practice to train multi-task networks by optimizing an objective function, which is a weighted average of the task-specific objective functions. Although the computational advantages of this strategy are clear, the complexity of the resulting loss landscape has not been studied in the literature. Arguably, its optimization may be more difficult than a separate optimization of the constituting task-specific objectives. In this work, we investigate the benefits of such an alternative, by alternating independent gradient descent steps on the different task-specific objective functions and we formulate a novel way to combine this approach with state-of-the-art optimizers. As the separation of task-specific objectives comes at the cost of increased computational time, we propose a random task grouping as a trade-off between better optimization and computational efficiency. Experimental results over three well-known visual MTL datasets show better overall absolute performance on losses and standard metrics compared to an averaged objective function and other state-of-the-art MTL methods. In particular, our method shows the most benefits when dealing with tasks of different nature and it enables a wider exploration of the shared parameter space. We also show that our random grouping strategy allows to trade-off between these benefits and computational efficiency.