Quantifying Task Priority for Multi-Task Optimization

TL;DR

This paper introduces a novel approach to multi-task learning by quantifying task priority through connection strength, leading to improved Pareto optimal solutions and better overall performance compared to previous gradient-based methods.

Contribution

The paper proposes a new method that learns task priority via connection strength, enabling more effective gradient modification for multi-task optimization.

Findings

Enhanced multi-task performance over previous methods

Effective identification of task-specific parameter contributions

Achieved better Pareto optimal solutions

Abstract

The goal of multi-task learning is to learn diverse tasks within a single unified network. As each task has its own unique objective function, conflicts emerge during training, resulting in negative transfer among them. Earlier research identified these conflicting gradients in shared parameters between tasks and attempted to realign them in the same direction. However, we prove that such optimization strategies lead to sub-optimal Pareto solutions due to their inability to accurately determine the individual contributions of each parameter across various tasks. In this paper, we propose the concept of task priority to evaluate parameter contributions across different tasks. To learn task priority, we identify the type of connections related to links between parameters influenced by task-specific losses during backpropagation. The strength of connections is gauged by the magnitude of parameters to determine task priority. Based on these, we present a new method named connection strength-based optimization for multi-task learning which consists of two phases. The first phase learns the task priority within the network, while the second phase modifies the gradients while upholding this priority. This ultimately leads to finding new Pareto optimal solutions for multiple tasks. Through extensive experiments, we show that our approach greatly enhances multi-task performance in comparison to earlier gradient manipulation methods.