GDOD: Effective Gradient Descent using Orthogonal Decomposition for Multi-Task Learning

TL;DR

GDOD introduces an orthogonal decomposition-based optimization method for multi-task learning, effectively managing gradient conflicts and improving performance across multiple datasets.

Contribution

The paper proposes GDOD, a novel gradient manipulation technique using orthogonal basis decomposition to enhance multi-task learning optimization.

Findings

GDOD significantly outperforms existing MTL models in AUC and Logloss.

GDOD effectively decomposes gradients into shared and conflicting components.

Theoretical convergence of GDOD is proven under convex and non-convex conditions.

Abstract

Multi-task learning (MTL) aims at solving multiple related tasks simultaneously and has experienced rapid growth in recent years. However, MTL models often suffer from performance degeneration with negative transfer due to learning several tasks simultaneously. Some related work attributed the source of the problem is the conflicting gradients. In this case, it is needed to select useful gradient updates for all tasks carefully. To this end, we propose a novel optimization approach for MTL, named GDOD, which manipulates gradients of each task using an orthogonal basis decomposed from the span of all task gradients. GDOD decomposes gradients into task-shared and task-conflict components explicitly and adopts a general update rule for avoiding interference across all task gradients. This allows guiding the update directions depending on the task-shared components. Moreover, we prove the convergence of GDOD theoretically under both convex and non-convex assumptions. Experiment results on several multi-task datasets not only demonstrate the significant improvement of GDOD performed to existing MTL models but also prove that our algorithm outperforms state-of-the-art optimization methods in terms of AUC and Logloss metrics.