Delve into the Applicability of Advanced Optimizers for Multi-Task Learning

TL;DR

This paper investigates the limitations of advanced optimizers in Multi-Task Learning, revealing the marginal role of instant-derived gradients and proposing APT, a framework that enhances optimizer effectiveness through adaptive momentum and direction preservation.

Contribution

The paper introduces APT, a novel framework with adaptive momentum and direction preservation techniques, to improve the application of advanced optimizers in Multi-Task Learning.

Findings

APT consistently improves performance across four MTL datasets.

Advanced optimizers' effectiveness is limited by marginal gradient influence.

Muon inherently functions as a multi-task learner due to its orthogonalization process.

Abstract

Multi-Task Learning (MTL) is a foundational machine learning problem that has seen extensive development over the past decade. Recently, various optimization-based MTL approaches have been proposed to learn multiple tasks simultaneously by altering the optimization trajectory. Although these methods strive to de-conflict and re-balance tasks, we empirically identify that their effectiveness is often undermined by an overlooked factor when employing advanced optimizers: the instant-derived gradients play only a marginal role in the actual parameter updates. This discrepancy prevents MTL frameworks from fully releasing its power on learning dynamics. Furthermore, we observe that Muon-a recently emerged advanced optimizer-inherently functions as a multi-task learner, which underscores the critical importance of the gradients used for its orthogonalization. To address these issues, we propose APT (Applicability of advanced oPTimizers), a framework featuring a simple adaptive momentum mechanism designed to balance the strengths between advanced optimizers and MTL. Additionally, we introduce a light direction preservation method to facilitate Muon's orthogonalization. Extensive experiments across four mainstream MTL datasets demonstrate that APT consistently augments existing MTL approaches, yielding substantial performance improvements.