Uniform Loss vs. Specialized Optimization: A Comparative Analysis in Multi-Task Learning

TL;DR

This paper compares specialized multi-task optimizers with uniform loss approaches, showing that fixed weights can often match the performance of SMTOs in complex multi-task learning scenarios.

Contribution

It provides an extensive empirical evaluation demonstrating that uniform loss can be competitive with SMTOs, challenging previous assumptions about the necessity of specialized optimizers.

Findings

SMTOs perform well compared to uniform loss.

Fixed weights can achieve competitive performance.

Uniform loss can perform similarly to SMTOs in some cases.

Abstract

Specialized Multi-Task Optimizers (SMTOs) balance task learning in Multi-Task Learning by addressing issues like conflicting gradients and differing gradient norms, which hinder equal-weighted task training. However, recent critiques suggest that equally weighted tasks can achieve competitive results compared to SMTOs, arguing that previous SMTO results were influenced by poor hyperparameter optimization and lack of regularization. In this work, we evaluate these claims through an extensive empirical evaluation of SMTOs, including some of the latest methods, on more complex multi-task problems to clarify this behavior. Our findings indicate that SMTOs perform well compared to uniform loss and that fixed weights can achieve competitive performance compared to SMTOs. Furthermore, we demonstrate why uniform loss perform similarly to SMTOs in some instances. The source code is available at https://github.com/Gabriel-SGama/UnitScal_vs_SMTOs.