AutoScale: Linear Scalarization Guided by Multi-Task Optimization Metrics

TL;DR

AutoScale leverages multi-task optimization metrics to guide linear scalarization weights, enabling efficient and effective multi-task learning without exhaustive hyperparameter tuning.

Contribution

The paper introduces AutoScale, a novel framework that uses MTO metrics to automatically select scalarization weights, bridging the gap between scalarization and complex MTO methods.

Findings

AutoScale outperforms traditional methods across multiple datasets.

It achieves high efficiency without exhaustive weight search.

Well-chosen scalarization weights correlate with specific MTO metric trends.

Abstract

Recent multi-task learning studies suggest that linear scalarization, when using well-chosen fixed task weights, can achieve comparable to or even better performance than complex multi-task optimization (MTO) methods. It remains unclear why certain weights yield optimal performance and how to determine these weights without relying on exhaustive hyperparameter search. This paper establishes a direct connection between linear scalarization and MTO methods, revealing through extensive experiments that well-performing scalarization weights exhibit specific trends in key MTO metrics, such as high gradient magnitude similarity. Building on this insight, we introduce AutoScale, a simple yet effective two-phase framework that uses these MTO metrics to guide weight selection for linear scalarization, without expensive weight search. AutoScale consistently shows superior performance with high efficiency across diverse datasets including a new large-scale benchmark.