Efficient Pareto Manifold Learning with Low-Rank Structure

TL;DR

This paper introduces a scalable method for multi-task learning that efficiently approximates the Pareto front by integrating a main network with low-rank matrices, reducing parameters and improving performance.

Contribution

The paper proposes a novel low-rank structured approach for Pareto manifold learning that addresses scalability issues in multi-task learning.

Findings

Outperforms state-of-the-art baselines on large-task datasets

Reduces model parameters significantly

Enhances shared feature extraction

Abstract

Multi-task learning, which optimizes performance across multiple tasks, is inherently a multi-objective optimization problem. Various algorithms are developed to provide discrete trade-off solutions on the Pareto front. Recently, continuous Pareto front approximations using a linear combination of base networks have emerged as a compelling strategy. However, it suffers from scalability issues when the number of tasks is large. To address this issue, we propose a novel approach that integrates a main network with several low-rank matrices to efficiently learn the Pareto manifold. It significantly reduces the number of parameters and facilitates the extraction of shared features. We also introduce orthogonal regularization to further bolster performance. Extensive experimental results demonstrate that the proposed approach outperforms state-of-the-art baselines, especially on datasets with a large number of tasks.