Optimizing Evaluation Metrics for Multi-Task Learning via the Alternating Direction Method of Multipliers

TL;DR

This paper introduces a novel optimization approach for multi-task learning that directly maximizes evaluation metrics like F-score and ROC AUC, using an alternating direction method of multipliers to handle non-smooth objectives.

Contribution

It presents a new formulation combining regularization and structured hinge losses, along with an efficient optimization algorithm for directly optimizing evaluation metrics in MTL.

Findings

Outperforms baseline methods on various MTL tasks.

Efficient primal-dual algorithm for non-smooth optimization.

Applicable to a large family of MTL problems.

Abstract

Multi-task learning (MTL) aims to improve the generalization performance of multiple tasks by exploiting the shared factors among them. Various metrics (e.g., F-score, Area Under the ROC Curve) are used to evaluate the performances of MTL methods. Most existing MTL methods try to minimize either the misclassified errors for classification or the mean squared errors for regression. In this paper, we propose a method to directly optimize the evaluation metrics for a large family of MTL problems. The formulation of MTL that directly optimizes evaluation metrics is the combination of two parts: (1) a regularizer defined on the weight matrix over all tasks, in order to capture the relatedness of these tasks; (2) a sum of multiple structured hinge losses, each corresponding to a surrogate of some evaluation metric on one task. This formulation is challenging in optimization because both of its parts are non-smooth. To tackle this issue, we propose a novel optimization procedure based on the alternating direction scheme of multipliers, where we decompose the whole optimization problem into a sub-problem corresponding to the regularizer and another sub-problem corresponding to the structured hinge losses. For a large family of MTL problems, the first sub-problem has closed-form solutions. To solve the second sub-problem, we propose an efficient primal-dual algorithm via coordinate ascent. Extensive evaluation results demonstrate that, in a large family of MTL problems, the proposed MTL method of directly optimization evaluation metrics has superior performance gains against the corresponding baseline methods.