Sign-regularized Multi-task Learning

TL;DR

This paper introduces a novel multi-task learning framework that regularizes feature weight signs across tasks, addressing key challenges in optimizing and sharing task-related sign and magnitude patterns for improved generalization.

Contribution

It proposes a new sign-regularized multi-task learning model with an efficient optimization algorithm and theoretical guarantees, advancing how task correlations are utilized.

Findings

Effective regularization of feature weight signs across tasks.

Improved generalization performance demonstrated on multiple datasets.

Efficient optimization with theoretical convergence guarantees.

Abstract

Multi-task learning is a framework that enforces different learning tasks to share their knowledge to improve their generalization performance. It is a hot and active domain that strives to handle several core issues; particularly, which tasks are correlated and similar, and how to share the knowledge among correlated tasks. Existing works usually do not distinguish the polarity and magnitude of feature weights and commonly rely on linear correlation, due to three major technical challenges in: 1) optimizing the models that regularize feature weight polarity, 2) deciding whether to regularize sign or magnitude, 3) identifying which tasks should share their sign and/or magnitude patterns. To address them, this paper proposes a new multi-task learning framework that can regularize feature weight signs across tasks. We innovatively formulate it as a biconvex inequality constrained optimization with slacks and propose a new efficient algorithm for the optimization with theoretical guarantees on generalization performance and convergence. Extensive experiments on multiple datasets demonstrate the proposed methods' effectiveness, efficiency, and reasonableness of the regularized feature weighted patterns.