Saliency-Regularized Deep Multi-Task Learning

TL;DR

This paper introduces a novel deep multitask learning framework that explicitly models task relations through gradient similarity and a new regularizer, improving generalization and interpretability.

Contribution

It proposes a method to learn explicit task relations via gradient similarity and a regularizer, addressing limitations of existing deep multitask learning models.

Findings

Improved generalization error bounds due to the regularizer.

Effective learning of task relations demonstrated on benchmarks.

Enhanced interpretability of task relation patterns.

Abstract

Multitask learning is a framework that enforces multiple learning tasks to share knowledge to improve their generalization abilities. While shallow multitask learning can learn task relations, it can only handle predefined features. Modern deep multitask learning can jointly learn latent features and task sharing, but they are obscure in task relation. Also, they predefine which layers and neurons should share across tasks and cannot learn adaptively. To address these challenges, this paper proposes a new multitask learning framework that jointly learns latent features and explicit task relations by complementing the strength of existing shallow and deep multitask learning scenarios. Specifically, we propose to model the task relation as the similarity between task input gradients, with a theoretical analysis of their equivalency. In addition, we innovatively propose a multitask learning objective that explicitly learns task relations by a new regularizer. Theoretical analysis shows that the generalizability error has been reduced thanks to the proposed regularizer. Extensive experiments on several multitask learning and image classification benchmarks demonstrate the proposed method effectiveness, efficiency as well as reasonableness in the learned task relation patterns.