Learning Multiple Tasks with Multilinear Relationship Networks

TL;DR

This paper introduces Multilinear Relationship Networks (MRN), a deep learning approach that models task relationships to improve multi-task learning performance and transferability.

Contribution

MRN is the first to use tensor normal priors over parameter tensors to explicitly discover and leverage task relationships in deep networks.

Findings

MRN achieves state-of-the-art results on three multi-task datasets.

MRN effectively alleviates negative transfer and under-transfer issues.

Joint learning of features and task relationships enhances multi-task learning.

Abstract

Deep networks trained on large-scale data can learn transferable features to promote learning multiple tasks. Since deep features eventually transition from general to specific along deep networks, a fundamental problem of multi-task learning is how to exploit the task relatedness underlying parameter tensors and improve feature transferability in the multiple task-specific layers. This paper presents Multilinear Relationship Networks (MRN) that discover the task relationships based on novel tensor normal priors over parameter tensors of multiple task-specific layers in deep convolutional networks. By jointly learning transferable features and multilinear relationships of tasks and features, MRN is able to alleviate the dilemma of negative-transfer in the feature layers and under-transfer in the classifier layer. Experiments show that MRN yields state-of-the-art results on three multi-task learning datasets.