Trace Norm Regularised Deep Multi-Task Learning

TL;DR

This paper introduces a novel deep multi-task learning framework that uses tensor trace norm regularization to automatically learn optimal parameter sharing strategies across neural networks, enhancing flexibility and efficiency.

Contribution

It presents a data-driven approach to learn parameter sharing in multi-task learning without predefining sharing strategies, using tensor trace norm regularization.

Findings

Effective automatic sharing learned across models

Improved multi-task learning performance

Flexible sharing strategy adapts to data

Abstract

We propose a framework for training multiple neural networks simultaneously. The parameters from all models are regularised by the tensor trace norm, so that each neural network is encouraged to reuse others' parameters if possible -- this is the main motivation behind multi-task learning. In contrast to many deep multi-task learning models, we do not predefine a parameter sharing strategy by specifying which layers have tied parameters. Instead, our framework considers sharing for all shareable layers, and the sharing strategy is learned in a data-driven way.