On the relationship between disentanglement and multi-task learning

TL;DR

This paper investigates how disentangled representations naturally emerge during multi-task neural network training with hard parameter sharing, suggesting a close relationship between the two concepts.

Contribution

It provides an empirical analysis demonstrating that disentanglement occurs naturally in multi-task learning, highlighting the potential for reusing representations across tasks.

Findings

Disentanglement appears naturally during multi-task training

Neural networks trained on multiple tasks develop disentangled representations

Standard metrics confirm the emergence of disentanglement in this setting

Abstract

One of the main arguments behind studying disentangled representations is the assumption that they can be easily reused in different tasks. At the same time finding a joint, adaptable representation of data is one of the key challenges in the multi-task learning setting. In this paper, we take a closer look at the relationship between disentanglement and multi-task learning based on hard parameter sharing. We perform a thorough empirical study of the representations obtained by neural networks trained on automatically generated supervised tasks. Using a set of standard metrics we show that disentanglement appears naturally during the process of multi-task neural network training.