Domain Expansion: A Latent Space Construction Framework for Multi-Task Learning

TL;DR

The paper introduces Domain Expansion, a framework that constructs a structured, orthogonal latent space to improve multi-task learning by preventing conflicting gradients and enabling interpretable representations.

Contribution

It proposes a novel orthogonal pooling mechanism to create a mutually orthogonal latent space for multiple objectives, addressing latent representation collapse.

Findings

Prevents latent representation collapse in multi-task learning.

Produces an interpretable and manipulable latent space.

Achieves superior performance on diverse benchmarks.

Abstract

Training a single network with multiple objectives often leads to conflicting gradients that degrade shared representations, forcing them into a compromised state that is suboptimal for any single task--a problem we term latent representation collapse. We introduce Domain Expansion, a framework that prevents these conflicts by restructuring the latent space itself. Our framework uses a novel orthogonal pooling mechanism to construct a latent space where each objective is assigned to a mutually orthogonal subspace. We validate our approach across diverse benchmarks--including ShapeNet, MPIIGaze, and Rotated MNIST--on challenging multi-objective problems combining classification with pose and gaze estimation. Our experiments demonstrate that this structure not only prevents collapse but also yields an explicit, interpretable, and compositional latent space where concepts can be directly manipulated.