Giving each task what it needs -- leveraging structured sparsity for tailored multi-task learning

TL;DR

This paper introduces Layer-Optimized Multi-Task (LOMT) models that leverage structured sparsity to select optimal features and layers for each task, improving multi-task learning performance especially in resource-constrained environments.

Contribution

The work proposes a novel two-step approach using structured sparsity to identify task-specific layers and decoders, enhancing multi-task learning efficiency and effectiveness.

Findings

LOMT models outperform conventional MTL models on NYU-v2 and CelebAMask-HD datasets.

Structured sparsity effectively identifies optimal layers for individual tasks.

Tailored architecture reduces redundancy and improves task-specific feature utilization.

Abstract

In the Multi-task Learning (MTL) framework, every task demands distinct feature representations, ranging from low-level to high-level attributes. It is vital to address the specific (feature/parameter) needs of each task, especially in computationally constrained environments. This work, therefore, introduces Layer-Optimized Multi-Task (LOMT) models that utilize structured sparsity to refine feature selection for individual tasks and enhance the performance of all tasks in a multi-task scenario. Structured or group sparsity systematically eliminates parameters from trivial channels and, sometimes, eventually, entire layers within a convolution neural network during training. Consequently, the remaining layers provide the most optimal features for a given task. In this two-step approach, we subsequently leverage this sparsity-induced optimal layer information to build the LOMT models by connecting task-specific decoders to these strategically identified layers, deviating from conventional approaches that uniformly connect decoders at the end of the network. This tailored architecture optimizes the network, focusing on essential features while reducing redundancy. We validate the efficacy of the proposed approach on two datasets, i.e., NYU-v2 and CelebAMask-HD datasets, for multiple heterogeneous tasks. A detailed performance analysis of the LOMT models, in contrast to the conventional MTL models, reveals that the LOMT models outperform for most task combinations. The excellent qualitative and quantitative outcomes highlight the effectiveness of employing structured sparsity for optimal layer (or feature) selection.