Less is More -- Towards parsimonious multi-task models using structured sparsity

TL;DR

This paper introduces a structured sparsity approach using channel-wise l1/l2 regularization to create highly sparse, efficient multi-task deep learning models that can outperform dense models on standard vision datasets.

Contribution

The work proposes a novel structured sparsity method with channel-wise l1/l2 regularization for multi-task models, achieving high sparsity and improved performance.

Findings

Multi-task models with ~70% sparsity outperform dense counterparts.

Sparsification reduces inference time significantly.

Model performance varies with degree of sparsity.

Abstract

Model sparsification in deep learning promotes simpler, more interpretable models with fewer parameters. This not only reduces the model's memory footprint and computational needs but also shortens inference time. This work focuses on creating sparse models optimized for multiple tasks with fewer parameters. These parsimonious models also possess the potential to match or outperform dense models in terms of performance. In this work, we introduce channel-wise l1/l2 group sparsity in the shared convolutional layers parameters (or weights) of the multi-task learning model. This approach facilitates the removal of extraneous groups i.e., channels (due to l1 regularization) and also imposes a penalty on the weights, further enhancing the learning efficiency for all tasks (due to l2 regularization). We analyzed the results of group sparsity in both single-task and multi-task settings on two widely-used Multi-Task Learning (MTL) datasets: NYU-v2 and CelebAMask-HQ. On both datasets, which consist of three different computer vision tasks each, multi-task models with approximately 70% sparsity outperform their dense equivalents. We also investigate how changing the degree of sparsification influences the model's performance, the overall sparsity percentage, the patterns of sparsity, and the inference time.