Structured Pruning for Multi-Task Deep Neural Networks

TL;DR

This paper explores structured pruning techniques for multi-task neural networks, comparing filter importance criteria and pruning strategies to optimize model efficiency without significant performance loss.

Contribution

It introduces an MTL-based filter pruning criterion and evaluates iterative pruning versus random re-initialization for multi-task models.

Findings

Different pruning methods yield similar performance when parameters are comparable.

Extreme pruning levels cause significant performance drops across tasks.

Re-training pruned models from scratch improves results.

Abstract

Although multi-task deep neural network (DNN) models have computation and storage benefits over individual single-task DNN models, they can be further optimized via model compression. Numerous structured pruning methods are already developed that can readily achieve speedups in single-task models, but the pruning of multi-task networks has not yet been extensively studied. In this work, we investigate the effectiveness of structured pruning on multi-task models. We use an existing single-task filter pruning criterion and also introduce an MTL-based filter pruning criterion for estimating the filter importance scores. We prune the model using an iterative pruning strategy with both pruning methods. We show that, with careful hyper-parameter tuning, architectures obtained from different pruning methods do not have significant differences in their performances across tasks when the number of parameters is similar. We also show that iterative structure pruning may not be the best way to achieve a well-performing pruned model because, at extreme pruning levels, there is a high drop in performance across all tasks. But when the same models are randomly initialized and re-trained, they show better results.