AdapMTL: Adaptive Pruning Framework for Multitask Learning Model

TL;DR

AdapMTL is an adaptive pruning framework for multitask learning models that dynamically balances sparsity and accuracy across tasks by co-optimizing soft thresholds and model weights, leading to efficient, high-performing models.

Contribution

It introduces learnable soft thresholds for different model components and an adaptive loss weighting mechanism to optimize pruning in multitask models.

Findings

Outperforms existing pruning methods on NYU-v2 and Tiny-Taskonomy datasets.

Effectively balances sparsity and accuracy across multiple tasks.

Demonstrates robustness and efficiency in model compression for multitask learning.

Abstract

In the domain of multimedia and multimodal processing, the efficient handling of diverse data streams such as images, video, and sensor data is paramount. Model compression and multitask learning (MTL) are crucial in this field, offering the potential to address the resource-intensive demands of processing and interpreting multiple forms of media simultaneously. However, effectively compressing a multitask model presents significant challenges due to the complexities of balancing sparsity allocation and accuracy performance across multiple tasks. To tackle these challenges, we propose AdapMTL, an adaptive pruning framework for MTL models. AdapMTL leverages multiple learnable soft thresholds independently assigned to the shared backbone and the task-specific heads to capture the nuances in different components' sensitivity to pruning. During training, it co-optimizes the soft thresholds and MTL model weights to automatically determine the suitable sparsity level at each component to achieve both high task accuracy and high overall sparsity. It further incorporates an adaptive weighting mechanism that dynamically adjusts the importance of task-specific losses based on each task's robustness to pruning. We demonstrate the effectiveness of AdapMTL through comprehensive experiments on popular multitask datasets, namely NYU-v2 and Tiny-Taskonomy, with different architectures, showcasing superior performance compared to state-of-the-art pruning methods.