Finding Task-specific Subnetworks in Multi-task Spoken Language Understanding Model

TL;DR

This paper introduces a method to identify task-specific subnetworks within multi-task SLU models using neural network pruning, enabling better adaptation to new tasks while reducing catastrophic forgetting.

Contribution

It proposes a novel pruning-based approach to find task-specific subnetworks in multi-task SLU models, improving adaptability and model efficiency.

Findings

Pruned subnetworks adapt well to new tasks with minimal performance loss.

The approach mitigates catastrophic forgetting in multi-task learning.

Experimental results on UniverSLU demonstrate effective task-specific subnetwork identification.

Abstract

Recently, multi-task spoken language understanding (SLU) models have emerged, designed to address various speech processing tasks. However, these models often rely on a large number of parameters. Also, they often encounter difficulties in adapting to new data for a specific task without experiencing catastrophic forgetting of previously trained tasks. In this study, we propose finding task-specific subnetworks within a multi-task SLU model via neural network pruning. In addition to model compression, we expect that the forgetting of previously trained tasks can be mitigated by updating only a task-specific subnetwork. We conduct experiments on top of the state-of-the-art multi-task SLU model ``UniverSLU'', trained for several tasks such as emotion recognition (ER), intent classification (IC), and automatic speech recognition (ASR). We show that pruned models were successful in adapting to additional ASR or IC data with minimal performance degradation on previously trained tasks.