Evaluating Parameter-Efficient Transfer Learning Approaches on SURE Benchmark for Speech Understanding

TL;DR

This paper introduces the SURE benchmark for evaluating parameter-efficient transfer learning methods in speech understanding, proposing a new ConvAdapter that outperforms standard adapters with fewer trainable parameters.

Contribution

The paper presents the SURE benchmark for speech tasks and introduces ConvAdapter, a novel convolution-based adapter that improves efficiency and performance in transfer learning.

Findings

ConvAdapter outperforms standard adapters on SURE tasks.

ConvAdapter achieves comparable results to prefix tuning and LoRA with only 0.94% of trainable parameters.

Parameter-efficient methods are effective for speech synthesis tasks.

Abstract

Fine-tuning is widely used as the default algorithm for transfer learning from pre-trained models. Parameter inefficiency can however arise when, during transfer learning, all the parameters of a large pre-trained model need to be updated for individual downstream tasks. As the number of parameters grows, fine-tuning is prone to overfitting and catastrophic forgetting. In addition, full fine-tuning can become prohibitively expensive when the model is used for many tasks. To mitigate this issue, parameter-efficient transfer learning algorithms, such as adapters and prefix tuning, have been proposed as a way to introduce a few trainable parameters that can be plugged into large pre-trained language models such as BERT, and HuBERT. In this paper, we introduce the Speech UndeRstanding Evaluation (SURE) benchmark for parameter-efficient learning for various speech-processing tasks. Additionally, we introduce a new adapter, ConvAdapter, based on 1D convolution. We show that ConvAdapter outperforms the standard adapters while showing comparable performance against prefix tuning and LoRA with only 0.94% of trainable parameters on some of the task in SURE. We further explore the effectiveness of parameter efficient transfer learning for speech synthesis task such as Text-to-Speech (TTS).