Exploring Efficient-tuning Methods in Self-supervised Speech Models

TL;DR

This paper investigates the use of adapter modules to efficiently fine-tune large self-supervised speech models, achieving high performance with significantly fewer trainable parameters.

Contribution

It provides the first comprehensive analysis of various adapter types in speech SSL models, demonstrating over 90% parameter reduction without sacrificing performance.

Findings

Achieved performance parity with over 90% fewer parameters.

First extensive study of adapters in speech self-supervised learning.

Discussed advantages and disadvantages of efficient tuning methods.

Abstract

In this study, we aim to explore efficient tuning methods for speech self-supervised learning. Recent studies show that self-supervised learning (SSL) can learn powerful representations for different speech tasks. However, fine-tuning pre-trained models for each downstream task is parameter-inefficient since SSL models are notoriously large with millions of parameters. Adapters are lightweight modules commonly used in NLP to solve this problem. In downstream tasks, the parameters of SSL models are frozen, and only the adapters are trained. Given the lack of studies generally exploring the effectiveness of adapters for self-supervised speech tasks, we intend to fill this gap by adding various adapter modules in pre-trained speech SSL models. We show that the performance parity can be achieved with over 90% parameter reduction, and discussed the pros and cons of efficient tuning techniques. This is the first comprehensive investigation of various adapter types across speech tasks.