Parameter Efficient Transfer Learning for Various Speech Processing Tasks

TL;DR

This paper introduces a novel adapter architecture for speech processing that enhances transfer learning efficiency by requiring fewer parameters and better leveraging features across layers, achieving comparable or superior performance.

Contribution

A new adapter design that adaptively utilizes multi-level features for diverse speech tasks, improving parameter efficiency and performance over existing methods.

Findings

Achieved comparable or better performance than naive fine-tuning.

Utilized only 11% of learnable parameters.

Outperformed existing adapter architectures.

Abstract

Fine-tuning of self-supervised models is a powerful transfer learning method in a variety of fields, including speech processing, since it can utilize generic feature representations obtained from large amounts of unlabeled data. Fine-tuning, however, requires a new parameter set for each downstream task, which is parameter inefficient. Adapter architecture is proposed to partially solve this issue by inserting lightweight learnable modules into a frozen pre-trained model. However, existing adapter architectures fail to adaptively leverage low- to high-level features stored in different layers, which is necessary for solving various kinds of speech processing tasks. Thus, we propose a new adapter architecture to acquire feature representations more flexibly for various speech tasks. In experiments, we applied this adapter to WavLM on four speech tasks. It performed on par or better than naive fine-tuning, with only 11% of learnable parameters. It also outperformed an existing adapter architecture.