Parameter-Efficient Fine-Tuning of Foundation Models for CLP Speech Classification

TL;DR

This paper explores parameter-efficient fine-tuning of foundation models like Wav2Vec2, WavLM, and Whisper for detecting and classifying the severity of cleft lip and palate speech, showing significant improvements over traditional methods.

Contribution

It introduces PEFT techniques such as LoRA and DoRA to enhance foundation model performance in CLP speech classification tasks.

Findings

PEFT methods improve F1 scores significantly.

Foundation models outperform handcrafted features.

Results show up to 63.4% relative F1 improvement.

Abstract

We propose the use of parameter-efficient fine-tuning (PEFT) of foundation models for cleft lip and palate (CLP) detection and severity classification. In CLP, nasalization increases with severity due to the abnormal passage between the oral and nasal tracts; this causes oral stops to be replaced by glottal stops and alters formant trajectories and vowel space. Since foundation models are trained for grapheme prediction or long-term quantized representation prediction, they may better discriminate CLP severity when fine-tuned on domain-specific data. We conduct experiments on two datasets: English (NMCPC) and Kannada (AIISH). We perform a comparative analysis using embeddings from self-supervised models Wav2Vec2 and WavLM, and the weakly supervised Whisper, each paired with SVM classifiers, and compare them with traditional handcrafted features eGeMAPS and ComParE. Finally, we fine-tune the best-performing Whisper model using PEFT techniques: Low-Rank Adapter (LoRA) and Decomposed Low-Rank Adapter (DoRA). Our results demonstrate that the proposed approach achieves relative improvements of 26.4% and 63.4% in macro-average F1 score over the best foundation model and handcrafted feature baselines on the NMCPC dataset, and improvements of 6.1% and 52.9% on the AIISH dataset, respectively.