Foundation Model Assisted Automatic Speech Emotion Recognition: Transcribing, Annotating, and Augmenting

TL;DR

This paper investigates how foundational models can automate and improve speech emotion recognition by transcribing, annotating, and augmenting datasets, reducing manual effort and enhancing system performance.

Contribution

It introduces a novel approach leveraging foundational models to automate transcription, annotation, and augmentation in speech emotion recognition tasks.

Findings

Foundational models improve transcription quality for SER.

Combining multiple LLM outputs enhances emotion annotation accuracy.

Augmentation of datasets with unlabeled speech is feasible and beneficial.

Abstract

Significant advances are being made in speech emotion recognition (SER) using deep learning models. Nonetheless, training SER systems remains challenging, requiring both time and costly resources. Like many other machine learning tasks, acquiring datasets for SER requires substantial data annotation efforts, including transcription and labeling. These annotation processes present challenges when attempting to scale up conventional SER systems. Recent developments in foundational models have had a tremendous impact, giving rise to applications such as ChatGPT. These models have enhanced human-computer interactions including bringing unique possibilities for streamlining data collection in fields like SER. In this research, we explore the use of foundational models to assist in automating SER from transcription and annotation to augmentation. Our study demonstrates that these models can generate transcriptions to enhance the performance of SER systems that rely solely on speech data. Furthermore, we note that annotating emotions from transcribed speech remains a challenging task. However, combining outputs from multiple LLMs enhances the quality of annotations. Lastly, our findings suggest the feasibility of augmenting existing speech emotion datasets by annotating unlabeled speech samples.