Leveraging Speech PTM, Text LLM, and Emotional TTS for Speech Emotion Recognition

TL;DR

This paper enhances speech emotion recognition by integrating advanced speech models, large language models, and emotional speech synthesis, demonstrating improved performance through various data augmentation techniques on the IEMOCAP dataset.

Contribution

It introduces a novel combination of speech PTM, GPT-4, and Azure TTS for generating synthetic emotional data to improve SER accuracy.

Findings

Data2Vec shows strong representation ability for SER.

Synthetic emotional speech improves recognition performance.

Data augmentation methods outperform other techniques.

Abstract

In this paper, we explored how to boost speech emotion recognition (SER) with the state-of-the-art speech pre-trained model (PTM), data2vec, text generation technique, GPT-4, and speech synthesis technique, Azure TTS. First, we investigated the representation ability of different speech self-supervised pre-trained models, and we found that data2vec has a good representation ability on the SER task. Second, we employed a powerful large language model (LLM), GPT-4, and emotional text-to-speech (TTS) model, Azure TTS, to generate emotionally congruent text and speech. We carefully designed the text prompt and dataset construction, to obtain the synthetic emotional speech data with high quality. Third, we studied different ways of data augmentation to promote the SER task with synthetic speech, including random mixing, adversarial training, transfer learning, and curriculum learning. Experiments and ablation studies on the IEMOCAP dataset demonstrate the effectiveness of our method, compared with other data augmentation methods, and data augmentation with other synthetic data.