End-to-end Acoustic-linguistic Emotion and Intent Recognition Enhanced by Semi-supervised Learning

TL;DR

This paper introduces semi-supervised learning methods to improve end-to-end acoustic and linguistic emotion and intent recognition from speech, leveraging large unlabelled datasets to enhance performance.

Contribution

It presents a novel application of semi-supervised learning, including fix-match and full-match approaches, for joint emotion and intent recognition in speech.

Findings

Semi-supervised learning improves recognition accuracy.

Late fusion of acoustic and linguistic models enhances performance.

Proposed methods outperform baseline models by significant margins.

Abstract

Emotion and intent recognition from speech is essential and has been widely investigated in human-computer interaction. The rapid development of social media platforms, chatbots, and other technologies has led to a large volume of speech data streaming from users. Nevertheless, annotating such data manually is expensive, making it challenging to train machine learning models for recognition purposes. To this end, we propose applying semi-supervised learning to incorporate a large scale of unlabelled data alongside a relatively smaller set of labelled data. We train end-to-end acoustic and linguistic models, each employing multi-task learning for emotion and intent recognition. Two semi-supervised learning approaches, including fix-match learning and full-match learning, are compared. The experimental results demonstrate that the semi-supervised learning approaches improve model performance in speech emotion and intent recognition from both acoustic and text data. The late fusion of the best models outperforms the acoustic and text baselines by joint recognition balance metrics of 12.3% and 10.4%, respectively.