Towards a Common Speech Analysis Engine

TL;DR

This paper proposes a unified speech analysis engine based on HuBERT that handles multiple tasks with high accuracy, supports small datasets, and enables distributed training, surpassing current state-of-the-art results.

Contribution

The paper introduces a common speech analysis engine architecture leveraging HuBERT, capable of multi-task processing, small dataset adaptation, and distributed training, achieving superior performance.

Findings

Outperforms state-of-the-art on language identification and emotion recognition.

Effective with reduced training data for emotion recognition.

Supports distributed training with private data.

Abstract

Recent innovations in self-supervised representation learning have led to remarkable advances in natural language processing. That said, in the speech processing domain, self-supervised representation learning-based systems are not yet considered state-of-the-art. We propose leveraging recent advances in self-supervised-based speech processing to create a common speech analysis engine. Such an engine should be able to handle multiple speech processing tasks, using a single architecture, to obtain state-of-the-art accuracy. The engine must also enable support for new tasks with small training datasets. Beyond that, a common engine should be capable of supporting distributed training with client in-house private data. We present the architecture for a common speech analysis engine based on the HuBERT self-supervised speech representation. Based on experiments, we report our results for language identification and emotion recognition on the standard evaluations NIST-LRE 07 and IEMOCAP. Our results surpass the state-of-the-art performance reported so far on these tasks. We also analyzed our engine on the emotion recognition task using reduced amounts of training data and show how to achieve improved results.