Joint Automatic Speech Recognition And Structure Learning For Better Speech Understanding

TL;DR

This paper introduces a joint speech recognition and structure learning framework that improves speech understanding by simultaneously transcribing speech and extracting structured content, outperforming traditional methods on multiple datasets.

Contribution

The paper presents a novel end-to-end model for joint speech recognition and structure learning, enabling simultaneous transcription and content extraction with superior performance.

Findings

Outperforms traditional sequence-to-sequence methods in transcription accuracy

Achieves state-of-the-art results on AISHELL-NER and SLURP datasets

Effectively extracts structured content during speech recognition

Abstract

Spoken language understanding (SLU) is a structure prediction task in the field of speech. Recently, many works on SLU that treat it as a sequence-to-sequence task have achieved great success. However, This method is not suitable for simultaneous speech recognition and understanding. In this paper, we propose a joint speech recognition and structure learning framework (JSRSL), an end-to-end SLU model based on span, which can accurately transcribe speech and extract structured content simultaneously. We conduct experiments on name entity recognition and intent classification using the Chinese dataset AISHELL-NER and the English dataset SLURP. The results show that our proposed method not only outperforms the traditional sequence-to-sequence method in both transcription and extraction capabilities but also achieves state-of-the-art performance on the two datasets.