End-to-end speech-to-dialog-act recognition

TL;DR

This paper introduces an end-to-end speech-to-dialog-act recognition model that integrates acoustic features and dialog act detection, improving accuracy and robustness over traditional pipeline methods.

Contribution

The paper presents a novel end-to-end model combining acoustic-to-word ASR with dialog act recognition, enabling joint training and improved performance.

Findings

Significant accuracy improvement over traditional methods

Robustness against ASR errors demonstrated

Joint DA segmentation further enhances results

Abstract

Spoken language understanding, which extracts intents and/or semantic concepts in utterances, is conventionally formulated as a post-processing of automatic speech recognition. It is usually trained with oracle transcripts, but needs to deal with errors by ASR. Moreover, there are acoustic features which are related with intents but not represented with the transcripts. In this paper, we present an end-to-end model which directly converts speech into dialog acts without the deterministic transcription process. In the proposed model, the dialog act recognition network is conjunct with an acoustic-to-word ASR model at its latent layer before the softmax layer, which provides a distributed representation of word-level ASR decoding information. Then, the entire network is fine-tuned in an end-to-end manner. This allows for stable training as well as robustness against ASR errors. The model is further extended to conduct DA segmentation jointly. Evaluations with the Switchboard corpus demonstrate that the proposed method significantly improves dialog act recognition accuracy from the conventional pipeline framework.