An Effective Non-Autoregressive Model for Spoken Language Understanding

TL;DR

This paper introduces a novel non-autoregressive SLU model called Layered-Refine Transformer that improves inference speed by over 10 times while enhancing accuracy through a layered refinement mechanism and slot label generation tasks.

Contribution

The paper proposes a new non-autoregressive SLU model with a layered refinement mechanism and slot label generation, addressing uncoordinated-slot issues and boosting performance.

Findings

Achieves 1.5% improvement in overall accuracy.

Speeds up inference by more than 10 times.

Outperforms state-of-the-art baseline.

Abstract

Spoken Language Understanding (SLU), a core component of the task-oriented dialogue system, expects a shorter inference latency due to the impatience of humans. Non-autoregressive SLU models clearly increase the inference speed but suffer uncoordinated-slot problems caused by the lack of sequential dependency information among each slot chunk. To gap this shortcoming, in this paper, we propose a novel non-autoregressive SLU model named Layered-Refine Transformer, which contains a Slot Label Generation (SLG) task and a Layered Refine Mechanism (LRM). SLG is defined as generating the next slot label with the token sequence and generated slot labels. With SLG, the non-autoregressive model can efficiently obtain dependency information during training and spend no extra time in inference. LRM predicts the preliminary SLU results from Transformer's middle states and utilizes them to guide the final prediction. Experiments on two public datasets indicate that our model significantly improves SLU performance (1.5\% on Overall accuracy) while substantially speed up (more than 10 times) the inference process over the state-of-the-art baseline.