DIET: Lightweight Language Understanding for Dialogue Systems

TL;DR

DIET introduces a lightweight transformer architecture for dialogue language understanding that outperforms larger models and does not rely on pre-trained embeddings, offering efficiency and high accuracy.

Contribution

The paper presents the DIET architecture, demonstrating its effectiveness and efficiency in dialogue NLU tasks without needing large pre-trained models.

Findings

DIET outperforms state-of-the-art models on multi-domain NLU datasets.

Pre-trained models show no clear benefit over supervised training for this task.

DIET is approximately six times faster to train than BERT-based models.

Abstract

Large-scale pre-trained language models have shown impressive results on language understanding benchmarks like GLUE and SuperGLUE, improving considerably over other pre-training methods like distributed representations (GloVe) and purely supervised approaches. We introduce the Dual Intent and Entity Transformer (DIET) architecture, and study the effectiveness of different pre-trained representations on intent and entity prediction, two common dialogue language understanding tasks. DIET advances the state of the art on a complex multi-domain NLU dataset and achieves similarly high performance on other simpler datasets. Surprisingly, we show that there is no clear benefit to using large pre-trained models for this task, and in fact DIET improves upon the current state of the art even in a purely supervised setup without any pre-trained embeddings. Our best performing model outperforms fine-tuning BERT and is about six times faster to train.