MIDAS: Multi-level Intent, Domain, And Slot Knowledge Distillation for Multi-turn NLU

TL;DR

This paper presents MIDAS, a multi-level knowledge distillation approach that enhances multi-turn natural language understanding by integrating intent, domain, and slot information from specialized teacher models.

Contribution

Introduces a novel multi-level knowledge distillation framework for multi-turn NLU, combining multiple teacher models to improve understanding of complex dialogues.

Findings

Improved accuracy in multi-turn dialogue intent detection.

Enhanced slot filling performance in multi-turn conversations.

Open-sourced implementation available for further research.

Abstract

Although Large Language Models (LLMs) can generate coherent text, they often struggle to recognise user intent behind queries. In contrast, Natural Language Understanding (NLU) models interpret the purpose and key information of user input for responsive interactions. Existing NLU models typically map utterances to a dual-level semantic frame, involving sentence-level intent (SI) and word-level slot (WS) labels. However, real-life conversations primarily consist of multi-turn dialogues, requiring the interpretation of complex and extended exchanges. Researchers encounter challenges in addressing all facets of multi-turn dialogue using a unified NLU model. This paper introduces MIDAS, a novel approach leveraging multi-level intent, domain, and slot knowledge distillation for multi-turn NLU. We construct distinct teachers for SI detection, WS filling, and conversation-level domain (CD) classification, each fine-tuned for specific knowledge. A multi-teacher loss is proposed to facilitate the integration of these teachers, guiding a student model in multi-turn dialogue tasks. Results demonstrate the efficacy of our model in improving multi-turn conversation understanding, showcasing the potential for advancements in NLU through multi-level dialogue knowledge distillation. Our implementation is open-sourced on https://github.com/adlnlp/Midas.