Slot Induction via Pre-trained Language Model Probing and Multi-level Contrastive Learning

TL;DR

This paper introduces a novel approach for slot induction in task-oriented dialogue systems using pre-trained language model probing and contrastive learning, reducing reliance on extensive token-level annotations.

Contribution

It proposes a new unsupervised method leveraging PLM probing and contrastive learning for slot induction, improving slot filling performance without token-level supervision.

Findings

Effective slot induction without token-level annotations

Bridges performance gap with supervised models on benchmarks

Enhances slot label representations for emerging intents

Abstract

Recent advanced methods in Natural Language Understanding for Task-oriented Dialogue (TOD) Systems (e.g., intent detection and slot filling) require a large amount of annotated data to achieve competitive performance. In reality, token-level annotations (slot labels) are time-consuming and difficult to acquire. In this work, we study the Slot Induction (SI) task whose objective is to induce slot boundaries without explicit knowledge of token-level slot annotations. We propose leveraging Unsupervised Pre-trained Language Model (PLM) Probing and Contrastive Learning mechanism to exploit (1) unsupervised semantic knowledge extracted from PLM, and (2) additional sentence-level intent label signals available from TOD. Our approach is shown to be effective in SI task and capable of bridging the gaps with token-level supervised models on two NLU benchmark datasets. When generalized to emerging intents, our SI objectives also provide enhanced slot label representations, leading to improved performance on the Slot Filling tasks.