ConVEx: Data-Efficient and Few-Shot Slot Labeling

TL;DR

ConVEx introduces a novel pretraining approach for dialog slot-labeling that achieves state-of-the-art results with reduced pretraining time and efficient fine-tuning, especially effective in few-shot scenarios.

Contribution

It presents a new pairwise cloze pretraining task aligned with sequence labeling, enabling efficient domain-specific slot labeler fine-tuning with minimal pretraining.

Findings

State-of-the-art performance across diverse domains.

Significant improvements in few-shot learning setups.

Reduced pretraining time and computational cost.

Abstract

We propose ConVEx (Conversational Value Extractor), an efficient pretraining and fine-tuning neural approach for slot-labeling dialog tasks. Instead of relying on more general pretraining objectives from prior work (e.g., language modeling, response selection), ConVEx's pretraining objective, a novel pairwise cloze task using Reddit data, is well aligned with its intended usage on sequence labeling tasks. This enables learning domain-specific slot labelers by simply fine-tuning decoding layers of the pretrained general-purpose sequence labeling model, while the majority of the pretrained model's parameters are kept frozen. We report state-of-the-art performance of ConVEx across a range of diverse domains and data sets for dialog slot-labeling, with the largest gains in the most challenging, few-shot setups. We believe that ConVEx's reduced pretraining times (i.e., only 18 hours on 12 GPUs) and cost, along with its efficient fine-tuning and strong performance, promise wider portability and scalability for data-efficient sequence-labeling tasks in general.