The Devil is in the Details: On Models and Training Regimes for Few-Shot Intent Classification

TL;DR

This paper systematically analyzes various design choices in few-shot intent classification, revealing that combining cross-encoders with episodic meta-learning yields the best performance and emphasizing the importance of training details.

Contribution

It provides a comprehensive study of encoding architectures, similarity functions, and training regimes in FSIC, identifying optimal combinations and challenging common assumptions.

Findings

Cross-encoder with parameterized similarity and episodic meta-learning performs best.

Episodic training offers more robustness than non-episodic training.

Splitting episodes into support and query sets is not always necessary in meta-learning.

Abstract

Few-shot Intent Classification (FSIC) is one of the key challenges in modular task-oriented dialog systems. While advanced FSIC methods are similar in using pretrained language models to encode texts and nearest neighbour-based inference for classification, these methods differ in details. They start from different pretrained text encoders, use different encoding architectures with varying similarity functions, and adopt different training regimes. Coupling these mostly independent design decisions and the lack of accompanying ablation studies are big obstacle to identify the factors that drive the reported FSIC performance. We study these details across three key dimensions: (1) Encoding architectures: Cross-Encoder vs Bi-Encoders; (2) Similarity function: Parameterized (i.e., trainable) functions vs non-parameterized function; (3) Training regimes: Episodic meta-learning vs the straightforward (i.e., non-episodic) training. Our experimental results on seven FSIC benchmarks reveal three important findings. First, the unexplored combination of the cross-encoder architecture (with parameterized similarity scoring function) and episodic meta-learning consistently yields the best FSIC performance. Second, Episodic training yields a more robust FSIC classifier than non-episodic one. Third, in meta-learning methods, splitting an episode to support and query sets is not a must. Our findings paves the way for conducting state-of-the-art research in FSIC and more importantly raise the community's attention to details of FSIC methods. We release our code and data publicly.