Intent Detection in the Age of LLMs

TL;DR

This paper explores how large language models can be adapted for intent detection in dialogue systems, comparing their performance with traditional models, and proposing hybrid approaches to improve accuracy and reduce latency.

Contribution

It introduces a hybrid system combining LLMs and sentence transformers with uncertainty routing, and demonstrates improved OOS detection using internal LLM representations.

Findings

Hybrid system achieves near-native LLM accuracy with 50% less latency.

Internal LLM representations improve OOS detection accuracy by over 5%.

Performance is influenced by intent label scope and label space size.

Abstract

Intent detection is a critical component of task-oriented dialogue systems (TODS) which enables the identification of suitable actions to address user utterances at each dialog turn. Traditional approaches relied on computationally efficient supervised sentence transformer encoder models, which require substantial training data and struggle with out-of-scope (OOS) detection. The emergence of generative large language models (LLMs) with intrinsic world knowledge presents new opportunities to address these challenges. In this work, we adapt 7 SOTA LLMs using adaptive in-context learning and chain-of-thought prompting for intent detection, and compare their performance with contrastively fine-tuned sentence transformer (SetFit) models to highlight prediction quality and latency tradeoff. We propose a hybrid system using uncertainty based routing strategy to combine the two approaches that along with negative data augmentation results in achieving the best of both worlds ( i.e. within 2% of native LLM accuracy with 50% less latency). To better understand LLM OOS detection capabilities, we perform controlled experiments revealing that this capability is significantly influenced by the scope of intent labels and the size of the label space. We also introduce a two-step approach utilizing internal LLM representations, demonstrating empirical gains in OOS detection accuracy and F1-score by >5% for the Mistral-7B model.