Beyond the Known: Investigating LLMs Performance on Out-of-Domain Intent Detection

TL;DR

This paper evaluates large language models' ability to detect out-of-domain user queries in dialogue systems, highlighting their strengths in zero-shot and few-shot settings and identifying key challenges for future improvements.

Contribution

It provides a comprehensive evaluation of LLMs for OOD detection, analyzing their performance, limitations, and offering guidance for enhancing their capabilities.

Findings

LLMs show strong zero-shot and few-shot performance

LLMs are less effective than fully fine-tuned models

Challenges include knowledge transfer and understanding long instructions

Abstract

Out-of-domain (OOD) intent detection aims to examine whether the user's query falls outside the predefined domain of the system, which is crucial for the proper functioning of task-oriented dialogue (TOD) systems. Previous methods address it by fine-tuning discriminative models. Recently, some studies have been exploring the application of large language models (LLMs) represented by ChatGPT to various downstream tasks, but it is still unclear for their ability on OOD detection task.This paper conducts a comprehensive evaluation of LLMs under various experimental settings, and then outline the strengths and weaknesses of LLMs. We find that LLMs exhibit strong zero-shot and few-shot capabilities, but is still at a disadvantage compared to models fine-tuned with full resource. More deeply, through a series of additional analysis experiments, we discuss and summarize the challenges faced by LLMs and provide guidance for future work including injecting domain knowledge, strengthening knowledge transfer from IND(In-domain) to OOD, and understanding long instructions.