Factors affecting the in-context learning abilities of LLMs for dialogue state tracking

TL;DR

This paper investigates how various factors influence the effectiveness of in-context learning in large language models for dialogue state tracking, using a systematic approach with different models and demonstration selection methods.

Contribution

It introduces a systematic analysis of demonstration selection and prompt context factors affecting in-context learning performance in dialogue state tracking tasks.

Findings

Demonstration selection significantly impacts DST performance.

Prompt context length influences model accuracy.

Different models exhibit varying sensitivity to demonstration quality.

Abstract

This study explores the application of in-context learning (ICL) to the dialogue state tracking (DST) problem and investigates the factors that influence its effectiveness. We use a sentence embedding based k-nearest neighbour method to retrieve the suitable demonstrations for ICL. The selected demonstrations, along with the test samples, are structured within a template as input to the LLM. We then conduct a systematic study to analyse the impact of factors related to demonstration selection and prompt context on DST performance. This work is conducted using the MultiWoZ2.4 dataset and focuses primarily on the OLMo-7B-instruct, Mistral-7B-Instruct-v0.3, and Llama3.2-3B-Instruct models. Our findings provide several useful insights on in-context learning abilities of LLMs for dialogue state tracking.