Few-Shot Adaptation for Parsing Contextual Utterances with LLMs

TL;DR

This paper evaluates how large language models can adapt to limited annotated contextual utterances in conversational semantic parsing, comparing four paradigms and introducing a new dataset for benchmarking.

Contribution

It systematically compares four paradigms for few-shot adaptation in conversational semantic parsing and introduces SMCalFlow-EventQueries for cross-paradigm evaluation.

Findings

Rewrite-then-Parse paradigm performs best overall.

In-context learning and fine-tuning show different strengths.

SMCalFlow-EventQueries enables comprehensive paradigm comparison.

Abstract

We evaluate the ability of semantic parsers based on large language models (LLMs) to handle contextual utterances. In real-world settings, there typically exists only a limited number of annotated contextual utterances due to annotation cost, resulting in an imbalance compared to non-contextual utterances. Therefore, parsers must adapt to contextual utterances with a few training examples. We examine four major paradigms for doing so in conversational semantic parsing i.e., Parse-with-Utterance-History, Parse-with-Reference-Program, Parse-then-Resolve, and Rewrite-then-Parse. To facilitate such cross-paradigm comparisons, we construct SMCalFlow-EventQueries, a subset of contextual examples from SMCalFlow with additional annotations. Experiments with in-context learning and fine-tuning suggest that Rewrite-then-Parse is the most promising paradigm when holistically considering parsing accuracy, annotation cost, and error types.