Compositional Semantic Parsing with Large Language Models

TL;DR

This paper enhances large language models' ability to perform compositional semantic parsing in realistic tasks by refining prompting techniques, achieving state-of-the-art results with minimal training data.

Contribution

It introduces a least-to-most prompting method that decomposes semantic parsing tasks, improving performance and data efficiency in complex, real-world scenarios.

Findings

Sets new state-of-the-art on CFQ dataset

Achieves high accuracy with only 1% of traditional training data

Demonstrates general applicability to other tasks and domains

Abstract

Humans can reason compositionally when presented with new tasks. Previous research shows that appropriate prompting techniques enable large language models (LLMs) to solve artificial compositional generalization tasks such as SCAN. In this work, we identify additional challenges in more realistic semantic parsing tasks with larger vocabulary and refine these prompting techniques to address them. Our best method is based on least-to-most prompting: it decomposes the problem using prompting-based syntactic parsing, then uses this decomposition to select appropriate exemplars and to sequentially generate the semantic parse. This method allows us to set a new state of the art for CFQ while requiring only 1% of the training data used by traditional approaches. Due to the general nature of our approach, we expect similar efforts will lead to new results in other tasks and domains, especially for knowledge-intensive applications.