AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries

TL;DR

AMBROSIA is a new benchmark dataset designed to evaluate and improve semantic parsers' ability to interpret ambiguous natural language questions into SQL queries, highlighting current model limitations.

Contribution

The paper introduces AMBROSIA, a dataset with diverse ambiguities and a controlled database generation method, to advance text-to-SQL parsing research.

Findings

LLMs struggle with ambiguity interpretation in AMBROSIA

The dataset covers scope, attachment, and vagueness ambiguities

Benchmark results reveal significant challenges for current models

Abstract

Practical semantic parsers are expected to understand user utterances and map them to executable programs, even when these are ambiguous. We introduce a new benchmark, AMBROSIA, which we hope will inform and inspire the development of text-to-SQL parsers capable of recognizing and interpreting ambiguous requests. Our dataset contains questions showcasing three different types of ambiguity (scope ambiguity, attachment ambiguity, and vagueness), their interpretations, and corresponding SQL queries. In each case, the ambiguity persists even when the database context is provided. This is achieved through a novel approach that involves controlled generation of databases from scratch. We benchmark various LLMs on AMBROSIA, revealing that even the most advanced models struggle to identify and interpret ambiguity in questions.