Unanimous Prediction for 100% Precision with Application to Learning Semantic Mappings

TL;DR

This paper introduces the unanimity principle for semantic parsing, enabling models to guarantee 100% precision by only predicting when all consistent models agree, effectively balancing accuracy and abstention.

Contribution

It proposes a novel unanimity-based method that ensures perfect precision in semantic parsing by efficiently reasoning over all consistent models.

Findings

Achieves 100% precision on GeoQuery dataset

Requires only checking two models to reason over all consistent models

Works well even with limited training data from adversarial distributions

Abstract

Can we train a system that, on any new input, either says "don't know" or makes a prediction that is guaranteed to be correct? We answer the question in the affirmative provided our model family is well-specified. Specifically, we introduce the unanimity principle: only predict when all models consistent with the training data predict the same output. We operationalize this principle for semantic parsing, the task of mapping utterances to logical forms. We develop a simple, efficient method that reasons over the infinite set of all consistent models by only checking two of the models. We prove that our method obtains 100% precision even with a modest amount of training data from a possibly adversarial distribution. Empirically, we demonstrate the effectiveness of our approach on the standard GeoQuery dataset.