Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering

TL;DR

This paper investigates how unanswerable questions in QA datasets often stem from unverifiable presuppositions, proposing a framework to improve detection and handling of such questions, with promising initial results.

Contribution

It introduces a novel three-step framework for presupposition generation, verification, and explanation in QA systems, addressing a key gap in unanswerable question handling.

Findings

Adding presupposition verification improves unanswerability detection.

Current entailment models are insufficient for verification tasks.

Preliminary integration yields modest performance gains.

Abstract

Many Question-Answering (QA) datasets contain unanswerable questions, but their treatment in QA systems remains primitive. Our analysis of the Natural Questions (Kwiatkowski et al. 2019) dataset reveals that a substantial portion of unanswerable questions ($\sim$21%) can be explained based on the presence of unverifiable presuppositions. We discuss the shortcomings of current models in handling such questions, and describe how an improved system could handle them. Through a user preference study, we demonstrate that the oracle behavior of our proposed system that provides responses based on presupposition failure is preferred over the oracle behavior of existing QA systems. Then we discuss how our proposed system could be implemented, presenting a novel framework that breaks down the problem into three steps: presupposition generation, presupposition verification and explanation generation. We report our progress in tackling each subproblem, and present a preliminary approach to integrating these steps into an existing QA system. We find that adding presuppositions and their verifiability to an existing model yields modest gains in downstream performance and unanswerability detection. The biggest bottleneck is the verification component, which needs to be substantially improved for the integrated system to approach ideal behavior -- even transfer from the best entailment models currently falls short.