Do I have the Knowledge to Answer? Investigating Answerability of Knowledge Base Questions

TL;DR

This paper introduces GrailQAbility, a benchmark dataset for KBQA that includes unanswerable questions caused by KB incompleteness, revealing current models' struggles with unanswerability detection and robustness.

Contribution

It creates the first systematic benchmark for unanswerability in KBQA and evaluates state-of-the-art models' performance and limitations on this challenge.

Findings

Models' performance drops on unanswerable questions

Models often misdetect unanswerability reasons

Certain forms of unanswerability are particularly challenging

Abstract

When answering natural language questions over knowledge bases, missing facts, incomplete schema and limited scope naturally lead to many questions being unanswerable. While answerability has been explored in other QA settings, it has not been studied for QA over knowledge bases (KBQA). We create GrailQAbility, a new benchmark KBQA dataset with unanswerability, by first identifying various forms of KB incompleteness that make questions unanswerable, and then systematically adapting GrailQA (a popular KBQA dataset with only answerable questions). Experimenting with three state-of-the-art KBQA models, we find that all three models suffer a drop in performance even after suitable adaptation for unanswerable questions. In addition, these often detect unanswerability for wrong reasons and find specific forms of unanswerability particularly difficult to handle. This underscores the need for further research in making KBQA systems robust to unanswerability