Iterative Repair with Weak Verifiers for Few-shot Transfer in KBQA with Unanswerability

TL;DR

This paper introduces a new approach for KBQA that effectively handles unanswerable questions in few-shot transfer settings by using iterative repair with feedback from verifiers, supported by new datasets.

Contribution

It proposes FUn-FuSIC, a novel method extending FuSIC for unanswerable questions, incorporating feedback from verifiers and a new evaluation framework.

Findings

FUn-FuSIC outperforms existing models on new datasets.

The approach establishes new state-of-the-art results for answerable few-shot transfer.

Iterative repair with verifiers improves question answerability assessment.

Abstract

Real-world applications of KBQA require models to handle unanswerable questions with a limited volume of in-domain labeled training data. We propose the novel task of few-shot transfer for KBQA with unanswerable questions and contribute two new datasets for performance evaluation. We present FUn-FuSIC - a novel solution for our task that extends FuSIC KBQA, the state-of-the-art few-shot transfer model for answerable-only KBQA. We first note that FuSIC-KBQA's iterative repair makes a strong assumption that all questions are unanswerable. As a remedy, we propose Feedback for Unanswerability (FUn), which uses iterative repair using feedback from a suite of strong and weak verifiers, and an adaptation of self consistency for unanswerabilty to better assess the answerability of a question. Our experiments show that FUn-FuSIC significantly outperforms suitable adaptations of multiple LLM based and supervised SoTA models on our task, while establishing a new SoTA for answerable few-shot transfer as well.