Inferential Question Answering

TL;DR

This paper introduces Inferential QA, a new challenging question answering task requiring inference from clues, along with the QUIT dataset, revealing current models' limitations in reasoning from indirect evidence.

Contribution

The paper presents a novel inferential question answering task and dataset, highlighting the gaps in current models' ability to perform reasoning from indirect textual clues.

Findings

Retrievers underperform on inferential questions

Rerankers provide limited improvements

LLMs struggle to outperform smaller models in inference tasks

Abstract

Despite extensive research on a wide range of question answering (QA) systems, most existing work focuses on answer containment-i.e., assuming that answers can be directly extracted and/or generated from documents in the corpus. However, some questions require inference, i.e., deriving answers that are not explicitly stated but can be inferred from the available information. We introduce Inferential QA -- a new task that challenges models to infer answers from answer-supporting passages which provide only clues. To study this problem, we construct QUIT (QUestions requiring Inference from Texts) dataset, comprising 7,401 questions and 2.4M passages built from high-convergence human- and machine-authored hints, labeled across three relevance levels using LLM-based answerability and human verification. Through comprehensive evaluation of retrievers, rerankers, and LLM-based readers, we show that methods effective on traditional QA tasks struggle in inferential QA: retrievers underperform, rerankers offer limited gains, and fine-tuning provides inconsistent improvements. Even reasoning-oriented LLMs fail to outperform smaller general-purpose models. These findings reveal that current QA pipelines are not yet ready for inference-based reasoning. Inferential QA thus establishes a new class of QA tasks that move towards understanding and reasoning from indirect textual evidence.