Multi-hop Inference for Sentence-level TextGraphs: How Challenging is Meaningfully Combining Information for Science Question Answering?

TL;DR

This paper investigates the difficulty of multi-hop inference in science question answering, revealing that semantic drift severely limits the effective aggregation of multiple facts in sentence-level graphs.

Contribution

It empirically characterizes the challenges of multi-hop inference, quantifies semantic drift, and identifies scenarios that improve information aggregation quality.

Findings

Semantic drift ranges from 0.04% to 3%.

Building or traversing sentence graphs is highly challenging.

Certain scenarios can improve the likelihood of meaningful information combination.

Abstract

Question Answering for complex questions is often modeled as a graph construction or traversal task, where a solver must build or traverse a graph of facts that answer and explain a given question. This "multi-hop" inference has been shown to be extremely challenging, with few models able to aggregate more than two facts before being overwhelmed by "semantic drift", or the tendency for long chains of facts to quickly drift off topic. This is a major barrier to current inference models, as even elementary science questions require an average of 4 to 6 facts to answer and explain. In this work we empirically characterize the difficulty of building or traversing a graph of sentences connected by lexical overlap, by evaluating chance sentence aggregation quality through 9,784 manually-annotated judgments across knowledge graphs built from three free-text corpora (including study guides and Simple Wikipedia). We demonstrate semantic drift tends to be high and aggregation quality low, at between 0.04% and 3%, and highlight scenarios that maximize the likelihood of meaningfully combining information.