Why Someone Asked "Why": Foil Inference in Human and LLM Question Interpretation

TL;DR

This paper explores how humans and large language models infer the intended foil in 'why' questions, revealing that humans rely on hindsight expectations while LLMs show inconsistent coupling between expectations and foil choices.

Contribution

It introduces a novel investigation into foil inference in human and LLM question interpretation, highlighting differences in their reasoning processes.

Findings

Humans select foils based on hindsight expectations.

LLMs show inconsistent coupling between expectations and foil selection.

Hindsight expectation judgments predict human foil choices effectively.

Abstract

Explanations are inherently contrastive: E happened rather than E' because of C rather than C'. However, these contrasts, or "foils", are rarely mentioned explicitly but have to be inferred in context. Here, we investigate how people select the intended foil E' of a why-question. Participants read vignettes and judged, for each foil, their prior expectation (what will happen next), closeness (what is most similar to what happened), and hindsight expectation (what could have happened instead), as well as which foil they thought the question asker had in mind when they asked the why-question. We found that foil selections were best predicted by hindsight expectation judgments. This suggests that people infer the foil by considering what a question asker finds surprising after the outcome occurred. Since correct foil selection is relevant not only in human-human interaction but also increasingly in dialogues with large language models, we investigated their performance on the same task. The coupling between LLMs' explicit expectation judgments and their foil selections is inconsistent.