SituatedQA: Incorporating Extra-Linguistic Contexts into QA

TL;DR

SituatedQA introduces a new dataset for question answering that emphasizes the importance of extra-linguistic context like time and location, revealing current models' struggles with context-dependent answers and highlighting the need for more context-aware systems.

Contribution

The paper presents SituatedQA, a novel open-retrieval QA dataset incorporating temporal and geographical contexts, and analyzes the limitations of existing models in handling context-dependent questions.

Findings

Models perform poorly on context-dependent questions.

Existing models show a 15-point accuracy drop with updated evidence.

Many questions in existing datasets are context-sensitive, requiring extra-linguistic information.

Abstract

Answers to the same question may change depending on the extra-linguistic contexts (when and where the question was asked). To study this challenge, we introduce SituatedQA, an open-retrieval QA dataset where systems must produce the correct answer to a question given the temporal or geographical context. To construct SituatedQA, we first identify such questions in existing QA datasets. We find that a significant proportion of information seeking questions have context-dependent answers (e.g., roughly 16.5% of NQ-Open). For such context-dependent questions, we then crowdsource alternative contexts and their corresponding answers. Our study shows that existing models struggle with producing answers that are frequently updated or from uncommon locations. We further quantify how existing models, which are trained on data collected in the past, fail to generalize to answering questions asked in the present, even when provided with an updated evidence corpus (a roughly 15 point drop in accuracy). Our analysis suggests that open-retrieval QA benchmarks should incorporate extra-linguistic context to stay relevant globally and in the future. Our data, code, and datasheet are available at https://situatedqa.github.io/ .