SituatedGen: Incorporating Geographical and Temporal Contexts into Generative Commonsense Reasoning

TL;DR

SituatedGen introduces a new dataset and task for evaluating how well language models incorporate geographical and temporal contexts into commonsense reasoning, revealing current models' limitations compared to humans.

Contribution

The paper formalizes the SituatedGen task, creates a new dataset of contrastive sentence pairs with geographical and temporal contexts, and evaluates models' performance on this challenging task.

Findings

Models struggle with contextual commonsense reasoning.

Human performance significantly exceeds that of current models.

The dataset provides a benchmark for future research.

Abstract

Recently, commonsense reasoning in text generation has attracted much attention. Generative commonsense reasoning is the task that requires machines, given a group of keywords, to compose a single coherent sentence with commonsense plausibility. While existing datasets targeting generative commonsense reasoning focus on everyday scenarios, it is unclear how well machines reason under specific geographical and temporal contexts. We formalize this challenging task as SituatedGen, where machines with commonsense should generate a pair of contrastive sentences given a group of keywords including geographical or temporal entities. We introduce a corresponding English dataset consisting of 8,268 contrastive sentence pairs, which are built upon several existing commonsense reasoning benchmarks with minimal manual labor. Experiments show that state-of-the-art generative language models struggle to generate sentences with commonsense plausibility and still lag far behind human performance. Our dataset is publicly available at https://github.com/yunx-z/situated_gen.