JECC: Commonsense Reasoning Tasks Derived from Interactive Fictions

TL;DR

This paper introduces JECC, a new commonsense reasoning dataset derived from interactive fiction gameplay, emphasizing functional knowledge and multi-hop reasoning, and highlighting its challenge to existing models.

Contribution

The paper presents a novel dataset based on interactive fiction that focuses on functional commonsense reasoning and requires less human intervention in its construction.

Findings

Significant performance gap between models and humans (20%).

The dataset is challenging for existing machine reading models.

Focuses on functional rather than factual knowledge.

Abstract

Commonsense reasoning simulates the human ability to make presumptions about our physical world, and it is an essential cornerstone in building general AI systems. We propose a new commonsense reasoning dataset based on human's Interactive Fiction (IF) gameplay walkthroughs as human players demonstrate plentiful and diverse commonsense reasoning. The new dataset provides a natural mixture of various reasoning types and requires multi-hop reasoning. Moreover, the IF game-based construction procedure requires much less human interventions than previous ones. Different from existing benchmarks, our dataset focuses on the assessment of functional commonsense knowledge rules rather than factual knowledge. Hence, in order to achieve higher performance on our tasks, models need to effectively utilize such functional knowledge to infer the outcomes of actions, rather than relying solely on memorizing facts. Experiments show that the introduced dataset is challenging to previous machine reading models as well as the new large language models with a significant 20% performance gap compared to human experts.