Benchmarking Commonsense Knowledge Base Population with an Effective Evaluation Dataset

TL;DR

This paper introduces a large-scale, human-annotated dataset for benchmarking commonsense knowledge base population, highlighting the challenge of generalizing reasoning to unseen assertions and providing a new graph-based inductive reasoning model.

Contribution

It presents a new benchmark dataset with high-quality human annotations and proposes a novel graph-based inductive reasoning model for CSKB population tasks.

Findings

Models perform poorly on unseen assertions compared to human performance.

High accuracy models during training do not generalize well to evaluation.

The dataset enables more accurate assessment of commonsense reasoning abilities.

Abstract

Reasoning over commonsense knowledge bases (CSKB) whose elements are in the form of free-text is an important yet hard task in NLP. While CSKB completion only fills the missing links within the domain of the CSKB, CSKB population is alternatively proposed with the goal of reasoning unseen assertions from external resources. In this task, CSKBs are grounded to a large-scale eventuality (activity, state, and event) graph to discriminate whether novel triples from the eventuality graph are plausible or not. However, existing evaluations on the population task are either not accurate (automatic evaluation with randomly sampled negative examples) or of small scale (human annotation). In this paper, we benchmark the CSKB population task with a new large-scale dataset by first aligning four popular CSKBs, and then presenting a high-quality human-annotated evaluation set to probe neural models' commonsense reasoning ability. We also propose a novel inductive commonsense reasoning model that reasons over graphs. Experimental results show that generalizing commonsense reasoning on unseen assertions is inherently a hard task. Models achieving high accuracy during training perform poorly on the evaluation set, with a large gap between human performance. We will make the data publicly available for future contributions. Codes and data are available at https://github.com/HKUST-KnowComp/CSKB-Population.