PaCo: Preconditions Attributed to Commonsense Knowledge
Ehsan Qasemi, Filip Ilievski, Muhao Chen, Pedro Szekely

TL;DR
This paper introduces PaCo, a dataset and evaluation tasks to assess whether language models understand circumstantial preconditions in commonsense reasoning, revealing significant gaps compared to human performance.
Contribution
The paper presents a new dataset and evaluation framework for reasoning with circumstantial preconditions, highlighting limitations of current language models in this aspect.
Findings
Existing LMs lag behind humans by 10-30% in understanding preconditions.
The dataset includes 12.4K natural language preconditions for commonsense statements.
Reasoning with preconditions remains an open challenge for current models.
Abstract
Humans can seamlessly reason with circumstantial preconditions of commonsense knowledge. We understand that a glass is used for drinking water, unless the glass is broken or the water is toxic. Despite state-of-the-art (SOTA) language models' (LMs) impressive performance on inferring commonsense knowledge, it is unclear whether they understand the circumstantial preconditions. To address this gap, we propose a novel challenge of reasoning with circumstantial preconditions. We collect a dataset, called PaCo, consisting of 12.4 thousand preconditions of commonsense statements expressed in natural language. Based on this dataset, we create three canonical evaluation tasks and use them to examine the capability of existing LMs to understand situational preconditions. Our results reveal a 10-30% gap between machine and human performance on our tasks, which shows that reasoning with…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Natural Language Processing Techniques · Speech and dialogue systems
