LOGICAL-COMMONSENSEQA: A Benchmark for Logical Commonsense Reasoning

TL;DR

LOGICAL-COMMONSENSEQA is a benchmark that evaluates logical composition in commonsense reasoning, revealing models' strengths and weaknesses in handling conjunctions, disjunctions, and negations.

Contribution

It introduces a new benchmark that reframes commonsense reasoning as logical composition over atomic statements, enabling more nuanced evaluation.

Findings

Models perform well on conjunctive reasoning.

Moderate performance on disjunctive reasoning.

Sharp performance decline on negation questions.

Abstract

Commonsense reasoning often involves evaluating multiple plausible interpretations rather than selecting a single atomic answer, yet most benchmarks rely on single-label evaluation, obscuring whether statements are jointly plausible, mutually exclusive, or jointly implausible. We introduce LOGICAL-COMMONSENSEQA, a benchmark that reframes commonsense reasoning as logical composition over pairs of atomic statements using plausibility-level operators (AND, OR and NEITHER/NOR). Evaluating instruction-tuned, reasoning-specialized, and fine-tuned models under zero-shot, few-shot, and chain-of-thought prompting, we find that while models perform reasonably on conjunctive and moderately on disjunctive reasoning, performance degrades sharply on negation-based questions. LOGICAL-COMMONSENSEQA exposes fundamental reasoning limitations and provides a controlled framework for advancing compositional commonsense reasoning.