DiscoSense: Commonsense Reasoning with Discourse Connectives

TL;DR

DiscoSense is a new benchmark designed to evaluate commonsense reasoning by understanding discourse connectives, using adversarial techniques to generate challenging distractors, revealing current models' limitations.

Contribution

The paper introduces DiscoSense, a novel benchmark for commonsense reasoning with discourse connectives, and proposes Conditional Adversarial Filtering to create challenging distractors.

Findings

State-of-the-art models perform poorly on DiscoSense

DiscoSense effectively evaluates next-generation reasoning systems

The dataset highlights gaps in current language model capabilities

Abstract

We present DiscoSense, a benchmark for commonsense reasoning via understanding a wide variety of discourse connectives. We generate compelling distractors in DiscoSense using Conditional Adversarial Filtering, an extension of Adversarial Filtering that employs conditional generation. We show that state-of-the-art pre-trained language models struggle to perform well on DiscoSense, which makes this dataset ideal for evaluating next-generation commonsense reasoning systems.