Estimating Commonsense Plausibility through Semantic Shifts

TL;DR

ComPaSS is a discriminative framework that measures semantic shifts caused by augmentations to evaluate commonsense plausibility, outperforming generative methods across various models and tasks.

Contribution

Introduces ComPaSS, a novel discriminative approach for fine-grained commonsense plausibility estimation based on semantic shifts, outperforming existing generative methods.

Findings

ComPaSS outperforms baselines on multiple plausibility tasks.

VLMs outperform LMs when used with ComPaSS.

Contrastive pre-training enhances semantic nuance detection.

Abstract

Commonsense plausibility estimation is critical for evaluating language models (LMs), yet existing generative approaches--reliant on likelihoods or verbalized judgments--struggle with fine-grained discrimination. In this paper, we propose ComPaSS, a novel discriminative framework that quantifies commonsense plausibility by measuring semantic shifts when augmenting sentences with commonsense-related information. Plausible augmentations induce minimal shifts in semantics, while implausible ones result in substantial deviations. Evaluations on two types of fine-grained commonsense plausibility estimation tasks across different backbones, including LLMs and vision-language models (VLMs), show that ComPaSS consistently outperforms baselines. It demonstrates the advantage of discriminative approaches over generative methods in fine-grained commonsense plausibility evaluation. Experiments also show that (1) VLMs yield superior performance to LMs, when integrated with ComPaSS, on vision-grounded commonsense tasks. (2) contrastive pre-training sharpens backbone models' ability to capture semantic nuances, thereby further enhancing ComPaSS.