Counterfactual Samples Constructing and Training for Commonsense Statements Estimation

TL;DR

This paper introduces CCSG, a novel method that generates counterfactual samples to improve language models' ability to understand and reason about commonsense statements, making them more explainable and sensitive to subtle linguistic cues.

Contribution

The paper proposes CCSG, a model-agnostic approach that enhances commonsense plausibility estimation by generating counterfactual samples for contrastive training.

Findings

CCSG improves performance on nine datasets.

Achieves 3.07% higher accuracy than SOTA methods.

Enhances models' focus on critical words and subtle cues.

Abstract

Plausibility Estimation (PE) plays a crucial role for enabling language models to objectively comprehend the real world. While large language models (LLMs) demonstrate remarkable capabilities in PE tasks but sometimes produce trivial commonsense errors due to the complexity of commonsense knowledge. They lack two key traits of an ideal PE model: a) Language-explainable: relying on critical word segments for decisions, and b) Commonsense-sensitive: detecting subtle linguistic variations in commonsense. To address these issues, we propose a novel model-agnostic method, referred to as Commonsense Counterfactual Samples Generating (CCSG). By training PE models with CCSG, we encourage them to focus on critical words, thereby enhancing both their language-explainable and commonsense-sensitive capabilities. Specifically, CCSG generates counterfactual samples by strategically replacing key words and introducing low-level dropout within sentences. These counterfactual samples are then incorporated into a sentence-level contrastive training framework to further enhance the model's learning process. Experimental results across nine diverse datasets demonstrate the effectiveness of CCSG in addressing commonsense reasoning challenges, with our CCSG method showing 3.07% improvement against the SOTA methods.