On Commonsense Cues in BERT for Solving Commonsense Tasks

TL;DR

This paper investigates whether BERT utilizes structural commonsense cues in solving tasks like CommonsenseQA, finding that relevant knowledge correlates positively with model accuracy, thus confirming BERT's reliance on commonsense information.

Contribution

It provides a quantitative analysis of the presence and importance of structural commonsense cues in BERT for commonsense tasks, highlighting their role in model performance.

Findings

BERT uses relevant commonsense knowledge in task solving.

Presence of commonsense cues correlates with higher accuracy.

BERT relies on structural cues rather than spurious correlations.

Abstract

BERT has been used for solving commonsense tasks such as CommonsenseQA. While prior research has found that BERT does contain commonsense information to some extent, there has been work showing that pre-trained models can rely on spurious associations (e.g., data bias) rather than key cues in solving sentiment classification and other problems. We quantitatively investigate the presence of structural commonsense cues in BERT when solving commonsense tasks, and the importance of such cues for the model prediction. Using two different measures, we find that BERT does use relevant knowledge for solving the task, and the presence of commonsense knowledge is positively correlated to the model accuracy.