Evaluating Commonsense in Pre-trained Language Models

TL;DR

This paper evaluates the commonsense reasoning abilities of prominent pre-trained language models using seven benchmarks, revealing their surface-level understanding and highlighting areas for improvement in inference and robustness.

Contribution

It introduces a comprehensive evaluation of commonsense in models like GPT, BERT, XLNet, and RoBERTa, and releases the CATs benchmark dataset for future research.

Findings

Models perform well with language modeling objectives and larger datasets.

Current models struggle with inference-heavy tasks.

Models show confusion on correlated test cases, indicating surface-level commonsense learning.

Abstract

Contextualized representations trained over large raw text data have given remarkable improvements for NLP tasks including question answering and reading comprehension. There have been works showing that syntactic, semantic and word sense knowledge are contained in such representations, which explains why they benefit such tasks. However, relatively little work has been done investigating commonsense knowledge contained in contextualized representations, which is crucial for human question answering and reading comprehension. We study the commonsense ability of GPT, BERT, XLNet, and RoBERTa by testing them on seven challenging benchmarks, finding that language modeling and its variants are effective objectives for promoting models' commonsense ability while bi-directional context and larger training set are bonuses. We additionally find that current models do poorly on tasks require more necessary inference steps. Finally, we test the robustness of models by making dual test cases, which are correlated so that the correct prediction of one sample should lead to correct prediction of the other. Interestingly, the models show confusion on these test cases, which suggests that they learn commonsense at the surface rather than the deep level. We release a test set, named CATs publicly, for future research.