Probing Commonsense Knowledge in Pre-trained Language Models with Sense-level Precision and Expanded Vocabulary

TL;DR

This paper introduces SenseLAMA, a sense-level cloze task using an enriched vocabulary from WordNet, enabling more precise evaluation of commonsense knowledge in pre-trained language models like BERT.

Contribution

It presents a method to augment language models with a grounded sense inventory, allowing sense-level queries without additional training, and introduces the SenseLAMA benchmark for evaluation.

Findings

BERT with WordNet enrichment (SynBERT) learns non-trivial commonsense knowledge.

SynBERT outperforms similarity-based approaches in commonsense tasks.

Enrichment enables finer-grained, sense-level commonsense evaluation.

Abstract

Progress on commonsense reasoning is usually measured from performance improvements on Question Answering tasks designed to require commonsense knowledge. However, fine-tuning large Language Models (LMs) on these specific tasks does not directly evaluate commonsense learned during pre-training. The most direct assessments of commonsense knowledge in pre-trained LMs are arguably cloze-style tasks targeting commonsense assertions (e.g., A pen is used for [MASK].). However, this approach is restricted by the LM's vocabulary available for masked predictions, and its precision is subject to the context provided by the assertion. In this work, we present a method for enriching LMs with a grounded sense inventory (i.e., WordNet) available at the vocabulary level, without further training. This modification augments the prediction space of cloze-style prompts to the size of a large ontology while enabling finer-grained (sense-level) queries and predictions. In order to evaluate LMs with higher precision, we propose SenseLAMA, a cloze-style task featuring verbalized relations from disambiguated triples sourced from WordNet, WikiData, and ConceptNet. Applying our method to BERT, producing a WordNet-enriched version named SynBERT, we find that LMs can learn non-trivial commonsense knowledge from self-supervision, covering numerous relations, and more effectively than comparable similarity-based approaches.