Why Do Neural Language Models Still Need Commonsense Knowledge to Handle Semantic Variations in Question Answering?

TL;DR

This paper investigates the extent of commonsense knowledge in neural language models used for reading comprehension, revealing gaps in understanding and proposing external knowledge integration as a solution.

Contribution

It provides empirical analysis of commonsense knowledge in pretrained MNLMs and demonstrates how external knowledge can improve their handling of semantic variations.

Findings

Large portion of commonsense knowledge is not well-trained in MNLMs.

MNLMs often misinterpret relations and semantic meanings.

Enriching models with external knowledge improves performance on semantic variation tasks.

Abstract

Many contextualized word representations are now learned by intricate neural network models, such as masked neural language models (MNLMs) which are made up of huge neural network structures and trained to restore the masked text. Such representations demonstrate superhuman performance in some reading comprehension (RC) tasks which extract a proper answer in the context given a question. However, identifying the detailed knowledge trained in MNLMs is challenging owing to numerous and intermingled model parameters. This paper provides new insights and empirical analyses on commonsense knowledge included in pretrained MNLMs. First, we use a diagnostic test that evaluates whether commonsense knowledge is properly trained in MNLMs. We observe that a large proportion of commonsense knowledge is not appropriately trained in MNLMs and MNLMs do not often understand the semantic meaning of relations accurately. In addition, we find that the MNLM-based RC models are still vulnerable to semantic variations that require commonsense knowledge. Finally, we discover the fundamental reason why some knowledge is not trained. We further suggest that utilizing an external commonsense knowledge repository can be an effective solution. We exemplify the possibility to overcome the limitations of the MNLM-based RC models by enriching text with the required knowledge from an external commonsense knowledge repository in controlled experiments.