Using Distributional Principles for the Semantic Study of Contextual Language Models

TL;DR

This paper investigates the semantic properties of contextual language models using distributional principles, focusing on substitution-based probing in controlled and open settings to compare static and contextual models.

Contribution

It introduces a novel method for analyzing semantic similarity in language models and adapts it to compare static and contextual models in both controlled and open environments.

Findings

Distributional substitution reveals semantic properties of models

Differences identified between static and contextual models

Method applicable to various language model analyses

Abstract

Many studies were recently done for investigating the properties of contextual language models but surprisingly, only a few of them consider the properties of these models in terms of semantic similarity. In this article, we first focus on these properties for English by exploiting the distributional principle of substitution as a probing mechanism in the controlled context of SemCor and WordNet paradigmatic relations. Then, we propose to adapt the same method to a more open setting for characterizing the differences between static and contextual language models.