Deriving Contextualised Semantic Features from BERT (and Other Transformer Model) Embeddings

TL;DR

This paper introduces a method to derive contextualized semantic features from BERT embeddings, enhancing interpretability of word meanings in context and revealing how semantic features are represented across BERT's layers.

Contribution

It demonstrates that Binder semantic features can be extracted from BERT embeddings, enabling contextualized semantic analysis and improving interpretability of transformer-based models.

Findings

Binder features can be derived from BERT embeddings.

Semantic features vary across BERT layers.

Contextualized semantic features improve interpretability.

Abstract

Models based on the transformer architecture, such as BERT, have marked a crucial step forward in the field of Natural Language Processing. Importantly, they allow the creation of word embeddings that capture important semantic information about words in context. However, as single entities, these embeddings are difficult to interpret and the models used to create them have been described as opaque. Binder and colleagues proposed an intuitive embedding space where each dimension is based on one of 65 core semantic features. Unfortunately, the space only exists for a small dataset of 535 words, limiting its uses. Previous work (Utsumi, 2018, 2020, Turton, Vinson & Smith, 2020) has shown that Binder features can be derived from static embeddings and successfully extrapolated to a large new vocabulary. Taking the next step, this paper demonstrates that Binder features can be derived from the BERT embedding space. This provides contextualised Binder embeddings, which can aid in understanding semantic differences between words in context. It additionally provides insights into how semantic features are represented across the different layers of the BERT model.