SensePOLAR: Word sense aware interpretability for pre-trained contextual word embeddings

TL;DR

SensePOLAR enhances interpretability of contextual word embeddings by incorporating word sense awareness, enabling more nuanced understanding without sacrificing performance on standard NLP benchmarks.

Contribution

It introduces SensePOLAR, a novel extension of POLAR, that provides sense-aware interpretability for pre-trained contextual embeddings, addressing polysemy limitations.

Findings

Achieves interpretability comparable to original embeddings.

Successfully distinguishes word senses in interpretability.

Performs well on GLUE and SQuAD benchmarks.

Abstract

Adding interpretability to word embeddings represents an area of active research in text representation. Recent work has explored thepotential of embedding words via so-called polar dimensions (e.g. good vs. bad, correct vs. wrong). Examples of such recent approaches include SemAxis, POLAR, FrameAxis, and BiImp. Although these approaches provide interpretable dimensions for words, they have not been designed to deal with polysemy, i.e. they can not easily distinguish between different senses of words. To address this limitation, we present SensePOLAR, an extension of the original POLAR framework that enables word-sense aware interpretability for pre-trained contextual word embeddings. The resulting interpretable word embeddings achieve a level of performance that is comparable to original contextual word embeddings across a variety of natural language processing tasks including the GLUE and SQuAD benchmarks. Our work removes a fundamental limitation of existing approaches by offering users sense aware interpretations for contextual word embeddings.