Sparse associative memory based on contextual code learning for disambiguating word senses

TL;DR

This paper introduces a biologically inspired supervised method to convert dense language model representations into compressed, interpretable vectors for Word Sense Disambiguation, reducing memory use and improving performance.

Contribution

It presents a novel approach for transforming large pre-trained language model outputs into sparse, interpretable representations specifically for WSD tasks.

Findings

Improved interpretability of word representations.

Reduced memory footprint of language models.

Enhanced WSD performance.

Abstract

In recent literature, contextual pretrained Language Models (LMs) demonstrated their potential in generalizing the knowledge to several Natural Language Processing (NLP) tasks including supervised Word Sense Disambiguation (WSD), a challenging problem in the field of Natural Language Understanding (NLU). However, word representations from these models are still very dense, costly in terms of memory footprint, as well as minimally interpretable. In order to address such issues, we propose a new supervised biologically inspired technique for transferring large pre-trained language model representations into a compressed representation, for the case of WSD. Our produced representation contributes to increase the general interpretability of the framework and to decrease memory footprint, while enhancing performance.