Supervised Understanding of Word Embeddings

TL;DR

This paper introduces a supervised method to interpret and extract meaningful, keyword-specific dimensions from pre-trained word embeddings, enhancing their interpretability and utility in NLP tasks.

Contribution

The study presents a novel supervised projection technique that creates interpretable embedding dimensions and enables cross-lingual dictionary extraction.

Findings

Supervised projections correspond to interpretable keyword features.

Classifier activations align with specific vocabulary subsets.

Using these classifiers improves downstream NLP model accuracy.

Abstract

Pre-trained word embeddings are widely used for transfer learning in natural language processing. The embeddings are continuous and distributed representations of the words that preserve their similarities in compact Euclidean spaces. However, the dimensions of these spaces do not provide any clear interpretation. In this study, we have obtained supervised projections in the form of the linear keyword-level classifiers on word embeddings. We have shown that the method creates interpretable projections of original embedding dimensions. Activations of the trained classifier nodes correspond to a subset of the words in the vocabulary. Thus, they behave similarly to the dictionary features while having the merit of continuous value output. Additionally, such dictionaries can be grown iteratively with multiple rounds by adding expert labels on top-scoring words to an initial collection of the keywords. Also, the same classifiers can be applied to aligned word embeddings in other languages to obtain corresponding dictionaries. In our experiments, we have shown that initializing higher-order networks with these classifier weights gives more accurate models for downstream NLP tasks. We further demonstrate the usefulness of supervised dimensions in revealing the polysemous nature of a keyword of interest by projecting it's embedding using learned classifiers in different sub-spaces.