Interpretable Neural Embeddings with Sparse Self-Representation

TL;DR

This paper introduces a novel neural network-based method for learning sparse, interpretable word embeddings that outperform benchmarks on multiple downstream tasks.

Contribution

It proposes a new approach linking data self-representation with shallow neural networks to improve interpretability and stability of sparse word embeddings.

Findings

Embeddings achieve comparable or better interpretability than baselines.

Our method performs competitively on downstream NLP tasks.

Outperforms benchmark embeddings on most evaluated tasks.

Abstract

Interpretability benefits the theoretical understanding of representations. Existing word embeddings are generally dense representations. Hence, the meaning of latent dimensions is difficult to interpret. This makes word embeddings like a black-box and prevents them from being human-readable and further manipulation. Many methods employ sparse representation to learn interpretable word embeddings for better interpretability. However, they also suffer from the unstable issue of grouped selection in $\ell1$ and online dictionary learning. Therefore, they tend to yield different results each time. To alleviate this challenge, we propose a novel method to associate data self-representation with a shallow neural network to learn expressive, interpretable word embeddings. In experiments, we report that the resulting word embeddings achieve comparable and even slightly better interpretability than baseline embeddings. Besides, we also evaluate that our approach performs competitively well on all downstream tasks and outperforms benchmark embeddings on a majority of them.