Interpretable Syntactic Representations Enable Hierarchical Word Vectors

TL;DR

This paper introduces a method to convert dense word vectors into compact, interpretable syntactic representations, enabling hierarchical word vectors that improve interpretability and performance in benchmarks.

Contribution

The paper presents a novel approach to transform dense word vectors into interpretable syntactic representations and build hierarchical vectors efficiently.

Findings

Syntactic representations align well with human judgment.

Hierarchical vectors outperform original vectors in benchmarks.

The method is computationally efficient.

Abstract

The distributed representations currently used are dense and uninterpretable, leading to interpretations that themselves are relative, overcomplete, and hard to interpret. We propose a method that transforms these word vectors into reduced syntactic representations. The resulting representations are compact and interpretable allowing better visualization and comparison of the word vectors and we successively demonstrate that the drawn interpretations are in line with human judgment. The syntactic representations are then used to create hierarchical word vectors using an incremental learning approach similar to the hierarchical aspect of human learning. As these representations are drawn from pre-trained vectors, the generation process and learning approach are computationally efficient. Most importantly, we find out that syntactic representations provide a plausible interpretation of the vectors and subsequent hierarchical vectors outperform the original vectors in benchmark tests.