Residual Semantic Decomposition of Word Embeddings

TL;DR

This paper presents Residual Semantic Decomposition (RSD), a neural method for decomposing word embeddings to analyze semantic axes and residual information, balancing reconstruction and relational structure.

Contribution

RSD introduces a recursive binary decomposition approach for word embeddings, enabling local semantic axis extraction and residual analysis for ambiguous words.

Findings

RSD separates context anchors from controls in diagnostics.

Residual neighborhoods serve as qualitative diagnostics, not benchmarks.

Ambiguous words are not uniformly high-entropy boundary points.

Abstract

We introduce Residual Semantic Decomposition (RSD), a neural additive decomposition of word embeddings that balances embedding reconstruction with relational structure preservation. RSD supports recursive binary decomposition: each $K=2$ fit extracts a local semantic axis, while residuals expose information not absorbed by that axis. In manually specified paired-context diagnostics over ambiguous words, RSD separates supplied context anchors above shuffled-label controls, but entropy diagnostics show that ambiguous targets are not uniformly high-entropy boundary points in static GloVe. We therefore treat residual neighborhoods as qualitative diagnostics rather than benchmark sense predictions.