Understanding and Improving Multi-Sense Word Embeddings via Extended Robust Principal Component Analysis

TL;DR

This paper introduces Ex-RPCA, a novel dimensionality reduction technique that detects pseudo multi senses in unsupervised multi-sense word embeddings and improves their quality, leading to better performance on word similarity tasks.

Contribution

The paper proposes Ex-RPCA, a new principal analysis method for detecting pseudo and real multi senses in word embeddings, and demonstrates its effectiveness in enhancing embedding quality.

Findings

Pseudo multi senses are systematically generated in unsupervised embeddings.

Ex-RPCA effectively detects pseudo multi senses.

Improved embeddings outperform original ones by 5.6 points on SCWS.

Abstract

Unsupervised learned representations of polysemous words generate a large of pseudo multi senses since unsupervised methods are overly sensitive to contextual variations. In this paper, we address the pseudo multi-sense detection for word embeddings by dimensionality reduction of sense pairs. We propose a novel principal analysis method, termed Ex-RPCA, designed to detect both pseudo multi senses and real multi senses. With Ex-RPCA, we empirically show that pseudo multi senses are generated systematically in unsupervised method. Moreover, the multi-sense word embeddings can by improved by a simple linear transformation based on Ex-RPCA. Our improved word embedding outperform the original one by 5.6 points on Stanford contextual word similarity (SCWS) dataset. We hope our simple yet effective approach will help the linguistic analysis of multi-sense word embeddings in the future.