Causally Denoise Word Embeddings Using Half-Sibling Regression

TL;DR

This paper introduces a causal inference-based postprocessing method for word embeddings using Half-Sibling Regression, which effectively removes confounding noise and improves performance on lexical and sentiment analysis tasks.

Contribution

It presents a novel, interpretable postprocessing scheme for word vectors grounded in causal inference, outperforming previous methods.

Findings

Achieves state-of-the-art results on lexical evaluation tasks

Improves downstream sentiment analysis performance

Offers interpretability and transparency in word vector postprocessing

Abstract

Distributional representations of words, also known as word vectors, have become crucial for modern natural language processing tasks due to their wide applications. Recently, a growing body of word vector postprocessing algorithm has emerged, aiming to render off-the-shelf word vectors even stronger. In line with these investigations, we introduce a novel word vector postprocessing scheme under a causal inference framework. Concretely, the postprocessing pipeline is realized by Half-Sibling Regression (HSR), which allows us to identify and remove confounding noise contained in word vectors. Compared to previous work, our proposed method has the advantages of interpretability and transparency due to its causal inference grounding. Evaluated on a battery of standard lexical-level evaluation tasks and downstream sentiment analysis tasks, our method reaches state-of-the-art performance.