Toward Incorporation of Relevant Documents in word2vec

TL;DR

This paper proposes a neural-based explicit word representation method inspired by word2vec, which improves interpretability and incorporates local document information to enhance word similarity tasks in information retrieval.

Contribution

It introduces a new explicit word embedding method that maintains effectiveness while enabling the integration of local document information for IR tasks.

Findings

The proposed explicit representation outperforms existing explicit methods in word similarity ranking.

The method retains the effectiveness of the Skip-Gram model while offering interpretability.

Initial results show promising integration of local document information into global embeddings.

Abstract

Recent advances in neural word embedding provide significant benefit to various information retrieval tasks. However as shown by recent studies, adapting the embedding models for the needs of IR tasks can bring considerable further improvements. The embedding models in general define the term relatedness by exploiting the terms' co-occurrences in short-window contexts. An alternative (and well-studied) approach in IR for related terms to a query is using local information i.e. a set of top-retrieved documents. In view of these two methods of term relatedness, in this work, we report our study on incorporating the local information of the query in the word embeddings. One main challenge in this direction is that the dense vectors of word embeddings and their estimation of term-to-term relatedness remain difficult to interpret and hard to analyze. As an alternative, explicit word representations propose vectors whose dimensions are easily interpretable, and recent methods show competitive performance to the dense vectors. We introduce a neural-based explicit representation, rooted in the conceptual ideas of the word2vec Skip-Gram model. The method provides interpretable explicit vectors while keeping the effectiveness of the Skip-Gram model. The evaluation of various explicit representations on word association collections shows that the newly proposed method out- performs the state-of-the-art explicit representations when tasked with ranking highly similar terms. Based on the introduced ex- plicit representation, we discuss our approaches on integrating local documents in globally-trained embedding models and discuss the preliminary results.