LGDE: Local Graph-based Dictionary Expansion

TL;DR

LGDE is a novel method that uses manifold learning and network science to discover semantic neighborhoods of words, enabling more effective dictionary expansion for information retrieval tasks.

Contribution

LGDE introduces a graph diffusion-based approach for local community detection in word similarity graphs, enhancing dictionary expansion beyond direct similarities.

Findings

LGDE improves keyword expansion performance over traditional methods.

The method effectively captures complex nonlinear semantic relationships.

LGDE is validated on multiple corpora and a real-world communication science case.

Abstract

We present Local Graph-based Dictionary Expansion (LGDE), a method for data-driven discovery of the semantic neighbourhood of words using tools from manifold learning and network science. At the heart of LGDE lies the creation of a word similarity graph from the geometry of word embeddings followed by local community detection based on graph diffusion. The diffusion in the local graph manifold allows the exploration of the complex nonlinear geometry of word embeddings to capture word similarities based on paths of semantic association, over and above direct pairwise similarities. Exploiting such semantic neighbourhoods enables the expansion of dictionaries of pre-selected keywords, an important step for tasks in information retrieval, such as database queries and online data collection. We validate LGDE on two user-generated English-language corpora and show that LGDE enriches the list of keywords with improved performance relative to methods based on direct word similarities or co-occurrences. We further demonstrate our method through a real-world use case from communication science, where LGDE is evaluated quantitatively on the expansion of a conspiracy-related dictionary from online data collected and analysed by domain experts. Our empirical results and expert user assessment indicate that LGDE expands the seed dictionary with more useful keywords due to the manifold-learning-based similarity network.