Term-community-based topic detection with variable resolution

TL;DR

This paper introduces a community detection method for topic modeling in large text collections that allows adjustable granularity and improved interpretability through semantic embeddings, validated with expert evaluations.

Contribution

It presents a novel variable-resolution community detection approach for topic modeling that enhances interpretability and flexibility compared to existing methods like LDA.

Findings

Effective in detecting meaningful topics at various resolutions

Semantic embeddings facilitate interpretation of term communities

Comparable or superior to LDA in topic quality

Abstract

Network-based procedures for topic detection in huge text collections offer an intuitive alternative to probabilistic topic models. We present in detail a method that is especially designed with the requirements of domain experts in mind. Like similar methods, it employs community detection in term co-occurrence graphs, but it is enhanced by including a resolution parameter that can be used for changing the targeted topic granularity. We also establish a term ranking and use semantic word-embedding for presenting term communities in a way that facilitates their interpretation. We demonstrate the application of our method with a widely used corpus of general news articles and show the results of detailed social-sciences expert evaluations of detected topics at various resolutions. A comparison with topics detected by Latent Dirichlet Allocation is also included. Finally, we discuss factors that influence topic interpretation.