Anchor-Free Correlated Topic Modeling: Identifiability and Algorithm

TL;DR

This paper introduces an anchor-free, second-order moment-based framework for topic modeling that guarantees identifiability under milder conditions, improving robustness and scalability over traditional anchor-word methods.

Contribution

It proposes a novel anchor-free approach to topic modeling that relies on second-order moments, avoiding the need for anchor words or higher-order statistics, and provides an efficient, scalable algorithm.

Findings

Outperforms prior methods in coherence, similarity count, and clustering accuracy.

Guarantees topic identifiability under milder conditions than anchor-word assumptions.

Algorithm involves only eigen-decomposition and small linear programs.

Abstract

In topic modeling, many algorithms that guarantee identifiability of the topics have been developed under the premise that there exist anchor words -- i.e., words that only appear (with positive probability) in one topic. Follow-up work has resorted to three or higher-order statistics of the data corpus to relax the anchor word assumption. Reliable estimates of higher-order statistics are hard to obtain, however, and the identification of topics under those models hinges on uncorrelatedness of the topics, which can be unrealistic. This paper revisits topic modeling based on second-order moments, and proposes an anchor-free topic mining framework. The proposed approach guarantees the identification of the topics under a much milder condition compared to the anchor-word assumption, thereby exhibiting much better robustness in practice. The associated algorithm only involves one eigen-decomposition and a few small linear programs. This makes it easy to implement and scale up to very large problem instances. Experiments using the TDT2 and Reuters-21578 corpus demonstrate that the proposed anchor-free approach exhibits very favorable performance (measured using coherence, similarity count, and clustering accuracy metrics) compared to the prior art.