When Numbers Tell Half the Story: Human-Metric Alignment in Topic Model Evaluation

TL;DR

This paper introduces Topic Word Mixing (TWM), a new human evaluation task for assessing topic model quality that complements existing metrics and better aligns with human judgment in specialized domains.

Contribution

The paper presents TWM as a novel human evaluation method for topic models, providing a domain-specific, human-grounded measure of inter-topic distinctness that complements existing intra-topic coherence metrics.

Findings

TWM captures human-perceived topic distinctness effectively.

Word intrusion and coherence metrics often do not align in specialized domains.

TWM aligns with diversity metrics and offers a valuable complement to automated evaluation.

Abstract

Topic models uncover latent thematic structures in text corpora, yet evaluating their quality remains challenging, particularly in specialized domains. Existing methods often rely on automated metrics like topic coherence and diversity, which may not fully align with human judgment. Human evaluation tasks, such as word intrusion, provide valuable insights but are costly and primarily validated on general-domain corpora. This paper introduces Topic Word Mixing (TWM), a novel human evaluation task assessing inter-topic distinctness by testing whether annotators can distinguish between word sets from single or mixed topics. TWM complements word intrusion's focus on intra-topic coherence and provides a human-grounded counterpart to diversity metrics. We evaluate six topic models - both statistical and embedding-based (LDA, NMF, Top2Vec, BERTopic, CFMF, CFMF-emb) - comparing automated metrics with human evaluation methods based on nearly 4,000 annotations from a domain-specific corpus of philosophy of science publications. Our findings reveal that word intrusion and coherence metrics do not always align, particularly in specialized domains, and that TWM captures human-perceived distinctness while appearing to align with diversity metrics. We release the annotated dataset and task generation code. This work highlights the need for evaluation frameworks bridging automated and human assessments, particularly for domain-specific corpora.