Evaluating topic coherence measures

TL;DR

This paper assesses various topic coherence measures, including novel ones from scientific philosophy that evaluate complex word subsets, to improve the interpretability of topic models.

Contribution

It introduces and applies coherence measures from scientific philosophy that score complex word subsets, expanding beyond pairwise word evaluations.

Findings

New coherence measures effectively distinguish better topics

Complex subset scoring improves topic interpretability

First application of philosophical coherence measures to topic modeling

Abstract

Topic models extract representative word sets - called topics - from word counts in documents without requiring any semantic annotations. Topics are not guaranteed to be well interpretable, therefore, coherence measures have been proposed to distinguish between good and bad topics. Studies of topic coherence so far are limited to measures that score pairs of individual words. For the first time, we include coherence measures from scientific philosophy that score pairs of more complex word subsets and apply them to topic scoring.