Improving Neural Topic Models using Knowledge Distillation

TL;DR

This paper introduces a modular knowledge distillation framework that enhances neural topic models by integrating pretrained transformers, leading to significant improvements in topic coherence and interpretability across different architectures.

Contribution

The authors propose a novel, adaptable knowledge distillation method that can be applied to any neural topic model to improve its quality and coherence.

Findings

Achieved state-of-the-art topic coherence scores.

Improved performance both overall and in aligned topic comparisons.

Demonstrated applicability across different neural architectures.

Abstract

Topic models are often used to identify human-interpretable topics to help make sense of large document collections. We use knowledge distillation to combine the best attributes of probabilistic topic models and pretrained transformers. Our modular method can be straightforwardly applied with any neural topic model to improve topic quality, which we demonstrate using two models having disparate architectures, obtaining state-of-the-art topic coherence. We show that our adaptable framework not only improves performance in the aggregate over all estimated topics, as is commonly reported, but also in head-to-head comparisons of aligned topics.