Enhancing Short-Text Topic Modeling with LLM-Driven Context Expansion and Prefix-Tuned VAEs

TL;DR

This paper introduces a novel method that uses large language models with prefix tuning and variational autoencoders to enhance short-text topic modeling by expanding texts and improving coherence, outperforming existing models.

Contribution

The paper presents a new approach combining LLM-driven context expansion with prefix-tuned VAEs for more effective short-text topic modeling, addressing semantic inconsistency issues.

Findings

Significant performance improvement over state-of-the-art models

Effective handling of data sparsity in short texts

Demonstrated on real-world datasets with extensive experiments

Abstract

Topic modeling is a powerful technique for uncovering hidden themes within a collection of documents. However, the effectiveness of traditional topic models often relies on sufficient word co-occurrence, which is lacking in short texts. Therefore, existing approaches, whether probabilistic or neural, frequently struggle to extract meaningful patterns from such data, resulting in incoherent topics. To address this challenge, we propose a novel approach that leverages large language models (LLMs) to extend short texts into more detailed sequences before applying topic modeling. To further improve the efficiency and solve the problem of semantic inconsistency from LLM-generated texts, we propose to use prefix tuning to train a smaller language model coupled with a variational autoencoder for short-text topic modeling. Our method significantly improves short-text topic modeling performance, as demonstrated by extensive experiments on real-world datasets with extreme data sparsity, outperforming current state-of-the-art topic models.