Industry-Aligned Granular Topic Modeling

TL;DR

This paper introduces TIDE, a novel granular topic modeling framework leveraging large language models, designed for industrial applications, with superior performance demonstrated through extensive experiments on diverse datasets.

Contribution

The paper presents TIDE, a new granular topic modeling method based on LLMs, with additional functionalities for business scenarios, and demonstrates its effectiveness over existing methods.

Findings

TIDE outperforms modern topic modeling methods in experiments.

Auxiliary components support industrial business scenarios effectively.

Framework is open-sourced for community use.

Abstract

Topic modeling has extensive applications in text mining and data analysis across various industrial sectors. Although the concept of granularity holds significant value for business applications by providing deeper insights, the capability of topic modeling methods to produce granular topics has not been thoroughly explored. In this context, this paper introduces a framework called TIDE, which primarily provides a novel granular topic modeling method based on large language models (LLMs) as a core feature, along with other useful functionalities for business applications, such as summarizing long documents, topic parenting, and distillation. Through extensive experiments on a variety of public and real-world business datasets, we demonstrate that TIDE's topic modeling approach outperforms modern topic modeling methods, and our auxiliary components provide valuable support for dealing with industrial business scenarios. The TIDE framework is currently undergoing the process of being open sourced.