Multi-scale Hybridized Topic Modeling: A Pipeline for Analyzing Unstructured Text Datasets via Topic Modeling

TL;DR

This paper introduces a multi-scale hybridized topic modeling approach that combines NMF and BERTopic to improve the accuracy and efficiency of extracting topics from unstructured interview transcripts, aiding better interpretation and indexing.

Contribution

The paper presents a novel hierarchical pipeline combining NMF and BERTopic for multi-scale topic modeling of unstructured text datasets.

Findings

Effective extraction of hidden topics from interview transcripts.

Enhanced interpretability and indexing of unstructured text.

Promising results on real-world datasets.

Abstract

We propose a multi-scale hybridized topic modeling method to find hidden topics from transcribed interviews more accurately and efficiently than traditional topic modeling methods. Our multi-scale hybridized topic modeling method (MSHTM) approaches data at different scales and performs topic modeling in a hierarchical way utilizing first a classical method, Nonnegative Matrix Factorization, and then a transformer-based method, BERTopic. It harnesses the strengths of both NMF and BERTopic. Our method can help researchers and the public better extract and interpret the interview information. Additionally, it provides insights for new indexing systems based on the topic level. We then deploy our method on real-world interview transcripts and find promising results.