A Reproducible Framework for Neural Topic Modeling in Focus Group Analysis

TL;DR

This paper introduces a reproducible framework for applying neural topic modeling to focus group data, demonstrating improved coherence and interpretability through systematic hyperparameter tuning and validation.

Contribution

It presents a systematic, reproducible approach for neural topic modeling in qualitative research, including hyperparameter exploration and multi-criteria evaluation for focus group analysis.

Findings

Transformer-based models outperform LDA in coherence for small datasets

Stability metrics and coherence can diverge, requiring multi-criteria evaluation

Reproducible framework and code are provided for future research

Abstract

Focus group discussions generate rich qualitative data but their analysis traditionally relies on labor-intensive manual coding that limits scalability and reproducibility. We present a systematic framework for applying BERTopic to focus group transcripts using data from ten focus groups exploring HPV vaccine perceptions in Tunisia (1,075 utterances). We conducted comprehensive hyperparameter exploration across 27 configurations, evaluating each through bootstrap stability analysis, performance metrics, and comparison with LDA baseline. Bootstrap analysis revealed that stability metrics (NMI and ARI) exhibited strong disagreement (r = -0.691) and showed divergent relationships with coherence, demonstrating that stability is multifaceted rather than monolithic. Our multi-criteria selection framework yielded a 7-topic model achieving 18\% higher coherence than optimized LDA (0.573 vs. 0.486) with interpretable topics validated through independent human evaluation (ICC = 0.700, weighted Cohen's kappa = 0.678). These findings demonstrate that transformer-based topic modeling can extract interpretable themes from small focus group transcript corpora when systematically configured and validated, while revealing that quality metrics capture distinct, sometimes conflicting constructs requiring multi-criteria evaluation. We provide complete documentation and code to support reproducibility.