Proposing Topic Models and Evaluation Frameworks for Analyzing Associations with External Outcomes: An Application to Leadership Analysis Using Large-Scale Corporate Review Data

TL;DR

This paper introduces a novel topic modeling method using large language models and an evaluation framework to analyze associations with external outcomes, demonstrated through leadership analysis in corporate review data.

Contribution

It proposes a new approach combining large language models and tailored evaluation criteria to improve topic interpretability, specificity, and polarity consistency for external outcome analysis.

Findings

Proposed method outperforms existing approaches in interpretability and consistency.

Topics generated show higher explanatory power for external outcomes.

Framework effectively evaluates topic quality based on specificity and polarity.

Abstract

Analyzing topics extracted from text data in relation to external outcomes is important across fields such as computational social science and organizational research. However, existing topic modeling methods struggle to simultaneously achieve interpretability, topic specificity (alignment with concrete actions or characteristics), and polarity stance consistency (absence of mixed positive and negative evaluations within a topic). Focusing on leadership analysis using corporate review data, this study proposes a method leveraging large language models to generate topics that satisfy these properties, along with an evaluation framework tailored to external outcome analysis. The framework explicitly incorporates topic specificity and polarity stance consistency as evaluation criteria and examines automated evaluation methods based on existing metrics. Using employee reviews from OpenWork, a major corporate review platform in Japan, the proposed method achieves improved interpretability, specificity, and polarity consistency compared to existing approaches. In analyses of external outcomes such as employee morale, it also produces topics with higher explanatory power. These results suggest that the proposed method and evaluation framework provide a generalized approach for topic analysis in applications involving external outcomes.