Distributed Asymmetric Allocation: A Topic Model for Large Imbalanced Corpora in Social Sciences

TL;DR

This paper introduces Distributed Asymmetric Allocation (DAA), a new topic model designed for large, imbalanced social science corpora, which improves speed and accuracy over traditional LDA by optimizing Dirichlet priors.

Contribution

The paper presents DAA, a novel topic model that integrates multiple algorithms to efficiently identify important topics in large, imbalanced datasets, outperforming LDA.

Findings

DAA classifies sentences more accurately than LDA.

DAA operates faster than traditional LDA.

Optimizing Dirichlet priors enhances content analysis accuracy.

Abstract

Social scientists employ latent Dirichlet allocation (LDA) to find highly specific topics in large corpora, but they often struggle in this task because (1) LDA, in general, takes a significant amount of time to fit on large corpora; (2) unsupervised LDA fragments topics into sub-topics in short documents; (3) semi-supervised LDA fails to identify specific topics defined using seed words. To solve these problems, I have developed a new topic model called distributed asymmetric allocation (DAA) that integrates multiple algorithms for efficiently identifying sentences about important topics in large corpora. I evaluate the ability of DAA to identify politically important topics by fitting it to the transcripts of speeches at the United Nations General Assembly between 1991 and 2017. The results show that DAA can classify sentences significantly more accurately and quickly than LDA thanks to the new algorithms. More generally, the results demonstrate that it is important for social scientists to optimize Dirichlet priors of LDA to perform content analysis accurately.