Author Clustering and Topic Estimation for Short Texts

TL;DR

This paper introduces a novel model for short text analysis that jointly clusters authors and estimates topics, improving accuracy over traditional methods, demonstrated on political Twitter data to reveal meaningful partisan groups and echo chambers.

Contribution

The paper proposes a new model extending LDA to incorporate user-level dependencies and simultaneous clustering, enhancing topic and author grouping in short texts.

Findings

Model outperforms traditional approaches in topic estimation.

Successfully identifies meaningful political clusters and topics.

Develops a new measure of echo chambers among politicians.

Abstract

Analysis of short text, such as social media posts, is extremely difficult because of their inherent brevity. In addition to classifying topics of such posts, a common downstream task is grouping the authors of these documents for subsequent analyses. We propose a novel model that expands on the Latent Dirichlet Allocation by modeling strong dependence among the words in the same document, with user-level topic distributions. We also simultaneously cluster users, removing the need for post-hoc cluster estimation and improving topic estimation by shrinking noisy user-level topic distributions towards typical values. Our method performs as well as -- or better -- than traditional approaches, and we demonstrate its usefulness on a dataset of tweets from United States Senators, recovering both meaningful topics and clusters that reflect partisan ideology. We also develop a novel measure of echo chambers among these politicians by characterizing insularity of topics discussed by groups of Senators and provide uncertainty quantification.