Short text classification with machine learning in the social sciences: The case of climate change on Twitter

TL;DR

This study evaluates machine learning classifiers for short text classification in social sciences, specifically analyzing Twitter data on climate change, highlighting the effectiveness of traditional methods over deep learning in resource-constrained scenarios.

Contribution

It compares the performance of various machine learning classifiers on small, imbalanced datasets in social science research, demonstrating the efficiency of traditional methods.

Findings

Supervised machine learning outperforms lexicons, especially with higher class balance.

Traditional methods like logistic regression and random forest perform comparably to deep learning.

Traditional methods require less training time and computational resources.

Abstract

To analyse large numbers of texts, social science researchers are increasingly confronting the challenge of text classification. When manual labeling is not possible and researchers have to find automatized ways to classify texts, computer science provides a useful toolbox of machine-learning methods whose performance remains understudied in the social sciences. In this article, we compare the performance of the most widely used text classifiers by applying them to a typical research scenario in social science research: a relatively small labeled dataset with infrequent occurrence of categories of interest, which is a part of a large unlabeled dataset. As an example case, we look at Twitter communication regarding climate change, a topic of increasing scholarly interest in interdisciplinary social science research. Using a novel dataset including 5,750 tweets from various international organizations regarding the highly ambiguous concept of climate change, we evaluate the performance of methods in automatically classifying tweets based on whether they are about climate change or not. In this context, we highlight two main findings. First, supervised machine-learning methods perform better than state-of-the-art lexicons, in particular as class balance increases. Second, traditional machine-learning methods, such as logistic regression and random forest, perform similarly to sophisticated deep-learning methods, whilst requiring much less training time and computational resources. The results have important implications for the analysis of short texts in social science research.