A Framework for Generating Annotated Social Media Corpora with Demographics, Stance, Civility, and Topicality

TL;DR

This paper presents a framework for annotating social media texts with demographic and topical information, enabling socio-technical analysis and scalable annotation via prediction models, demonstrated on Facebook comments about student loans.

Contribution

The paper introduces a comprehensive annotation framework and releases datasets for social media analysis, combining demographic, stance, civility, and topical annotations.

Findings

Annotated Facebook comment datasets for multiple categories.

Effective use of small samples to train models for large-scale annotation.

Facilitates socio-technical analysis of social media discussions.

Abstract

In this paper we introduce a framework for annotating a social media text corpora for various categories. Since, social media data is generated via individuals, it is important to annotate the text for the individuals demographic attributes to enable a socio-technical analysis of the corpora. Furthermore, when analyzing a large data-set we can often annotate a small sample of data and then train a prediction model using this sample to annotate the full data for the relevant categories. We use a case study of a Facebook comment corpora on student loan discussion which was annotated for gender, military affiliation, age-group, political leaning, race, stance, topicalilty, neoliberlistic views and civility of the comment. We release three datasets of Facebook comments for further research at: https://github.com/socialmediaie/StudentDebtFbComments