Uncovering the Internet's Hidden Values: An Empirical Study of Desirable Behavior Using Highly-Upvoted Content on Reddit

TL;DR

This study analyzes highly-upvoted Reddit comments over two years to extract community values, revealing limitations of existing prosociality models and highlighting the need for nuanced desirability measures.

Contribution

It introduces a large-scale, LLM-based approach to identify community values from upvoted content, uncovering new values and assessing existing models' adequacy.

Findings

Existing models captured only 18% of the values.

The approach identified 64 and 72 macro, meso, micro values in 2016 and 2022.

The method uncovered new community values beyond prosocial measures.

Abstract

A major task for moderators of online spaces is norm-setting, essentially creating shared norms for user behavior in their communities. Platform design principles emphasize the importance of highlighting norm-adhering examples and explicitly stating community norms. However, norms and values vary between communities and go beyond content-level attributes, making it challenging for platforms and researchers to provide automated ways to identify desirable behavior to be highlighted. Current automated approaches to detect desirability are limited to measures of prosocial behavior, but we do not know whether these measures fully capture the spectrum of what communities value. In this paper, we use upvotes, which express community approval, as a proxy for desirability and examine 16,000 highly-upvoted comments across 80 popular sub-communities on Reddit. Using a large language model, we extract values from these comments across two years (2016 and 2022) and compile 64 and 72 $\textit{macro}$, $\textit{meso}$, and $\textit{micro}$ values for 2016 and 2022 respectively, based on their frequency across communities. Furthermore, we find that existing computational models for measuring prosociality were inadequate to capture on average $82\%$ of the values we extracted. Finally, we show that our approach can not only extract most of the qualitatively-identified values from prior taxonomies, but also uncover new values that are actually encouraged in practice. Our findings highlight the need for nuanced models of desirability that go beyond preexisting prosocial measures. This work has implications for improving moderator understanding of their community values and provides a framework that can supplement qualitative approaches with larger-scale content analyses.