Whose Values? Measuring the (Subjective) Expression of Basic Human Values in Social Media

TL;DR

This paper develops a personalized framework using large-language models to measure and classify subjective expressions of human values in social media posts, accounting for individual differences in perception.

Contribution

It introduces a scalable, personalized approach to measure Schwartz human values in social media, addressing subjectivity and low inter-rater agreement.

Findings

Personalized models outperform non-personalized ones in aligning with individual annotations.

Low inter-rater agreement highlights the subjectivity in value annotation.

The framework effectively captures individual differences in value expression perception.

Abstract

The value alignment of sociotechnical systems has become a central debate, but progress depends on how human values are perceived in the content these systems surface and how such perceptions can be measured at scale. Social media platforms are a prominent class of sociotechnical systems where algorithmic curation shapes exposure to value-laden content at scale. Large-language models offer new opportunities for measuring expressions of human values (e.g., humility or equality) in social media data, but value expressions can be subjective: different people will annotate the same post with different values. In this paper, we draw on the Schwartz value system as a broadly encompassing and theoretically grounded set of basic human values, and introduce a framework to personalize the measurement of expressions of Schwartz values in social media posts at scale. We collect 32,370 ground truth value expression annotations from N=1,079 people on 5,211 social media posts representative of real users' feeds. Due to the subjectivity of the task, we observe low levels of inter-rater agreement between people, and low agreement between human raters and LLM-based methods. In response, we construct a personalization architecture for classifying value expressions by learning from a small number of highly informative calibration annotations per user. In evaluation, we find that modeling these differences successfully yields value expression predictions that people agree with more than they agree with other people. These results contribute new methods and understanding for the measurement of human values in social media data.