The Computational Anatomy of Humility: Modeling Intellectual Humility in Online Public Discourse

TL;DR

This paper develops computational methods to measure intellectual humility in online discourse, using large language models trained on manually annotated data, aiming to promote healthier, more constructive online discussions.

Contribution

It introduces a novel approach to automatically detect intellectual humility in social media posts, combining manual annotation with LLM-based models, advancing NLP tools for social science research.

Findings

Best model achieves Macro-F1 of 0.64 for IH detection

Model outperforms naive baseline but below human upper bound

Highlights challenges and opportunities in automating IH measurement

Abstract

The ability for individuals to constructively engage with one another across lines of difference is a critical feature of a healthy pluralistic society. This is also true in online discussion spaces like social media platforms. To date, much social media research has focused on preventing ills -- like political polarization and the spread of misinformation. While this is important, enhancing the quality of online public discourse requires not just reducing ills but also promoting foundational human virtues. In this study, we focus on one particular virtue: ``intellectual humility'' (IH), or acknowledging the potential limitations in one's own beliefs. Specifically, we explore the development of computational methods for measuring IH at scale. We manually curate and validate an IH codebook on 350 posts about religion drawn from subreddits and use them to develop LLM-based models for automating this measurement. Our best model achieves a Macro-F1 score of 0.64 across labels (and 0.70 when predicting IH/IA/Neutral at the coarse level), higher than an expected naive baseline of 0.51 (0.32 for IH/IA/Neutral) but lower than a human annotator-informed upper bound of 0.85 (0.83 for IH/IA/Neutral). Our results both highlight the challenging nature of detecting IH online -- opening the door to new directions in NLP research -- and also lay a foundation for computational social science researchers interested in analyzing and fostering more IH in online public discourse.