The News Comment Gap and Algorithmic Agenda Setting in Online Forums

TL;DR

This study examines the differences between journalist and reader comment preferences on news websites, analyzes how ranking algorithms influence comment visibility, and introduces a new metric to evaluate these algorithms' effects.

Contribution

The paper provides an analysis of the 'News Comment Gap' using a large dataset and introduces FORUM, a new metric for assessing comment ranking algorithms' impact.

Findings

Journalists prefer positive, timely, complex, direct comments.

Readers favor comments similar to article content from elite authors.

Ranking algorithms significantly influence comment visibility based on sentiment and relevance.

Abstract

The disparity between news stories valued by journalists and those preferred by readers, known as the "News Gap", is well-documented. However, the difference in expectations regarding news related user-generated content is less studied. Comment sections, hosted by news websites, are popular venues for reader engagement, yet still subject to editorial decisions. It is thus important to understand journalist vs reader comment preferences and how these are served by various comment ranking algorithms that represent discussions differently. We analyse 1.2 million comments from Austrian newspaper Der Standard to understand the "News Comment Gap" and the effects of different ranking algorithms. We find that journalists prefer positive, timely, complex, direct responses, while readers favour comments similar to article content from elite authors. We introduce the versatile Feature-Oriented Ranking Utility Metric (FORUM) to assess the impact of different ranking algorithms and find dramatic differences in how they prioritise the display of comments by sentiment, topical relevance, lexical diversity, and readability. Journalists can exert substantial influence over the discourse through both curatorial and algorithmic means. Understanding these choices' implications is vital in fostering engaging and civil discussions while aligning with journalistic objectives, especially given the increasing legal scrutiny and societal importance of online discourse.