Hybrid moderation in the newsroom: Recommending featured posts to content moderators

TL;DR

This paper introduces a hybrid recommender system that combines user and textual features to assist content moderators in selecting featured comments, improving efficiency and transparency in news outlet comment moderation.

Contribution

The paper presents a novel ranking-based recommender system integrating textual and user features for content moderation, with demonstrated improvements in classification and ranking metrics.

Findings

Optimal F1-score of 0.44 on test set

Mean NDCG@5 of 0.87 on validation set

Moderators successfully identified suitable comments in most recommendations

Abstract

Online news outlets are grappling with the moderation of user-generated content within their comment section. We present a recommender system based on ranking class probabilities to support and empower the moderator in choosing featured posts, a time-consuming task. By combining user and textual content features we obtain an optimal classification F1-score of 0.44 on the test set. Furthermore, we observe an optimum mean NDCG@5 of 0.87 on a large set of validation articles. As an expert evaluation, content moderators assessed the output of a random selection of articles by choosing comments to feature based on the recommendations, which resulted in a NDCG score of 0.83. We conclude that first, adding text features yields the best score and second, while choosing featured content remains somewhat subjective, content moderators found suitable comments in all but one evaluated recommendations. We end the paper by analyzing our best-performing model, a step towards transparency and explainability in hybrid content moderation.