Models for Predicting Community-Specific Interest in News Articles

TL;DR

This study develops content-based models to predict community interest in news articles on Reddit, demonstrating high accuracy but highlighting challenges with feature degradation over time and recommending hierarchical classification and retraining strategies.

Contribution

The paper introduces models that predict community-specific interest using only article content features and analyzes their temporal robustness and generalization challenges.

Findings

Models achieve ROC AUC between 0.81 and 1.0 in classifying community interest.

Feature groups degrade differently over time, affecting model performance.

Hierarchical classifiers and retraining are recommended for better long-term accuracy.

Abstract

In this work, we ask two questions: 1. Can we predict the type of community interested in a news article using only features from the article content? and 2. How well do these models generalize over time? To answer these questions, we compute well-studied content-based features on over 60K news articles from 4 communities on reddit.com. We train and test models over three different time periods between 2015 and 2017 to demonstrate which features degrade in performance the most due to concept drift. Our models can classify news articles into communities with high accuracy, ranging from 0.81 ROC AUC to 1.0 ROC AUC. However, while we can predict the community-specific popularity of news articles with high accuracy, practitioners should approach these models carefully. Predictions are both community-pair dependent and feature group dependent. Moreover, these feature groups generalize over time differently, with some only degrading slightly over time, but others degrading greatly. Therefore, we recommend that community-interest predictions are done in a hierarchical structure, where multiple binary classifiers can be used to separate community pairs, rather than a traditional multi-class model. Second, these models should be retrained over time based on accuracy goals and the availability of training data.