RADio -- Rank-Aware Divergence Metrics to Measure Normative Diversity in News Recommendations

TL;DR

RADio introduces a rank-aware Jensen Shannon divergence framework to measure normative diversity in news recommendations, capturing social science perspectives and distributional shifts for improved evaluation.

Contribution

The paper presents RADio, a novel metrics framework that incorporates rank-awareness and normative concepts to evaluate diversity in news recommendation systems.

Findings

RADio effectively reflects normative diversity concepts in news recommendations.

It captures distributional shifts rather than just point estimates.

Provides insightful metrics for news recommender system design.

Abstract

In traditional recommender system literature, diversity is often seen as the opposite of similarity, and typically defined as the distance between identified topics, categories or word models. However, this is not expressive of the social science's interpretation of diversity, which accounts for a news organization's norms and values and which we here refer to as normative diversity. We introduce RADio, a versatile metrics framework to evaluate recommendations according to these normative goals. RADio introduces a rank-aware Jensen Shannon (JS) divergence. This combination accounts for (i) a user's decreasing propensity to observe items further down a list and (ii) full distributional shifts as opposed to point estimates. We evaluate RADio's ability to reflect five normative concepts in news recommendations on the Microsoft News Dataset and six (neural) recommendation algorithms, with the help of our metadata enrichment pipeline. We find that RADio provides insightful estimates that can potentially be used to inform news recommender system design.