Stay on Topic, Please: Aligning User Comments to the Content of a News Article

TL;DR

This paper introduces a BERT-based classification method to determine the relevance of user comments to news articles, significantly improving accuracy and addressing the challenge of aligning informal comments with formal news content.

Contribution

The study presents a novel BERTAC model with an ordinal loss function for better comment-article relevance classification, outperforming existing baselines.

Findings

Up to 36% accuracy improvement over baselines

The proposed loss influences the learning process positively

Moderate human agreement indicates the task's difficulty

Abstract

Social scientists have shown that up to 50% if the content posted to a news article have no relation to its journalistic content. In this study we propose a classification algorithm to categorize user comments posted to a new article base don their alignment to its content. The alignment seek to match user comments to an article based on similarity off content, entities in discussion, and topic. We proposed a BERTAC, BAERT-based approach that learn jointly article-comment embeddings and infers the relevance class of comments. We introduce an ordinal classification loss that penalizes the difference between the predicted and true label. We conduct a thorough study to show influence of the proposed loss on the learning process. The results on five representative news outlets show that our approach can learn the comment class with up to 36% average accuracy improvement compering to the baselines, and up to 25% compering to the BA-BC model. BA-BC is out approach that consists of two models aimed to capture dis-jointly the formal language of news articles and the informal language of comments. We also conduct a user study to evaluate human labeling performance to understand the difficulty of the classification task. The user agreement on comment-article alignment is "moderate" per Krippendorff's alpha score, which suggests that the classification task is difficult.