Evaluating Trustworthiness of Online News Publishers via Article Classification

TL;DR

This paper presents a classification approach to assess the trustworthiness of online news publishers by analyzing article content, using a dataset of thousands of stories labeled by a journalistic organization, with promising results for practical trust evaluation.

Contribution

It introduces a method for automatically classifying news articles' trustworthiness to evaluate online news sources based on content analysis, leveraging a large labeled dataset.

Findings

High accuracy in classifying trustworthiness levels

Effective in identifying untrustworthy news sources

Supports practical applications for media evaluation

Abstract

The proliferation of low-quality online information in today's era has underscored the need for robust and automatic mechanisms to evaluate the trustworthiness of online news publishers. In this paper, we analyse the trustworthiness of online news media outlets by leveraging a dataset of 4033 news stories from 40 different sources. We aim to infer the trustworthiness level of the source based on the classification of individual articles' content. The trust labels are obtained from NewsGuard, a journalistic organization that evaluates news sources using well-established editorial and publishing criteria. The results indicate that the classification model is highly effective in classifying the trustworthiness levels of the news articles. This research has practical applications in alerting readers to potentially untrustworthy news sources, assisting journalistic organizations in evaluating new or unfamiliar media outlets and supporting the selection of articles for their trustworthiness assessment.