Construction of Large-Scale Misinformation Labeled Datasets from Social Media Discourse using Label Refinement

TL;DR

This paper presents a novel framework for constructing large-scale misinformation datasets from social media by leveraging weak labels from news sources and refining them through model-guided techniques, reducing human effort.

Contribution

It introduces a label refinement framework that uses self-training and social context to improve misinformation dataset quality with minimal human intervention.

Findings

Effective in identifying and correcting inaccurate labels.

Successfully applied to COVID-19 vaccine misinformation dataset.

Achieves large-scale dataset construction with reduced manual labeling.

Abstract

Malicious accounts spreading misinformation has led to widespread false and misleading narratives in recent times, especially during the COVID-19 pandemic, and social media platforms struggle to eliminate these contents rapidly. This is because adapting to new domains requires human intensive fact-checking that is slow and difficult to scale. To address this challenge, we propose to leverage news-source credibility labels as weak labels for social media posts and propose model-guided refinement of labels to construct large-scale, diverse misinformation labeled datasets in new domains. The weak labels can be inaccurate at the article or social media post level where the stance of the user does not align with the news source or article credibility. We propose a framework to use a detection model self-trained on the initial weak labels with uncertainty sampling based on entropy in predictions of the model to identify potentially inaccurate labels and correct for them using self-supervision or relabeling. The framework will incorporate social context of the post in terms of the community of its associated user for surfacing inaccurate labels towards building a large-scale dataset with minimum human effort. To provide labeled datasets with distinction of misleading narratives where information might be missing significant context or has inaccurate ancillary details, the proposed framework will use the few labeled samples as class prototypes to separate high confidence samples into false, unproven, mixture, mostly false, mostly true, true, and debunk information. The approach is demonstrated for providing a large-scale misinformation dataset on COVID-19 vaccines.