Model Generalization on COVID-19 Fake News Detection

TL;DR

This paper develops robust transformer-based models for COVID-19 fake news detection, emphasizing model generalization across different datasets and proposing methods like robust loss functions and influence-based data cleansing.

Contribution

It introduces two approaches—robust fine-tuning and influence-based data cleansing—to improve model robustness and generalization in COVID-19 fake news detection.

Findings

Achieved 98.13% W-F1 on shared task

Attained 54.33% W-F1 on external test set with data cleansing

Highlighted the importance of model generalization in fake news detection

Abstract

Amid the pandemic COVID-19, the world is facing unprecedented infodemic with the proliferation of both fake and real information. Considering the problematic consequences that the COVID-19 fake-news have brought, the scientific community has put effort to tackle it. To contribute to this fight against the infodemic, we aim to achieve a robust model for the COVID-19 fake-news detection task proposed at CONSTRAINT 2021 (FakeNews-19) by taking two separate approaches: 1) fine-tuning transformers based language models with robust loss functions and 2) removing harmful training instances through influence calculation. We further evaluate the robustness of our models by evaluating on different COVID-19 misinformation test set (Tweets-19) to understand model generalization ability. With the first approach, we achieve 98.13% for weighted F1 score (W-F1) for the shared task, whereas 38.18% W-F1 on the Tweets-19 highest. On the contrary, by performing influence data cleansing, our model with 99% cleansing percentage can achieve 54.33% W-F1 score on Tweets-19 with a trade-off. By evaluating our models on two COVID-19 fake-news test sets, we suggest the importance of model generalization ability in this task to step forward to tackle the COVID-19 fake-news problem in online social media platforms.