Overview of the Shared Task on Fake News Detection in Urdu at FIRE 2020

TL;DR

This paper summarizes the first shared task on fake news detection in Urdu, involving multiple teams using various machine learning methods, with BERT-based models achieving the highest accuracy.

Contribution

It introduces the first Urdu fake news detection dataset and benchmark, and compares different machine learning approaches for the task.

Findings

BERT-based models achieved an F-score of 0.90.

The dataset covers five news domains.

Multiple teams participated using diverse machine learning techniques.

Abstract

This overview paper describes the first shared task on fake news detection in Urdu language. The task was posed as a binary classification task, in which the goal is to differentiate between real and fake news. We provided a dataset divided into 900 annotated news articles for training and 400 news articles for testing. The dataset contained news in five domains: (i) Health, (ii) Sports, (iii) Showbiz, (iv) Technology, and (v) Business. 42 teams from 6 different countries (India, China, Egypt, Germany, Pakistan, and the UK) registered for the task. 9 teams submitted their experimental results. The participants used various machine learning methods ranging from feature-based traditional machine learning to neural networks techniques. The best performing system achieved an F-score value of 0.90, showing that the BERT-based approach outperforms other machine learning techniques