Performance Analysis of Transformer Based Models (BERT, ALBERT and RoBERTa) in Fake News Detection

TL;DR

This paper evaluates transformer-based models like BERT, ALBERT, and RoBERTa for fake news detection in Bahasa Indonesia, finding ALBERT performs best with high accuracy and efficiency.

Contribution

It compares the performance of BERT, ALBERT, and RoBERTa models specifically for fake news detection in Bahasa Indonesia, highlighting ALBERT's superior results.

Findings

ALBERT achieved 87.6% accuracy.

ALBERT had 86.9% precision and F1-score.

ALBERT demonstrated faster run-time per epoch.

Abstract

Fake news is fake material in a news media format but is not processed properly by news agencies. The fake material can provoke or defame significant entities or individuals or potentially even for the personal interests of the creators, causing problems for society. Distinguishing fake news and real news is challenging due to limited of domain knowledge and time constraints. According to the survey, the top three areas most exposed to hoaxes and misinformation by residents are in Banten, DKI Jakarta and West Java. The model of transformers is referring to an approach in the field of artificial intelligence (AI) in natural language processing utilizing the deep learning architectures. Transformers exercise a powerful attention mechanism to process text in parallel and produce rich and contextual word representations. A previous study indicates a superior performance of a transformer model known as BERT over and above non transformer approach. However, some studies suggest the performance can be improved with the use of improved BERT models known as ALBERT and RoBERTa. However, the modified BERT models are not well explored for detecting fake news in Bahasa Indonesia. In this research, we explore those transformer models and found that ALBERT outperformed other models with 87.6% accuracy, 86.9% precision, 86.9% F1-score, and 174.5 run-time (s/epoch) respectively. Source code available at: https://github.com/Shafna81/fakenewsdetection.git