BitAbuse: A Dataset of Visually Perturbed Texts for Defending Phishing Attacks

TL;DR

The paper introduces BitAbuse, a large dataset of real-world visually perturbed phishing texts, to improve language model robustness against adversarial attacks, demonstrating significant performance gains over previous synthetic datasets.

Contribution

It provides the first large-scale dataset of real-world visually perturbed phishing texts, enabling more effective training of models to defend against adversarial attacks.

Findings

Language models trained on BitAbuse achieved ~96% accuracy.

Significant performance gap between real-world and synthetic datasets.

The dataset enhances model robustness against visually perturbed phishing texts.

Abstract

Phishing often targets victims through visually perturbed texts to bypass security systems. The noise contained in these texts functions as an adversarial attack, designed to deceive language models and hinder their ability to accurately interpret the content. However, since it is difficult to obtain sufficient phishing cases, previous studies have used synthetic datasets that do not contain real-world cases. In this study, we propose the BitAbuse dataset, which includes real-world phishing cases, to address the limitations of previous research. Our dataset comprises a total of 325,580 visually perturbed texts. The dataset inputs are drawn from the raw corpus, consisting of visually perturbed sentences and sentences generated through an artificial perturbation process. Each input sentence is labeled with its corresponding ground truth, representing the restored, non-perturbed version. Language models trained on our proposed dataset demonstrated significantly better performance compared to previous methods, achieving an accuracy of approximately 96%. Our analysis revealed a significant gap between real-world and synthetic examples, underscoring the value of our dataset for building reliable pre-trained models for restoration tasks. We release the BitAbuse dataset, which includes real-world phishing cases annotated with visual perturbations, to support future research in adversarial attack defense.