Evaluating the Effectiveness and Robustness of Visual Similarity-based Phishing Detection Models

TL;DR

This study evaluates the real-world effectiveness and robustness of visual similarity-based phishing detection models, revealing significant vulnerabilities and emphasizing the need for more resilient techniques against sophisticated evasion strategies.

Contribution

It provides a comprehensive large-scale evaluation of existing models' performance on real-world data and introduces adversarial testing to assess their robustness against evasion tactics.

Findings

High accuracy on curated datasets but low on real-world data

Attackers evade detection through logo mimicking and simple manipulations

Several models show vulnerabilities to adversarial logo manipulations

Abstract

Phishing attacks pose a significant threat to Internet users, with cybercriminals elaborately replicating the visual appearance of legitimate websites to deceive victims. Visual similarity-based detection systems have emerged as an effective countermeasure, but their effectiveness and robustness in real-world scenarios have been underexplored. In this paper, we comprehensively scrutinize and evaluate the effectiveness and robustness of popular visual similarity-based anti-phishing models using a large-scale dataset of 451k real-world phishing websites. Our analyses of the effectiveness reveal that while certain visual similarity-based models achieve high accuracy on curated datasets in the experimental settings, they exhibit notably low performance on real-world datasets, highlighting the importance of real-world evaluation. Furthermore, we find that the attackers evade the detectors mainly in three ways: (1) directly attacking the model pipelines, (2) mimicking benign logos, and (3) employing relatively simple strategies such as eliminating logos from screenshots. To statistically assess the resilience and robustness of existing models against adversarial attacks, we categorize the strategies attackers employ into visible and perturbation-based manipulations and apply them to website logos. We then evaluate the models' robustness using these adversarial samples. Our findings reveal potential vulnerabilities in several models, emphasizing the need for more robust visual similarity techniques capable of withstanding sophisticated evasion attempts. We provide actionable insights for enhancing the security of phishing defense systems, encouraging proactive actions.