Can Features for Phishing URL Detection Be Trusted Across Diverse Datasets? A Case Study with Explainable AI

TL;DR

This study investigates whether features used in machine learning models for phishing URL detection are reliable across different datasets, revealing that features often depend on specific datasets and may not generalize well.

Contribution

The paper provides an empirical analysis of feature generalizability in phishing detection across datasets using explainable AI techniques.

Findings

Features are often dataset-dependent and may not generalize across datasets.

Explainable AI reveals differences in feature contributions between datasets.

Model performance drops when trained on one dataset and tested on another.

Abstract

Phishing has been a prevalent cyber threat that manipulates users into revealing sensitive private information through deceptive tactics, designed to masquerade as trustworthy entities. Over the years, proactively detection of phishing URLs (or websites) has been established as an widely-accepted defense approach. In literature, we often find supervised Machine Learning (ML) models with highly competitive performance for detecting phishing websites based on the extracted features from both phishing and benign (i.e., legitimate) websites. However, it is still unclear if these features or indicators are dependent on a particular dataset or they are generalized for overall phishing detection. In this paper, we delve deeper into this issue by analyzing two publicly available phishing URL datasets, where each dataset has its own set of unique and overlapping features related to URL string and website contents. We want to investigate if overlapping features are similar in nature across datasets and how does the model perform when trained on one dataset and tested on the other. We conduct practical experiments and leverage explainable AI (XAI) methods such as SHAP plots to provide insights into different features' contributions in case of phishing detection to answer our primary question, "Can features for phishing URL detection be trusted across diverse dataset?". Our case study experiment results show that features for phishing URL detection can often be dataset-dependent and thus may not be trusted across different datasets even though they share same set of feature behaviors.