Man versus Machine: AutoML and Human Experts' Role in Phishing Detection

TL;DR

This paper evaluates whether AutoML frameworks can outperform human-designed models in phishing detection, finding AutoML excels in complex, non-linear datasets but still requires expert input for real-world applications.

Contribution

It provides a comparative analysis of AutoML frameworks versus human-crafted models across multiple phishing datasets, highlighting strengths and limitations of AutoML in cybersecurity.

Findings

AutoML outperforms manual models in complex, non-linear datasets.

AutoML struggles with incremental learning and unlabeled data in real-world scenarios.

Human expertise remains crucial for effective phishing detection systems.

Abstract

Machine learning (ML) has developed rapidly in the past few years and has successfully been utilized for a broad range of tasks, including phishing detection. However, building an effective ML-based detection system is not a trivial task, and requires data scientists with knowledge of the relevant domain. Automated Machine Learning (AutoML) frameworks have received a lot of attention in recent years, enabling non-ML experts in building a machine learning model. This brings to an intriguing question of whether AutoML can outperform the results achieved by human data scientists. Our paper compares the performances of six well-known, state-of-the-art AutoML frameworks on ten different phishing datasets to see whether AutoML-based models can outperform manually crafted machine learning models. Our results indicate that AutoML-based models are able to outperform manually developed machine learning models in complex classification tasks, specifically in datasets where the features are not quite discriminative, and datasets with overlapping classes or relatively high degrees of non-linearity. Challenges also remain in building a real-world phishing detection system using AutoML frameworks due to the current support only on supervised classification problems, leading to the need for labeled data, and the inability to update the AutoML-based models incrementally. This indicates that experts with knowledge in the domain of phishing and cybersecurity are still essential in the loop of the phishing detection pipeline.