Anti-Phishing Training (Still) Does Not Work: A Large-Scale Reproduction of Phishing Training Inefficacy Grounded in the NIST Phish Scale

TL;DR

This large-scale study found that current phishing training methods do not significantly reduce click rates or improve reporting, although the NIST Phish Scale effectively predicts user susceptibility, highlighting the need for more effective cybersecurity training strategies.

Contribution

The study provides a rigorous, large-scale reproduction of phishing training efficacy, validating the NIST Phish Scale and introducing organizational resilience metrics, revealing training's limited impact.

Findings

Training showed no significant effect on click or reporting rates

NIST Phish Scale predicts user susceptibility to phishing

Organizational resilience patterns were mixed and unaffected by training

Abstract

Social engineering attacks delivered via email, commonly known as phishing, represent a persistent cybersecurity threat leading to significant organizational incidents and data breaches. Although many organizations train employees on phishing, often mandated by compliance requirements, the real-world effectiveness of this training remains debated. To contribute to evidence-based cybersecurity policy, we conducted a large-scale reproduction study (N = 12,511) at a US-based financial technology firm. Our experimental design refined prior work by comparing training modalities in operational environments, validating NIST's standardized phishing difficulty measurement, and introducing novel organizational-level temporal resilience metrics. Echoing prior work, training interventions showed no significant main effects on click rates (p=0.450) or reporting rates (p=0.417), with negligible effect sizes. However, we found that the NIST Phish Scale predicted user behavior, with click rates increasing from 7.0% for easy lures to 15.0% for hard lures. Our organizational-level resilience result was mixed: 36-55% of campaigns achieved "inoculation" patterns where reports preceded clicks, but training did not significantly improve organizational-level temporal protection. In summary, our results confirm the ineffectiveness of current phishing training approaches while offering a refined study design for future work.