Breaking Bad Email Habits: Bounding the Impact of Simulated Phishing Campaigns

TL;DR

This paper develops a new analytical framework combining marginal structural models and correlated random effects to accurately assess the true impact of simulated phishing campaigns on employee behavior, correcting for biases caused by endogenous training triggers.

Contribution

It introduces a portable, combined MSM+CRE framework for analyzing simulated phishing logs, addressing biases from endogenous training and disentangling habit formation from stable individual differences.

Findings

Most repeat clicking reflects employee traits, not recent training effects.

Persistence of clicking behavior is context-dependent and influenced by campaign cues.

Teachable-moment features like emotion framing reduce behavioral persistence.

Abstract

Simulated phishing campaigns are widely deployed, yet the behavioral data they produce is endogenous: because training is triggered by clicking, the employees receiving intervention have already demonstrated susceptibility. This endogeneity, combined with the difficulty of separating genuine habit formation from stable individual differences, means standard analyses can mischaracterize program effectiveness. In this Research Note, we develop a generalizable analytic framework addressing both biases simultaneously. We utilize marginal structural models (MSMs) to correct for the endogenous, click-triggered assignment of training, while integrating correlated random effects (CRE) to disentangle true state dependence from stable employee heterogeneity. Applying the MSM+CRE estimator to logs from 17 campaigns delivered to university staff (192,840 observations) reveals that analyses ignoring stable differences overstate the causal persistence of clicking; most repeat clicking reflects who employees are, not the effect of recent failures. This persistence is context-dependent, amplifying when successive campaigns share persuasion cues. Teachable-moment features also matter: emotion framing and explicit reporting pitches can largely eliminate persistence, while annotated-email cues modestly exacerbate it. Finally, employees engaging with the education page exhibit greater persistence than those dismissing it, consistent with an emboldening mechanism. We contribute methodologically by integrating MSMs and CRE into a portable framework for analyzing standard simulation logs, and practically by identifying specific design levers so organizations can better sequence and evaluate their phishing programs.