Why Most Results of Socio-Technical Security User Studies Are False

TL;DR

This paper critically assesses the validity of positive findings in socio-technical security user studies, revealing that most are likely false due to weak evidence and statistical biases, thus questioning the reliability of current research results.

Contribution

It introduces a comprehensive probabilistic framework to evaluate the strength of evidence in cyber security user studies, highlighting the prevalence of false positives.

Findings

Most positive reports have low a posteriori probabilities.

Few studies achieve strong evidence even with high prior likelihoods.

The field's overall evidence strength is weak, with many results likely false.

Abstract

Background. In recent years, cyber security user studies have been scrutinized for their reporting completeness, statistical reporting fidelity, statistical reliability and biases. It remains an open question what strength of evidence positive reports of such studies actually yield. We focus on the extent to which positive reports indicate relation true in reality, that is, a probabilistic assessment. Aim. This study aims at establishing the overall strength of evidence in cyber security user studies, with the dimensions -- Positive Predictive Value (PPV) and its complement False Positive Risk (FPR), -- Likelihood Ratio (LR), and -- Reverse-Bayesian Prior (RBP) for a fixed tolerated False Positive Risk. Method. Based on $431$ coded statistical inferences in $146$ cyber security user studies from a published SLR covering the years 2006-2016, we first compute a simulation of the a posteriori false positive risk based on assumed prior and bias thresholds. Second, we establish the observed likelihood ratios for positive reports. Third, we compute the reverse Bayesian argument on the observed positive reports by computing the prior required for a fixed a posteriori false positive rate. Results. We obtain a comprehensive analysis of the strength of evidence including an account of appropriate multiple comparison corrections. The simulations show that even in face of well-controlled conditions and high prior likelihoods, only few studies achieve good a posteriori probabilities. Conclusions. Our work shows that the strength of evidence of the field is weak and that most positive reports are likely false. From this, we learn what to watch out for in studies to advance the knowledge of the field.