Mental Models of Adversarial Machine Learning

TL;DR

This study explores practitioners' mental models of adversarial machine learning, revealing misconceptions and emphasizing the importance of understanding security within entire workflows for better risk management.

Contribution

It provides a qualitative analysis of practitioners' perceptions, highlighting misconceptions and the holistic view of security in machine learning workflows, which is less addressed in academic research.

Findings

Practitioners often confuse ML security with unrelated threats.

Security is perceived as related to entire workflows, not just models.

Understanding these mental models can improve risk communication and security practices.

Abstract

Although machine learning is widely used in practice, little is known about practitioners' understanding of potential security challenges. In this work, we close this substantial gap and contribute a qualitative study focusing on developers' mental models of the machine learning pipeline and potentially vulnerable components. Similar studies have helped in other security fields to discover root causes or improve risk communication. Our study reveals two \facets of practitioners' mental models of machine learning security. Firstly, practitioners often confuse machine learning security with threats and defences that are not directly related to machine learning. Secondly, in contrast to most academic research, our participants perceive security of machine learning as not solely related to individual models, but rather in the context of entire workflows that consist of multiple components. Jointly with our additional findings, these two facets provide a foundation to substantiate mental models for machine learning security and have implications for the integration of adversarial machine learning into corporate workflows, \new{decreasing practitioners' reported uncertainty}, and appropriate regulatory frameworks for machine learning security.