The Feasibility and Inevitability of Stealth Attacks

TL;DR

This paper demonstrates that AI systems, especially deep neural networks, are vulnerable to stealth attacks where minimal modifications can control outputs without detection, highlighting a significant security concern.

Contribution

It introduces new attack strategies that can subtly manipulate AI decisions while remaining undetectable, and analyzes their feasibility and implications.

Findings

Stealth attacks can be made transparent on fixed validation sets.

Single neuron modifications can cause significant vulnerabilities.

Strategies to defend against such attacks are proposed.

Abstract

We develop and study new adversarial perturbations that enable an attacker to gain control over decisions in generic Artificial Intelligence (AI) systems including deep learning neural networks. In contrast to adversarial data modification, the attack mechanism we consider here involves alterations to the AI system itself. Such a stealth attack could be conducted by a mischievous, corrupt or disgruntled member of a software development team. It could also be made by those wishing to exploit a ``democratization of AI'' agenda, where network architectures and trained parameter sets are shared publicly. We develop a range of new implementable attack strategies with accompanying analysis, showing that with high probability a stealth attack can be made transparent, in the sense that system performance is unchanged on a fixed validation set which is unknown to the attacker, while evoking any desired output on a trigger input of interest. The attacker only needs to have estimates of the size of the validation set and the spread of the AI's relevant latent space. In the case of deep learning neural networks, we show that a one neuron attack is possible - a modification to the weights and bias associated with a single neuron - revealing a vulnerability arising from over-parameterization. We illustrate these concepts using state of the art architectures on two standard image data sets. Guided by the theory and computational results, we also propose strategies to guard against stealth attacks.