On the security relevance of weights in deep learning

TL;DR

This paper demonstrates that simple, task-independent permutations of initial weights in deep learning models can significantly reduce accuracy, highlighting the critical security role of weight initialization.

Contribution

It reveals a broad, data-independent threat to deep learning models through initial weight permutations, emphasizing the importance of weight security.

Findings

Weight permutations can limit accuracy to 50% on Fashion MNIST

The attack is effective across MNIST and CIFAR datasets

Weight statistics and loss metrics do not reveal the attack

Abstract

Recently, a weight-based attack on stochastic gradient descent inducing overfitting has been proposed. We show that the threat is broader: A task-independent permutation on the initial weights suffices to limit the achieved accuracy to for example 50% on the Fashion MNIST dataset from initially more than $90$%. These findings are confirmed on MNIST and CIFAR. We formally confirm that the attack succeeds with high likelihood and does not depend on the data. Empirically, weight statistics and loss appear unsuspicious, making it hard to detect the attack if the user is not aware. Our paper is thus a call for action to acknowledge the importance of the initial weights in deep learning.