Why Do Unlearnable Examples Work: A Novel Perspective of Mutual Information

TL;DR

This paper introduces a mutual information perspective to understand and improve unlearnable examples, proposing a new method that effectively impedes model generalization by reducing mutual information through covariance minimization.

Contribution

It provides a theoretical analysis linking mutual information reduction to unlearnability and proposes a novel method, MI-UE, that outperforms existing approaches.

Findings

Effective unlearnable examples decrease mutual information between features.

Deeper networks benefit more from mutual information reduction.

Proposed MI-UE method outperforms previous methods under defenses.

Abstract

The volume of freely scraped data on the Internet has driven the tremendous success of deep learning. Along with this comes the growing concern about data privacy and security. Numerous methods for generating unlearnable examples have been proposed to prevent data from being illicitly learned by unauthorized deep models by impeding generalization. However, the existing approaches primarily rely on empirical heuristics, making it challenging to enhance unlearnable examples with solid explanations. In this paper, we analyze and improve unlearnable examples from a novel perspective: mutual information reduction. We demonstrate that effective unlearnable examples always decrease mutual information between clean features and poisoned features, and when the network gets deeper, the unlearnability goes better together with lower mutual information. Further, we prove from a covariance reduction perspective that minimizing the conditional covariance of intra-class poisoned features reduces the mutual information between distributions. Based on the theoretical results, we propose a novel unlearnable method called Mutual Information Unlearnable Examples (MI-UE) that reduces covariance by maximizing the cosine similarity among intra-class features, thus impeding the generalization effectively. Extensive experiments demonstrate that our approach significantly outperforms the previous methods, even under defense mechanisms.