Towards Provably Unlearnable Examples via Bayes Error Optimisation

TL;DR

This paper introduces a formal, optimization-based method to create unlearnable data examples by maximising Bayes error, ensuring they remain unlearnable even when mixed with clean data, with proven guarantees and practical effectiveness.

Contribution

It proposes a novel, theoretically grounded approach to generate unlearnable examples by maximising Bayes error using projected gradient ascent, addressing limitations of heuristic methods.

Findings

Method provably increases Bayes error in data.

Effective in preventing model learning when mixed with clean data.

Consistent results across multiple datasets and models.

Abstract

The recent success of machine learning models, especially large-scale classifiers and language models, relies heavily on training with massive data. These data are often collected from online sources. This raises serious concerns about the protection of user data, as individuals may not have given consent for their data to be used in training. To address this concern, recent studies introduce the concept of unlearnable examples, i.e., data instances that appear natural but are intentionally altered to prevent models from effectively learning from them. While existing methods demonstrate empirical effectiveness, they typically rely on heuristic trials and lack formal guarantees. Besides, when unlearnable examples are mixed with clean data, as is often the case in practice, their unlearnability disappears. In this work, we propose a novel approach to constructing unlearnable examples by systematically maximising the Bayes error, a measurement of irreducible classification error. We develop an optimisation-based approach and provide an efficient solution using projected gradient ascent. Our method provably increases the Bayes error and remains effective when the unlearning examples are mixed with clean samples. Experimental results across multiple datasets and model architectures are consistent with our theoretical analysis and show that our approach can restrict data learnability, effectively in practice.