Multimodal Unlearnable Examples: Protecting Data against Multimodal Contrastive Learning

TL;DR

This paper introduces Multi-step Error Minimization (MEM), a novel method to generate multimodal unlearnable examples that protect data privacy in contrastive learning by misleading models with optimized noise and text triggers.

Contribution

It extends the Error-Minimization framework to multimodal data, effectively creating unlearnable examples for image-caption pairs in contrastive learning.

Findings

MEM significantly reduces retrieval accuracy, approaching half of random chance.

The method demonstrates high transferability across different models.

Experiments confirm the effectiveness of multimodal unlearnable examples for privacy protection.

Abstract

Multimodal contrastive learning (MCL) has shown remarkable advances in zero-shot classification by learning from millions of image-caption pairs crawled from the Internet. However, this reliance poses privacy risks, as hackers may unauthorizedly exploit image-text data for model training, potentially including personal and privacy-sensitive information. Recent works propose generating unlearnable examples by adding imperceptible perturbations to training images to build shortcuts for protection. However, they are designed for unimodal classification, which remains largely unexplored in MCL. We first explore this context by evaluating the performance of existing methods on image-caption pairs, and they do not generalize effectively to multimodal data and exhibit limited impact to build shortcuts due to the lack of labels and the dispersion of pairs in MCL. In this paper, we propose Multi-step Error Minimization (MEM), a novel optimization process for generating multimodal unlearnable examples. It extends the Error-Minimization (EM) framework to optimize both image noise and an additional text trigger, thereby enlarging the optimized space and effectively misleading the model to learn the shortcut between the noise features and the text trigger. Specifically, we adopt projected gradient descent to solve the noise minimization problem and use HotFlip to approximate the gradient and replace words to find the optimal text trigger. Extensive experiments demonstrate the effectiveness of MEM, with post-protection retrieval results nearly half of random guessing, and its high transferability across different models. Our code is available on the https://github.com/thinwayliu/Multimodal-Unlearnable-Examples