Robust Defense Strategies for Multimodal Contrastive Learning: Efficient Fine-tuning Against Backdoor Attacks

TL;DR

This paper proposes a novel, efficient method to detect and mitigate backdoor attacks in multimodal contrastive models like CLIP, by identifying triggers and affected labels to fine-tune and restore model robustness.

Contribution

It introduces an innovative strategy that uses an image segmentation oracle and algorithms to identify backdoor triggers and affected labels, enabling targeted fine-tuning of poisoned CLIP models.

Findings

Effective detection of backdoor triggers and victim labels.

Successful rectification of poisoned CLIP models.

Improved robustness demonstrated on visual benchmarks.

Abstract

The advent of multimodal deep learning models, such as CLIP, has unlocked new frontiers in a wide range of applications, from image-text understanding to classification tasks. However, these models are not safe for adversarial attacks, particularly backdoor attacks, which can subtly manipulate model behavior. Moreover, existing defense methods typically involve training from scratch or fine-tuning using a large dataset without pinpointing the specific labels that are affected. In this study, we introduce an innovative strategy to enhance the robustness of multimodal contrastive learning models against such attacks. In particular, given a poisoned CLIP model, our approach can identify the backdoor trigger and pinpoint the victim samples and labels in an efficient manner. To that end, an image segmentation ``oracle'' is introduced as the supervisor for the output of the poisoned CLIP. We develop two algorithms to rectify the poisoned model: (1) differentiating between CLIP and Oracle's knowledge to identify potential triggers; (2) pinpointing affected labels and victim samples, and curating a compact fine-tuning dataset. With this knowledge, we are allowed to rectify the poisoned CLIP model to negate backdoor effects. Extensive experiments on visual recognition benchmarks demonstrate our strategy is effective in CLIP-based backdoor defense.