ToxicTextCLIP: Text-Based Poisoning and Backdoor Attacks on CLIP Pre-training

TL;DR

ToxicTextCLIP introduces a method to generate adversarial texts that can poison or backdoor CLIP models during pre-training, exposing vulnerabilities in the model's reliance on web-sourced data.

Contribution

It presents a novel framework for creating high-quality adversarial texts targeting CLIP, addressing challenges of semantic misalignment and background consistency.

Findings

Achieves up to 95.83% poisoning success rate

Reaches 98.68% backdoor Hit@1 accuracy

Successfully bypasses existing defenses like RoCLIP, CleanCLIP, SafeCLIP

Abstract

The Contrastive Language-Image Pretraining (CLIP) model has significantly advanced vision-language modeling by aligning image-text pairs from large-scale web data through self-supervised contrastive learning. Yet, its reliance on uncurated Internet-sourced data exposes it to data poisoning and backdoor risks. While existing studies primarily investigate image-based attacks, the text modality, which is equally central to CLIP's training, remains underexplored. In this work, we introduce ToxicTextCLIP, a framework for generating high-quality adversarial texts that target CLIP during the pre-training phase. The framework addresses two key challenges: semantic misalignment caused by background inconsistency with the target class, and the scarcity of background-consistent texts. To this end, ToxicTextCLIP iteratively applies: 1) a background-aware selector that prioritizes texts with background content aligned to the target class, and 2) a background-driven augmenter that generates semantically coherent and diverse poisoned samples. Extensive experiments on classification and retrieval tasks show that ToxicTextCLIP achieves up to 95.83% poisoning success and 98.68% backdoor Hit@1, while bypassing RoCLIP, CleanCLIP and SafeCLIP defenses. The source code can be accessed via https://github.com/xinyaocse/ToxicTextCLIP/.