Robustness in Both Domains: CLIP Needs a Robust Text Encoder

TL;DR

This paper introduces LEAF, an adversarial finetuning method that enhances the robustness of CLIP's text encoder, improving performance in adversarial settings without sacrificing vision capabilities.

Contribution

LEAF is a scalable adversarial finetuning approach that significantly improves CLIP's text encoder robustness across multiple tasks while maintaining existing vision model performance.

Findings

Improved zero-shot adversarial accuracy in text domain

Enhanced multimodal retrieval recall under adversarial noise

Better input text reconstruction from embeddings

Abstract

Adversarial input attacks can cause a significant shift of CLIP embeddings. This can affect the downstream robustness of models incorporating CLIP in the pipeline, such as text-to-image generative models or large vision language models. While some efforts have been done towards making the CLIP image encoders robust, the robustness of text encoders remains unexplored. In this work, we cover this gap in the literature. We propose LEAF: an efficient adversarial finetuning method for the text domain, with the ability to scale to large CLIP models. Our models significantly improve the zero-shot adversarial accuracy in the text domain, while maintaining the vision performance provided by robust image encoders. When combined with text-to-image diffusion models, we can improve the generation quality under adversarial noise. In multimodal retrieval tasks, LEAF improves the recall under adversarial noise over standard CLIP models. Finally, we show that robust text encoders facilitate better reconstruction of input text from its embedding via direct optimization. We open-source our code ( https://github.com/LIONS-EPFL/LEAF ) and models ( https://huggingface.co/LEAF-CLIP ).