Stealthy Backdoor Attack in Self-Supervised Learning Vision Encoders for Large Vision Language Models

TL;DR

This paper uncovers a new backdoor vulnerability in self-supervised learning vision encoders used in large vision language models, enabling high success rate attacks that induce visual hallucinations while evading detection.

Contribution

It introduces BadVision, a novel backdoor attack method exploiting SSL vision encoders, with techniques for trigger optimization and backdoor learning, demonstrating high effectiveness and stealthiness.

Findings

Over 99% attack success rate in inducing hallucinations

77.6% increase in visual understanding error

Existing detection methods fail to identify the attack

Abstract

Self-supervised learning (SSL) vision encoders learn high-quality image representations and thus have become a vital part of developing vision modality of large vision language models (LVLMs). Due to the high cost of training such encoders, pre-trained encoders are widely shared and deployed into many LVLMs, which are security-critical or bear societal significance. Under this practical scenario, we reveal a new backdoor threat that significant visual hallucinations can be induced into these LVLMs by merely compromising vision encoders. Because of the sharing and reuse of these encoders, many downstream LVLMs may inherit backdoor behaviors from encoders, leading to widespread backdoors. In this work, we propose BadVision, the first method to exploit this vulnerability in SSL vision encoders for LVLMs with novel trigger optimization and backdoor learning techniques. We evaluate BadVision on two types of SSL encoders and LVLMs across eight benchmarks. We show that BadVision effectively drives the LVLMs to attacker-chosen hallucination with over 99% attack success rate, causing a 77.6% relative visual understanding error while maintaining the stealthiness. SoTA backdoor detection methods cannot detect our attack effectively.