Defending LVLMs Against Vision Attacks through Partial-Perception Supervision

TL;DR

This paper introduces DPS, a training-free black-box method that uses partial perception responses to defend LVLMs against vision attacks, significantly reducing attack success rates while maintaining response quality.

Contribution

Proposes DPS, a novel partial-perception supervision technique that enhances LVLM robustness against vision attacks without additional training or model modifications.

Findings

Reduces attack success rate by 76.3% on average across datasets.

Outperforms baseline methods in defending LVLMs.

Maintains high response quality on clean images.

Abstract

Recent studies have raised significant concerns regarding the vulnerability of Large Vision Language Models (LVLMs) to maliciously injected or perturbed input images, which can mislead their responses. Existing defense methods show that such vision attacks are sensitive to image modifications especially cropping, using majority voting across responses of modified images as corrected responses. However, these modifications often result in partial images and distort the semantics, which reduces response quality on clean images after voting. Instead of directly using responses from partial images for voting, we investigate using them to supervise the LVLM's responses to the original images. We propose a black-box, training-free method called DPS (Defense through Partial-Perception Supervision). In this approach, the model is prompted using the responses generated by a model that perceives only a partial image. With DPS, the model can adjust its response based on partial image understanding when under attack, while confidently maintaining its original response for clean input. Our findings show that the weak model can supervise the strong model: when faced with an attacked input, the strong model becomes less confident and adjusts its response based on the weak model's partial understanding, effectively defending against the attack. With clean input, it confidently maintains its original response. Empirical experiments show our method outperforms the baseline, cutting the average attack success rate by 76.3% across six datasets on three popular models.