Proxy Robustness in Vision Language Models is Effortlessly Transferable

TL;DR

This paper introduces a novel framework for transferring adversarial robustness between vision-language models like CLIP, leveraging intrinsic defensive capabilities and a decoupled training process to balance robustness and generalization.

Contribution

We propose Heterogeneous Proxy Transfer (HPT) and Generalization-Pivot Decoupling (GPD), enabling efficient robustness transfer across CLIP variants while maintaining natural generalization.

Findings

Effective robustness transfer demonstrated on 15 zero-shot datasets.

HPT-GPD achieves a balance between adversarial robustness and natural generalization.

Proxy transfer significantly improves robustness with minimal computational overhead.

Abstract

As a pivotal technique for improving the defense of deep models, adversarial robustness transfer via distillation has demonstrated remarkable success in conventional image classification tasks. However, this paradigm encounters critical challenges when applied to vision-language models (VLM) (e.g., CLIP): constructing adversarially robust teacher for large-scale multi-modal models demands prohibitively high computational resources. We bridge this gap by revealing an interesting phenomenon: vanilla CLIP (without adversarial training) exhibits intrinsic defensive capabilities against adversarial examples generated by another CLIP with different architectures. We formally define this as proxy adversarial robustness, and naturally propose a Heterogeneous Proxy Transfer (HPT) framework that establishes cross-architectural robustness distillation channels between CLIP variants, effortlessly enabling the VLM robustness transfer from proxy to target models. Yet, such proxy transfer paradigm easily induces severe overfitting, leading to a sharp degradation in zero-shot natural generalization. To resolve that, we design Generalization-Pivot Decoupling (GPD) by leveraging the difference in learning rate scheduling. This decouples the proxy transfer process into a generalization-anchored warm-up that maintains generalization and a generalization-pulled HPT that promotes adversarial robustness, to achieve an equilibrium between natural generalization and adversarial robustness. Extensive experiments on 15 zero-shot datasets demonstrate the effectiveness of our HPT-GPD method. The code is available at the website of github.com/fxw13/HPT-GPD.