Diversifying Counterattacks: Orthogonal Exploration for Robust CLIP Inference

TL;DR

This paper introduces Directional Orthogonal Counterattack (DOC), a novel method that enhances the diversity of test-time adversarial counterattacks for vision-language models, leading to improved robustness against adversarial attacks.

Contribution

The paper proposes DOC, which incorporates orthogonal gradient directions and momentum to diversify counterattacks, addressing overfitting and narrow exploration issues in prior methods.

Findings

DOC improves robustness across 16 datasets.

Enhanced counterattack diversity leads to better adversarial defense.

Maintains competitive accuracy on clean data.

Abstract

Vision-language pre-training models (VLPs) demonstrate strong multimodal understanding and zero-shot generalization, yet remain vulnerable to adversarial examples, raising concerns about their reliability. Recent work, Test-Time Counterattack (TTC), improves robustness by generating perturbations that maximize the embedding deviation of adversarial inputs using PGD, pushing them away from their adversarial representations. However, due to the fundamental difference in optimization objectives between adversarial attacks and counterattacks, generating counterattacks solely based on gradients with respect to the adversarial input confines the search to a narrow space. As a result, the counterattacks could overfit limited adversarial patterns and lack the diversity to fully neutralize a broad range of perturbations. In this work, we argue that enhancing the diversity and coverage of counterattacks is crucial to improving adversarial robustness in test-time defense. Accordingly, we propose Directional Orthogonal Counterattack (DOC), which augments counterattack optimization by incorporating orthogonal gradient directions and momentum-based updates. This design expands the exploration of the counterattack space and increases the diversity of perturbations, which facilitates the discovery of more generalizable counterattacks and ultimately improves the ability to neutralize adversarial perturbations. Meanwhile, we present a directional sensitivity score based on averaged cosine similarity to boost DOC by improving example discrimination and adaptively modulating the counterattack strength. Extensive experiments on 16 datasets demonstrate that DOC improves adversarial robustness under various attacks while maintaining competitive clean accuracy. Code is available at https://github.com/bookman233/DOC.