SIFT-Graph: Benchmarking Multimodal Defense Against Image Adversarial Attacks With Robust Feature Graph
Jingjie He, Weijie Liang, Zihan Shan, Matthew Caesar

TL;DR
SIFT-Graph introduces a multimodal defense framework that combines handcrafted and learned features to improve the robustness of vision models against adversarial attacks, with minimal impact on accuracy.
Contribution
It proposes a novel integration of SIFT keypoints and Graph Attention Networks to create a structure-aware, robust feature embedding for defending against adversarial perturbations.
Findings
Enhanced robustness against white-box adversarial attacks.
Marginal decrease in clean image accuracy.
Effective fusion of handcrafted and learned features.
Abstract
Adversarial attacks expose a fundamental vulnerability in modern deep vision models by exploiting their dependence on dense, pixel-level representations that are highly sensitive to imperceptible perturbations. Traditional defense strategies typically operate within this fragile pixel domain, lacking mechanisms to incorporate inherently robust visual features. In this work, we introduce SIFT-Graph, a multimodal defense framework that enhances the robustness of traditional vision models by aggregating structurally meaningful features extracted from raw images using both handcrafted and learned modalities. Specifically, we integrate Scale-Invariant Feature Transform keypoints with a Graph Attention Network to capture scale and rotation invariant local structures that are resilient to perturbations. These robust feature embeddings are then fused with traditional vision model, such as…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdversarial Robustness in Machine Learning · Ethics and Social Impacts of AI · Generative Adversarial Networks and Image Synthesis
