Understanding Robustness of Visual State Space Models for Image Classification

TL;DR

This paper thoroughly investigates the robustness of the Visual State Space Model (VMamba) in image classification, revealing its strengths against certain adversarial attacks and weaknesses in scalability and sensitivity to image structure variations.

Contribution

It provides the first comprehensive analysis of VMamba's robustness, including adversarial resistance, generalizability, and vulnerabilities, offering insights for future improvements.

Findings

VMamba shows superior adversarial robustness compared to Transformers.

It exhibits strong generalizability to out-of-distribution data.

Scalability weaknesses and vulnerabilities to image structure variations are identified.

Abstract

Visual State Space Model (VMamba) has recently emerged as a promising architecture, exhibiting remarkable performance in various computer vision tasks. However, its robustness has not yet been thoroughly studied. In this paper, we delve into the robustness of this architecture through comprehensive investigations from multiple perspectives. Firstly, we investigate its robustness to adversarial attacks, employing both whole-image and patch-specific adversarial attacks. Results demonstrate superior adversarial robustness compared to Transformer architectures while revealing scalability weaknesses. Secondly, the general robustness of VMamba is assessed against diverse scenarios, including natural adversarial examples, out-of-distribution data, and common corruptions. VMamba exhibits exceptional generalizability with out-of-distribution data but shows scalability weaknesses against natural adversarial examples and common corruptions. Additionally, we explore VMamba's gradients and back-propagation during white-box attacks, uncovering unique vulnerabilities and defensive capabilities of its novel components. Lastly, the sensitivity of VMamba to image structure variations is examined, highlighting vulnerabilities associated with the distribution of disturbance areas and spatial information, with increased susceptibility closer to the image center. Through these comprehensive studies, we contribute to a deeper understanding of VMamba's robustness, providing valuable insights for refining and advancing the capabilities of deep neural networks in computer vision applications.