Towards Evaluating the Robustness of Visual State Space Models

TL;DR

This paper evaluates the robustness of Vision State Space Models (VSSMs) against various natural and adversarial perturbations, comparing their performance with other architectures and analyzing their resilience in complex visual scenarios.

Contribution

It provides a comprehensive robustness evaluation of VSSMs across multiple perturbation types and benchmarks, highlighting their strengths and limitations in complex visual tasks.

Findings

VSSMs show robustness to certain corruptions but are vulnerable to others.

Frequency analysis reveals differential performance against low and high-frequency attacks.

VSSMs outperform some architectures in specific robustness scenarios.

Abstract

Vision State Space Models (VSSMs), a novel architecture that combines the strengths of recurrent neural networks and latent variable models, have demonstrated remarkable performance in visual perception tasks by efficiently capturing long-range dependencies and modeling complex visual dynamics. However, their robustness under natural and adversarial perturbations remains a critical concern. In this work, we present a comprehensive evaluation of VSSMs' robustness under various perturbation scenarios, including occlusions, image structure, common corruptions, and adversarial attacks, and compare their performance to well-established architectures such as transformers and Convolutional Neural Networks. Furthermore, we investigate the resilience of VSSMs to object-background compositional changes on sophisticated benchmarks designed to test model performance in complex visual scenes. We also assess their robustness on object detection and segmentation tasks using corrupted datasets that mimic real-world scenarios. To gain a deeper understanding of VSSMs' adversarial robustness, we conduct a frequency-based analysis of adversarial attacks, evaluating their performance against low-frequency and high-frequency perturbations. Our findings highlight the strengths and limitations of VSSMs in handling complex visual corruptions, offering valuable insights for future research. Our code and models will be available at https://github.com/HashmatShadab/MambaRobustness.