CLIP-Mamba: CLIP Pretrained Mamba Models with OOD and Hessian Evaluation

TL;DR

This paper introduces CLIP-pretrained Mamba models, demonstrating their competitive zero-shot classification performance, robustness to OOD data, and analyzing their complex training landscape through Hessian analysis.

Contribution

First to train and evaluate transferable Mamba models with CLIP pretraining across diverse datasets and OOD conditions, revealing their efficiency and training challenges.

Findings

Mamba models with 67M parameters match 307M ViT in zero-shot tasks.

Mamba models excel in OOD robustness under contrast and high-pass filtering.

Hessian analysis shows Mamba models have sharper, more non-convex loss landscapes.

Abstract

State space models and Mamba-based models have been increasingly applied across various domains, achieving state-of-the-art performance. This technical report introduces the first attempt to train a transferable Mamba model utilizing contrastive language-image pretraining (CLIP). We have trained Mamba models of varying sizes and undertaken comprehensive evaluations of these models on 26 zero-shot classification datasets and 16 out-of-distribution (OOD) datasets. Our findings reveal that a Mamba model with 67 million parameters is on par with a 307 million-parameter Vision Transformer (ViT) model in zero-shot classification tasks, highlighting the parameter efficiency of Mamba models. In tests of OOD generalization, Mamba-based models exhibit exceptional performance in conditions of OOD image contrast or when subjected to high-pass filtering. However, a Hessian analysis indicates that Mamba models feature a sharper and more non-convex landscape compared to ViT-based models, making them more challenging to train. The source code is available at https://github.com/raytrun/mamba-clip.