SpikeCLIP: A Contrastive Language-Image Pretrained Spiking Neural Network

TL;DR

SpikeCLIP introduces a novel spiking neural network framework that effectively integrates visual and linguistic features, achieving competitive performance with lower energy consumption in multimodal tasks.

Contribution

The paper presents SpikeCLIP, a new approach combining alignment pre-training and dual-loss fine-tuning to unify multimodal features in SNNs, advancing energy-efficient multimodal learning.

Findings

SNNs achieve comparable results to ANNs in multimodal tasks.

SpikeCLIP significantly reduces energy consumption.

Maintains robust classification even for out-of-category classes.

Abstract

Spiking Neural Networks (SNNs) have emerged as a promising alternative to conventional Artificial Neural Networks (ANNs), demonstrating comparable performance in both visual and linguistic tasks while offering the advantage of improved energy efficiency. Despite these advancements, the integration of linguistic and visual features into a unified representation through spike trains poses a significant challenge, and the application of SNNs to multimodal scenarios remains largely unexplored. This paper presents SpikeCLIP, a novel framework designed to bridge the modality gap in spike-based computation. Our approach employs a two-step recipe: an ``alignment pre-training'' to align features across modalities, followed by a ``dual-loss fine-tuning'' to refine the model's performance. Extensive experiments reveal that SNNs achieve results on par with ANNs while substantially reducing energy consumption across various datasets commonly used for multimodal model evaluation. Furthermore, SpikeCLIP maintains robust image classification capabilities, even when dealing with classes that fall outside predefined categories. This study marks a significant advancement in the development of energy-efficient and biologically plausible multimodal learning systems. Our code is available at https://github.com/Lvchangze/SpikeCLIP.