Improving vision-language alignment with graph spiking hybrid Networks

TL;DR

This paper introduces a novel graph spiking hybrid network that leverages panoptic segmentation and contrastive learning to improve vision-language alignment by capturing rich semantic relations and contextual features.

Contribution

It proposes a new GSHN model combining SNNs and GATs, utilizing panoptic segmentation and a novel pre-training method to enhance semantic representation in VL tasks.

Findings

GSHN outperforms existing models on multiple VL benchmarks.

The use of contrastive learning improves embedding similarity and model robustness.

Panoptic segmentation enhances the quality of visual semantic features.

Abstract

To bridge the semantic gap between vision and language (VL), it is necessary to develop a good alignment strategy, which includes handling semantic diversity, abstract representation of visual information, and generalization ability of models. Recent works use detector-based bounding boxes or patches with regular partitions to represent visual semantics. While current paradigms have made strides, they are still insufficient for fully capturing the nuanced contextual relations among various objects. This paper proposes a comprehensive visual semantic representation module, necessitating the utilization of panoptic segmentation to generate coherent fine-grained semantic features. Furthermore, we propose a novel Graph Spiking Hybrid Network (GSHN) that integrates the complementary advantages of Spiking Neural Networks (SNNs) and Graph Attention Networks (GATs) to encode visual semantic information. Intriguingly, the model not only encodes the discrete and continuous latent variables of instances but also adeptly captures both local and global contextual features, thereby significantly enhancing the richness and diversity of semantic representations. Leveraging the spatiotemporal properties inherent in SNNs, we employ contrastive learning (CL) to enhance the similarity-based representation of embeddings. This strategy alleviates the computational overhead of the model and enriches meaningful visual representations by constructing positive and negative sample pairs. We design an innovative pre-training method, Spiked Text Learning (STL), which uses text features to improve the encoding ability of discrete semantics. Experiments show that the proposed GSHN exhibits promising results on multiple VL downstream tasks.