Multi-Timescale Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning

TL;DR

This paper introduces MDST++, a novel spiking transformer model that decouples semantic and motion information across multiple timescales, improving zero-shot learning for audio-visual video classification by reducing background bias and capturing dynamic motion more effectively.

Contribution

The paper proposes a dual-stream spiking transformer with motion decoupling and adaptive neuron thresholds, enhancing zero-shot learning performance and motion understanding in video data.

Findings

MDST++ outperforms state-of-the-art methods on benchmarks.

Incorporating motion and multi-timescale info boosts HM and ZSL accuracy by 26.2% and 39.9%.

Adaptive thresholding improves temporal and motion cue extraction.

Abstract

Audio-visual zero-shot learning (ZSL) has been extensively researched for its capability to classify video data from unseen classes during training. Nevertheless, current methodologies often struggle with background scene biases and inadequate motion detail. This paper proposes a novel dual-stream Multi-Timescale Motion-Decoupled Spiking Transformer (MDST++), which decouples contextual semantic information and sparse dynamic motion information. The recurrent joint learning unit is proposed to extract contextual semantic information and capture joint knowledge across various modalities to understand the environment of actions. By converting RGB images to events, our method captures motion information more accurately and mitigates background scene biases. Moreover, we introduce a discrepancy analysis block to model audio motion information. To enhance the robustness of SNNs in extracting temporal and motion cues, we dynamically adjust the threshold of Leaky Integrate-and-Fire neurons based on global motion and contextual semantic information. Our experiments validate the effectiveness of MDST++, demonstrating their consistent superiority over state-of-the-art methods on mainstream benchmarks. Additionally, incorporating motion and multi-timescale information significantly improves HM and ZSL accuracy by 26.2\% and 39.9\%.