Event USKT : U-State Space Model in Knowledge Transfer for Event Cameras

TL;DR

This paper introduces USKT, a U-shaped state space model for event-to-RGB knowledge transfer, enabling effective use of event camera data with minimal tuning and improved performance on multiple datasets.

Contribution

The paper proposes a novel USKT framework with a bidirectional reverse state space model for efficient event-to-RGB knowledge transfer and resource conservation.

Findings

USKT improves model accuracy on multiple datasets.

The bidirectional reverse state space model reduces computational costs.

USKT demonstrates adaptability across different tasks.

Abstract

Event cameras, as an emerging imaging technology, offer distinct advantages over traditional RGB cameras, including reduced energy consumption and higher frame rates. However, the limited quantity of available event data presents a significant challenge, hindering their broader development. To alleviate this issue, we introduce a tailored U-shaped State Space Model Knowledge Transfer (USKT) framework for Event-to-RGB knowledge transfer. This framework generates inputs compatible with RGB frames, enabling event data to effectively reuse pre-trained RGB models and achieve competitive performance with minimal parameter tuning. Within the USKT architecture, we also propose a bidirectional reverse state space model. Unlike conventional bidirectional scanning mechanisms, the proposed Bidirectional Reverse State Space Model (BiR-SSM) leverages a shared weight strategy, which facilitates efficient modeling while conserving computational resources. In terms of effectiveness, integrating USKT with ResNet50 as the backbone improves model performance by 0.95%, 3.57%, and 2.9% on DVS128 Gesture, N-Caltech101, and CIFAR-10-DVS datasets, respectively, underscoring USKT's adaptability and effectiveness. The code will be made available upon acceptance.