Expanding Event Modality Applications through a Robust CLIP-Based Encoder

TL;DR

This paper presents a robust CLIP-based encoder that effectively transfers image-model capabilities to event data, enabling improved zero-shot, few-shot, and cross-modal applications across diverse domains.

Contribution

We adapt CLIP architecture for event data, supporting zero-shot learning and cross-modal interactions without additional training, expanding its applicability.

Findings

Strong object recognition performance in event data

Effective zero-shot and few-shot learning results

Generalizes well to video-derived events

Abstract

This paper introduces a powerful encoder that transfers CLIP`s capabilities to event-based data, enhancing its utility and expanding its applicability across diverse domains. While large-scale datasets have significantly advanced image-based models, the scarcity of comprehensive event datasets has limited performance potential in event modality. To address this challenge, we adapt CLIP`s architecture to align event embeddings with image embeddings, supporting zero-shot learning and preserving text alignment while mitigating catastrophic forgetting. Our encoder achieves strong performance in object recognition, with competitive results in zero-shot and few-shot learning tasks. Notably, it generalizes effectively to events extracted from video data without requiring additional training, highlighting its versatility. Additionally, we integrate this encoder within a cross-modality framework that facilitates interaction across five modalities-Image, Event, Text, Sound, and Depth-expanding the possibilities for cross-modal applications. Overall, this work underscores the transformative potential of a robust event encoder, broadening the scope and utility of event-based data across various fields.