EZSR: Event-based Zero-Shot Recognition

TL;DR

This paper introduces EZSR, a novel event-based zero-shot object recognition method that leverages a new event encoder and data synthesis techniques to outperform existing approaches on benchmark datasets.

Contribution

The study develops an event encoder without reconstruction networks, analyzes performance bottlenecks, and proposes a scalar-wise modulation strategy for improved zero-shot recognition.

Findings

Achieves 47.84% zero-shot accuracy on N-ImageNet with ViT/B-16.

Demonstrates superior performance over previous methods on standard benchmarks.

Shows effective scaling with increased parameters and synthesized data.

Abstract

This paper studies zero-shot object recognition using event camera data. Guided by CLIP, which is pre-trained on RGB images, existing approaches achieve zero-shot object recognition by optimizing embedding similarities between event data and RGB images respectively encoded by an event encoder and the CLIP image encoder. Alternatively, several methods learn RGB frame reconstructions from event data for the CLIP image encoder. However, they often result in suboptimal zero-shot performance. This study develops an event encoder without relying on additional reconstruction networks. We theoretically analyze the performance bottlenecks of previous approaches: the embedding optimization objectives are prone to suffer from the spatial sparsity of event data, causing semantic misalignments between the learned event embedding space and the CLIP text embedding space. To mitigate the issue, we explore a scalar-wise modulation strategy. Furthermore, to scale up the number of events and RGB data pairs for training, we also study a pipeline for synthesizing event data from static RGB images in mass. Experimentally, we demonstrate an attractive scaling property in the number of parameters and synthesized data. We achieve superior zero-shot object recognition performance on extensive standard benchmark datasets, even compared with past supervised learning approaches. For example, our model with a ViT/B-16 backbone achieves 47.84% zero-shot accuracy on the N-ImageNet dataset.