Label-Free Event-based Object Recognition via Joint Learning with Image Reconstruction from Events

TL;DR

This paper introduces a novel label-free event-based object recognition method that jointly reconstructs images from events and recognizes objects using CLIP, leveraging category-guided losses and a prototype approach for improved zero-shot recognition.

Contribution

It proposes a joint learning framework combining image reconstruction and object recognition without labels, utilizing CLIP and category-guided losses for enhanced zero-shot performance.

Findings

Outperforms existing methods in zero-shot recognition tasks

Effective joint learning improves both image reconstruction and recognition accuracy

Prototype-based approach enhances prediction quality with unpaired images

Abstract

Recognizing objects from sparse and noisy events becomes extremely difficult when paired images and category labels do not exist. In this paper, we study label-free event-based object recognition where category labels and paired images are not available. To this end, we propose a joint formulation of object recognition and image reconstruction in a complementary manner. Our method first reconstructs images from events and performs object recognition through Contrastive Language-Image Pre-training (CLIP), enabling better recognition through a rich context of images. Since the category information is essential in reconstructing images, we propose category-guided attraction loss and category-agnostic repulsion loss to bridge the textual features of predicted categories and the visual features of reconstructed images using CLIP. Moreover, we introduce a reliable data sampling strategy and local-global reconstruction consistency to boost joint learning of two tasks. To enhance the accuracy of prediction and quality of reconstruction, we also propose a prototype-based approach using unpaired images. Extensive experiments demonstrate the superiority of our method and its extensibility for zero-shot object recognition. Our project code is available at \url{https://github.com/Chohoonhee/Ev-LaFOR}.