Towards Robust Event-based Networks for Nighttime via Unpaired Day-to-Night Event Translation

TL;DR

This paper introduces an unpaired event-to-event translation model using Diffusion GAN to convert day events into night events, improving nighttime scene understanding for event cameras with high dynamic range.

Contribution

It presents a novel unpaired day-to-night event translation framework with wavelet analysis and contrastive learning, addressing modality differences and data imbalance in event camera data.

Findings

Successful day-to-night event translation preserving event characteristics

Mitigates performance degradation in nighttime event-based tasks

Introduces new metrics for unpaired event translation evaluation

Abstract

Event cameras with high dynamic range ensure scene capture even in low-light conditions. However, night events exhibit patterns different from those captured during the day. This difference causes performance degradation when applying night events to a model trained solely on day events. This limitation persists due to a lack of annotated night events. To overcome the limitation, we aim to alleviate data imbalance by translating annotated day data into night events. However, generating events from different modalities challenges reproducing their unique properties. Accordingly, we propose an unpaired event-to-event day-to-night translation model that effectively learns to map from one domain to another using Diffusion GAN. The proposed translation model analyzes events in spatio-temporal dimension with wavelet decomposition and disentangled convolution layers. We also propose a new temporal contrastive learning with a novel shuffling and sampling strategy to regularize temporal continuity. To validate the efficacy of the proposed methodology, we redesign metrics for evaluating events translated in an unpaired setting, aligning them with the event modality for the first time. Our framework shows the successful day-to-night event translation while preserving the characteristics of events. In addition, through our translation method, we facilitate event-based modes to learn about night events by translating annotated day events into night events. Our approach effectively mitigates the performance degradation of applying real night events to downstream tasks. The code is available at https://github.com/jeongyh98/UDNET.