CrossZoom: Simultaneously Motion Deblurring and Event Super-Resolving

TL;DR

CrossZoom is a unified neural network that simultaneously improves motion clarity in images and resolution of event data, bridging the modality gap for enhanced vision applications.

Contribution

The paper introduces CZ-Net, a novel multi-scale fusion architecture with attention modules for joint motion deblurring and event super-resolution, along with a new high-resolution dataset.

Findings

Effective in restoring sharp images from blurry inputs

Enhances event resolution and reduces distortions

Outperforms existing methods on synthetic and real data

Abstract

Even though the collaboration between traditional and neuromorphic event cameras brings prosperity to frame-event based vision applications, the performance is still confined by the resolution gap crossing two modalities in both spatial and temporal domains. This paper is devoted to bridging the gap by increasing the temporal resolution for images, i.e., motion deblurring, and the spatial resolution for events, i.e., event super-resolving, respectively. To this end, we introduce CrossZoom, a novel unified neural Network (CZ-Net) to jointly recover sharp latent sequences within the exposure period of a blurry input and the corresponding High-Resolution (HR) events. Specifically, we present a multi-scale blur-event fusion architecture that leverages the scale-variant properties and effectively fuses cross-modality information to achieve cross-enhancement. Attention-based adaptive enhancement and cross-interaction prediction modules are devised to alleviate the distortions inherent in Low-Resolution (LR) events and enhance the final results through the prior blur-event complementary information. Furthermore, we propose a new dataset containing HR sharp-blurry images and the corresponding HR-LR event streams to facilitate future research. Extensive qualitative and quantitative experiments on synthetic and real-world datasets demonstrate the effectiveness and robustness of the proposed method. Codes and datasets are released at https://bestrivenzc.github.io/CZ-Net/.