CMTA: Cross-Modal Temporal Alignment for Event-guided Video Deblurring

TL;DR

This paper introduces a novel cross-modal temporal alignment approach leveraging event camera data to significantly improve video deblurring, especially in severe motion blur scenarios, by enhancing feature extraction and temporal alignment.

Contribution

It proposes two new modules for intra-frame feature enhancement and inter-frame temporal alignment, along with a real-world dataset for event-guided video deblurring, outperforming existing methods.

Findings

Outperforms state-of-the-art methods on synthetic datasets

Demonstrates effectiveness on real-world blurred videos

Provides a new dataset for event-guided deblurring

Abstract

Video deblurring aims to enhance the quality of restored results in motion-blurred videos by effectively gathering information from adjacent video frames to compensate for the insufficient data in a single blurred frame. However, when faced with consecutively severe motion blur situations, frame-based video deblurring methods often fail to find accurate temporal correspondence among neighboring video frames, leading to diminished performance. To address this limitation, we aim to solve the video deblurring task by leveraging an event camera with micro-second temporal resolution. To fully exploit the dense temporal resolution of the event camera, we propose two modules: 1) Intra-frame feature enhancement operates within the exposure time of a single blurred frame, iteratively enhancing cross-modality features in a recurrent manner to better utilize the rich temporal information of events, 2) Inter-frame temporal feature alignment gathers valuable long-range temporal information to target frames, aggregating sharp features leveraging the advantages of the events. In addition, we present a novel dataset composed of real-world blurred RGB videos, corresponding sharp videos, and event data. This dataset serves as a valuable resource for evaluating event-guided deblurring methods. We demonstrate that our proposed methods outperform state-of-the-art frame-based and event-based motion deblurring methods through extensive experiments conducted on both synthetic and real-world deblurring datasets. The code and dataset are available at https://github.com/intelpro/CMTA.