UniINR: Event-guided Unified Rolling Shutter Correction, Deblurring, and Interpolation

TL;DR

UniINR is a novel neural implicit representation method that simultaneously corrects rolling shutter distortion, deblurs, and interpolates frames to recover high-quality global shutter images from fast-moving camera videos using event data.

Contribution

This work introduces the first unified neural implicit approach for simultaneous RS correction, deblurring, and frame interpolation leveraging event cameras, outperforming prior separate methods.

Findings

Achieves high-quality GS frame recovery from RS blur with minimal artifacts.

Runs efficiently at 2.83ms per frame for 31x frame interpolation.

Outperforms existing methods significantly in experiments.

Abstract

Video frames captured by rolling shutter (RS) cameras during fast camera movement frequently exhibit RS distortion and blur simultaneously. Naturally, recovering high-frame-rate global shutter (GS) sharp frames from an RS blur frame must simultaneously consider RS correction, deblur, and frame interpolation. A naive way is to decompose the whole process into separate tasks and cascade existing methods; however, this results in cumulative errors and noticeable artifacts. Event cameras enjoy many advantages, e.g., high temporal resolution, making them potential for our problem. To this end, we propose the first and novel approach, named UniINR, to recover arbitrary frame-rate sharp GS frames from an RS blur frame and paired events. Our key idea is unifying spatial-temporal implicit neural representation (INR) to directly map the position and time coordinates to color values to address the interlocking degradations. Specifically, we introduce spatial-temporal implicit encoding (STE) to convert an RS blur image and events into a spatial-temporal representation (STR). To query a specific sharp frame (GS or RS), we embed the exposure time into STR and decode the embedded features pixel-by-pixel to recover a sharp frame. Our method features a lightweight model with only 0.38M parameters, and it also enjoys high inference efficiency, achieving 2.83ms/frame in 31 times frame interpolation of an RS blur frame. Extensive experiments show that our method significantly outperforms prior methods. Code is available at https://github.com/yunfanLu/UniINR.