Video Reconstruction by Spatio-Temporal Fusion of Blurred-Coded Image Pair

TL;DR

This paper introduces a deep learning framework that combines fully-exposed and coded images from a C2B camera to reconstruct high-quality, unambiguous videos, overcoming limitations of previous single-image methods.

Contribution

It proposes a novel fusion-based approach that leverages both static scene information and motion cues to improve video reconstruction from blurred-coded image pairs.

Findings

Outperforms single-image based methods in video quality

Reduces motion ambiguity in reconstructed videos

Demonstrates effectiveness on C2B camera data

Abstract

Learning-based methods have enabled the recovery of a video sequence from a single motion-blurred image or a single coded exposure image. Recovering video from a single motion-blurred image is a very ill-posed problem and the recovered video usually has many artifacts. In addition to this, the direction of motion is lost and it results in motion ambiguity. However, it has the advantage of fully preserving the information in the static parts of the scene. The traditional coded exposure framework is better-posed but it only samples a fraction of the space-time volume, which is at best 50% of the space-time volume. Here, we propose to use the complementary information present in the fully-exposed (blurred) image along with the coded exposure image to recover a high fidelity video without any motion ambiguity. Our framework consists of a shared encoder followed by an attention module to selectively combine the spatial information from the fully-exposed image with the temporal information from the coded image, which is then super-resolved to recover a non-ambiguous high-quality video. The input to our algorithm is a fully-exposed and coded image pair. Such an acquisition system already exists in the form of a Coded-two-bucket (C2B) camera. We demonstrate that our proposed deep learning approach using blurred-coded image pair produces much better results than those from just a blurred image or just a coded image.