Key frames assisted hybrid encoding for photorealistic compressive video sensing

TL;DR

This paper introduces KH-CVS, a hybrid encoding method for compressive video sensing that combines key frames and encoded frames, using deep learning to improve photorealistic reconstruction quality.

Contribution

The paper proposes a novel hybrid encoding paradigm with key frames and encoded frames, utilizing deep neural networks and optical flow for enhanced video reconstruction.

Findings

Outperforms existing methods in simulation and real data

Achieves higher reconstruction quality with hybrid encoding

Demonstrates effectiveness of deep learning in SCI reconstruction

Abstract

Snapshot compressive imaging (SCI) encodes high-speed scene video into a snapshot measurement and then computationally makes reconstructions, allowing for efficient high-dimensional data acquisition. Numerous algorithms, ranging from regularization-based optimization and deep learning, are being investigated to improve reconstruction quality, but they are still limited by the ill-posed and information-deficient nature of the standard SCI paradigm. To overcome these drawbacks, we propose a new key frames assisted hybrid encoding paradigm for compressive video sensing, termed KH-CVS, that alternatively captures short-exposure key frames without coding and long-exposure encoded compressive frames to jointly reconstruct photorealistic video. With the use of optical flow and spatial warping, a deep convolutional neural network framework is constructed to integrate the benefits of these two types of frames. Extensive experiments on both simulations and real data from the prototype we developed verify the superiority of the proposed method.