Exploit Camera Raw Data for Video Super-Resolution via Hidden Markov Model Inference

TL;DR

This paper introduces a novel video super-resolution method that directly utilizes raw camera sensor data, employing a Hidden Markov Model-based inference framework and specialized neural modules, leading to superior results over existing methods.

Contribution

The paper presents a new VSR approach that exploits raw camera data and a Hidden Markov Model inference-based architecture, along with a dedicated raw video dataset for training and evaluation.

Findings

Outperforms state-of-the-art VSR methods on raw data

Effectively separates super-resolution and color correction processes

Demonstrates adaptability to different camera-ISP pipelines

Abstract

To the best of our knowledge, the existing deep-learning-based Video Super-Resolution (VSR) methods exclusively make use of videos produced by the Image Signal Processor (ISP) of the camera system as inputs. Such methods are 1) inherently suboptimal due to information loss incurred by non-invertible operations in ISP, and 2) inconsistent with the real imaging pipeline where VSR in fact serves as a pre-processing unit of ISP. To address this issue, we propose a new VSR method that can directly exploit camera sensor data, accompanied by a carefully built Raw Video Dataset (RawVD) for training, validation, and testing. This method consists of a Successive Deep Inference (SDI) module and a reconstruction module, among others. The SDI module is designed according to the architectural principle suggested by a canonical decomposition result for Hidden Markov Model (HMM) inference; it estimates the target high-resolution frame by repeatedly performing pairwise feature fusion using deformable convolutions. The reconstruction module, built with elaborately designed Attention-based Residual Dense Blocks (ARDBs), serves the purpose of 1) refining the fused feature and 2) learning the color information needed to generate a spatial-specific transformation for accurate color correction. Extensive experiments demonstrate that owing to the informativeness of the camera raw data, the effectiveness of the network architecture, and the separation of super-resolution and color correction processes, the proposed method achieves superior VSR results compared to the state-of-the-art and can be adapted to any specific camera-ISP. Code and dataset are available at https://github.com/proteus1991/RawVSR.