GenMFSR: Generative Multi-Frame Image Restoration and Super-Resolution

TL;DR

GenMFSR is a novel generative pipeline that leverages foundation model priors to perform multi-frame raw-to-RGB super-resolution, effectively aligning raw frames and enhancing resolution for camera ISP applications.

Contribution

It introduces the first raw-to-RGB multi-frame super-resolution method that incorporates foundation model priors and alignment, addressing limitations of existing adversarial approaches.

Findings

Successfully aligns multiple raw frames for super-resolution

Restricts generation to high-frequency regions to prevent artifacts

Enhances resolution in camera ISP pipelines

Abstract

Camera pipelines receive raw Bayer-format frames that need to be denoised, demosaiced, and often super-resolved. Multiple frames are captured to utilize natural hand tremors and enhance resolution. Multi-frame super-resolution is therefore a fundamental problem in camera pipelines. Existing adversarial methods are constrained by the quality of ground truth. We propose GenMFSR, the first Generative Multi-Frame Raw-to-RGB Super Resolution pipeline, that incorporates image priors from foundation models to obtain sub-pixel information for camera ISP applications. GenMFSR can align multiple raw frames, unlike existing single-frame super-resolution methods, and we propose a loss term that restricts generation to high-frequency regions in the raw domain, thus preventing low-frequency artifacts.