DRIFT: Deep Restoration, ISP Fusion, and Tone-mapping

TL;DR

DRIFT is an efficient mobile camera pipeline that combines deep learning techniques for multi-frame processing and tone-mapping to produce high-quality RGB images from raw captures.

Contribution

It introduces a novel AI-based pipeline integrating multi-frame processing and tone-mapping optimized for mobile devices, improving image quality and tone control.

Findings

Outperforms state-of-the-art methods in qualitative and quantitative metrics.

Enables tone-tunable and consistent high-resolution image processing on mobile devices.

Efficiently combines denoising, demosaicing, super-resolution, and tone-mapping in a unified pipeline.

Abstract

Smartphone cameras have gained immense popularity with the adoption of high-resolution and high-dynamic range imaging. As a result, high-performance camera Image Signal Processors (ISPs) are crucial in generating high-quality images for the end user while keeping computational costs low. In this paper, we propose DRIFT (Deep Restoration, ISP Fusion, and Tone-mapping): an efficient AI mobile camera pipeline that generates high quality RGB images from hand-held raw captures. The first stage of DRIFT is a Multi-Frame Processing (MFP) network that is trained using a adversarial perceptual loss to perform multi-frame alignment, denoising, demosaicing, and super-resolution. Then, the output of DRIFT-MFP is processed by a novel deep-learning based tone-mapping (DRIFT-TM) solution that allows for tone tunability, ensures tone-consistency with a reference pipeline, and can be run efficiently for high-resolution images on a mobile device. We show qualitative and quantitative comparisons against state-of-the-art MFP and tone-mapping methods to demonstrate the effectiveness of our approach.