Merging-ISP: Multi-Exposure High Dynamic Range Image Signal Processing

TL;DR

This paper introduces Merging-ISP, a deep neural network-based pipeline for HDR imaging that jointly performs all processing steps, resulting in higher quality HDR images and outperforming traditional cascaded methods.

Contribution

The paper presents a novel end-to-end trainable neural network pipeline for multi-exposure HDR image processing, reducing error propagation and improving image quality.

Findings

Outperforms state-of-the-art cascaded pipelines by over 1 dB PSNR.

Provides high perceptual quality HDR reconstructions.

Jointly solves all HDR processing subtasks with a deep neural network.

Abstract

High dynamic range (HDR) imaging combines multiple images with different exposure times into a single high-quality image. The image signal processing pipeline (ISP) is a core component in digital cameras to perform these operations. It includes demosaicing of raw color filter array (CFA) data at different exposure times, alignment of the exposures, conversion to HDR domain, and exposure merging into an HDR image. Traditionally, such pipelines cascade algorithms that address these individual subtasks. However, cascaded designs suffer from error propagation, since simply combining multiple steps is not necessarily optimal for the entire imaging task. This paper proposes a multi-exposure HDR image signal processing pipeline (Merging-ISP) to jointly solve all these subtasks. Our pipeline is modeled by a deep neural network architecture. As such, it is end-to-end trainable, circumvents the use of hand-crafted and potentially complex algorithms, and mitigates error propagation. Merging-ISP enables direct reconstructions of HDR images of dynamic scenes from multiple raw CFA images with different exposures. We compare Merging-ISP against several state-of-the-art cascaded pipelines. The proposed method provides HDR reconstructions of high perceptual quality and it quantitatively outperforms competing ISPs by more than 1 dB in terms of PSNR.