Joint RGB-Spectral Decomposition Model Guided Image Enhancement in Mobile Photography

TL;DR

This paper introduces a joint RGB-spectral decomposition model for image enhancement in mobile photography, leveraging spectral data and priors to improve image quality and establish a foundation for spectral vision on mobile devices.

Contribution

It proposes a novel joint decomposition and prior-guided enhancement framework utilizing spectral and RGB data, along with constructing a new high-quality Mobile-Spec dataset.

Findings

Lr-MSI effectively improves tone enhancement results.

The proposed model enhances dynamic range and color mapping.

Experimental results validate the approach's effectiveness.

Abstract

The integration of miniaturized spectrometers into mobile devices offers new avenues for image quality enhancement and facilitates novel downstream tasks. However, the broader application of spectral sensors in mobile photography is hindered by the inherent complexity of spectral images and the constraints of spectral imaging capabilities. To overcome these challenges, we propose a joint RGB-Spectral decomposition model guided enhancement framework, which consists of two steps: joint decomposition and prior-guided enhancement. Firstly, we leverage the complementarity between RGB and Low-resolution Multi-Spectral Images (Lr-MSI) to predict shading, reflectance, and material semantic priors. Subsequently, these priors are seamlessly integrated into the established HDRNet to promote dynamic range enhancement, color mapping, and grid expert learning, respectively. Additionally, we construct a high-quality Mobile-Spec dataset to support our research, and our experiments validate the effectiveness of Lr-MSI in the tone enhancement task. This work aims to establish a solid foundation for advancing spectral vision in mobile photography. The code is available at \url{https://github.com/CalayZhou/JDM-HDRNet}.