DemosaicFormer: Coarse-to-Fine Demosaicing Network for HybridEVS Camera

TL;DR

DemosaicFormer is a novel coarse-to-fine neural network designed for HybridEVS cameras, effectively improving image reconstruction quality by integrating multi-scale features and robust training strategies, outperforming existing methods.

Contribution

The paper introduces DemosaicFormer, a new demosaicing framework with a Multi-Scale Gating Module and progressive training, tailored for HybridEVS cameras, addressing the lack of specialized ISP pipelines.

Findings

Achieves superior qualitative and quantitative performance.

Outperforms existing methods in the MIPI 2024 challenge.

Demonstrates robustness through data augmentation and progressive training.

Abstract

Hybrid Event-Based Vision Sensor (HybridEVS) is a novel sensor integrating traditional frame-based and event-based sensors, offering substantial benefits for applications requiring low-light, high dynamic range, and low-latency environments, such as smartphones and wearable devices. Despite its potential, the lack of Image signal processing (ISP) pipeline specifically designed for HybridEVS poses a significant challenge. To address this challenge, in this study, we propose a coarse-to-fine framework named DemosaicFormer which comprises coarse demosaicing and pixel correction. Coarse demosaicing network is designed to produce a preliminary high-quality estimate of the RGB image from the HybridEVS raw data while the pixel correction network enhances the performance of image restoration and mitigates the impact of defective pixels. Our key innovation is the design of a Multi-Scale Gating Module (MSGM) applying the integration of cross-scale features, which allows feature information to flow between different scales. Additionally, the adoption of progressive training and data augmentation strategies further improves model's robustness and effectiveness. Experimental results show superior performance against the existing methods both qualitatively and visually, and our DemosaicFormer achieves the best performance in terms of all the evaluation metrics in the MIPI 2024 challenge on Demosaic for Hybridevs Camera. The code is available at https://github.com/QUEAHREN/DemosaicFormer.