Efficient Unified Demosaicing for Bayer and Non-Bayer Patterned Image Sensors

TL;DR

This paper introduces KLAP, a unified, efficient demosaicing method applicable to both Bayer and non-Bayer CMOS image sensors, utilizing adaptive filters and meta-learning to improve performance and artifact removal in real RAW data.

Contribution

The paper presents a novel unified demosaicing approach that handles multiple CFA patterns with minimal filters and employs meta-learning for artifact removal, outperforming existing methods.

Findings

Achieved state-of-the-art performance on synthetic and real RAW data.

Effectively demosaics various CFA patterns with only 1% key filters.

Successfully removes sensor artifacts using meta-learning during inference.

Abstract

As the physical size of recent CMOS image sensors (CIS) gets smaller, the latest mobile cameras are adopting unique non-Bayer color filter array (CFA) patterns (e.g., Quad, Nona, QxQ), which consist of homogeneous color units with adjacent pixels. These non-Bayer sensors are superior to conventional Bayer CFA thanks to their changeable pixel-bin sizes for different light conditions but may introduce visual artifacts during demosaicing due to their inherent pixel pattern structures and sensor hardware characteristics. Previous demosaicing methods have primarily focused on Bayer CFA, necessitating distinct reconstruction methods for non-Bayer patterned CIS with various CFA modes under different lighting conditions. In this work, we propose an efficient unified demosaicing method that can be applied to both conventional Bayer RAW and various non-Bayer CFAs' RAW data in different operation modes. Our Knowledge Learning-based demosaicing model for Adaptive Patterns, namely KLAP, utilizes CFA-adaptive filters for only 1% key filters in the network for each CFA, but still manages to effectively demosaic all the CFAs, yielding comparable performance to the large-scale models. Furthermore, by employing meta-learning during inference (KLAP-M), our model is able to eliminate unknown sensor-generic artifacts in real RAW data, effectively bridging the gap between synthetic images and real sensor RAW. Our KLAP and KLAP-M methods achieved state-of-the-art demosaicing performance in both synthetic and real RAW data of Bayer and non-Bayer CFAs.