Learning Binary Color Filter Arrays with Trainable Hard Thresholding

TL;DR

This paper introduces HardMax, a deep learning method for learning binary color filter arrays that are practical for digital cameras, demonstrating superior reconstruction performance over existing methods.

Contribution

It proposes a novel binary CFA learning module using hard thresholding with a straight-through estimator, enabling physically implementable filters in camera systems.

Findings

Higher reconstruction performance than hand-crafted filters

Effective across various demosaicing models and configurations

Validated on Kodak and BSDS500 datasets

Abstract

Color Filter Arrays (CFA) are optical filters in digital cameras that capture specific color channels. Current commercial CFAs are hand-crafted patterns with different physical and application-specific considerations. This study proposes a binary CFA learning module based on hard thresholding with a deep learning-based demosaicing network in a joint architecture. Unlike most existing learnable CFAs that learn a channel from the whole color spectrum or linearly combine available digital colors, this method learns a binary channel selection, resulting in CFAs that are practical and physically implementable to digital cameras. The binary selection is based on adapting the hard thresholding operation into neural networks via a straight-through estimator, and therefore it is named HardMax. This paper includes the background on the CFA design problem, the description of the HardMax method, and the performance evaluation results. The evaluation of the proposed method includes tests for different demosaicing models, color configurations, filter sizes, and a comparison with existing methods in various reconstruction metrics. The proposed approach is tested with Kodak and BSDS500 datasets and provides higher reconstruction performance than hand-crafted or alternative learned binary filters.