Learning Sensor Multiplexing Design through Back-propagation

TL;DR

This paper introduces a method to jointly learn camera sensor multiplexing patterns and image reconstruction networks using back-propagation, resulting in more accurate color imaging than traditional methods.

Contribution

It presents a novel approach to optimize sensor design and inference simultaneously through end-to-end training, improving upon existing color filter arrangements.

Findings

Learned sensor patterns outperform Bayer pattern in accuracy.

Automatically discovers sparse and optimized color measurement layouts.

Joint training enhances overall image reconstruction quality.

Abstract

Recent progress on many imaging and vision tasks has been driven by the use of deep feed-forward neural networks, which are trained by propagating gradients of a loss defined on the final output, back through the network up to the first layer that operates directly on the image. We propose back-propagating one step further---to learn camera sensor designs jointly with networks that carry out inference on the images they capture. In this paper, we specifically consider the design and inference problems in a typical color camera---where the sensor is able to measure only one color channel at each pixel location, and computational inference is required to reconstruct a full color image. We learn the camera sensor's color multiplexing pattern by encoding it as layer whose learnable weights determine which color channel, from among a fixed set, will be measured at each location. These weights are jointly trained with those of a reconstruction network that operates on the corresponding sensor measurements to produce a full color image. Our network achieves significant improvements in accuracy over the traditional Bayer pattern used in most color cameras. It automatically learns to employ a sparse color measurement approach similar to that of a recent design, and moreover, improves upon that design by learning an optimal layout for these measurements.