Deep Demosaicing for Edge Implementation

TL;DR

This paper explores efficient deep learning architectures for demosaicing in digital cameras, aiming to optimize performance on low-end edge devices by balancing accuracy and model complexity.

Contribution

It provides an exhaustive search for neural network architectures that optimize the trade-off between CPSNR and model size, and clarifies conditions for loss function convergence.

Findings

Identified Pareto-optimal architectures for demosaicing.

Achieved state-of-the-art CPSNR with fewer parameters.

Provided theoretical insights into architecture search methods.

Abstract

Most digital cameras use sensors coated with a Color Filter Array (CFA) to capture channel components at every pixel location, resulting in a mosaic image that does not contain pixel values in all channels. Current research on reconstructing these missing channels, also known as demosaicing, introduces many artifacts, such as zipper effect and false color. Many deep learning demosaicing techniques outperform other classical techniques in reducing the impact of artifacts. However, most of these models tend to be over-parametrized. Consequently, edge implementation of the state-of-the-art deep learning-based demosaicing algorithms on low-end edge devices is a major challenge. We provide an exhaustive search of deep neural network architectures and obtain a pareto front of Color Peak Signal to Noise Ratio (CPSNR) as the performance criterion versus the number of parameters as the model complexity that beats the state-of-the-art. Architectures on the pareto front can then be used to choose the best architecture for a variety of resource constraints. Simple architecture search methods such as exhaustive search and grid search require some conditions of the loss function to converge to the optimum. We clarify these conditions in a brief theoretical study.

Tables

Table 1. Table 1: The negative CPSNR denoted by ℒ ( . ) {\mathcal{L}}(.) versus the number of parameters denoted by 𝒞 ( . ) {\mathcal{C}}(.) for few architectures of the search space on the MCM dataset. The first line in bold shows the state-of-the-art.

Model	No of blocks	Skip length	$ℒ(.)$	Standard Error	$𝒞(.)$
64 filter	20	20	$-37.58$	0.66	710275
64 filter	15	15	$-37.51$	0.62	487171
64 filter	10	10	$-37.45$	0.64	338435
256 filter	3	2	$-37.77$	0.68	420879
128 filter	7	2	$-37.94$	0.67	252431
128 filter	5	2	$-37.82$	0.69	182287

Equations

$$I(i,j)=\{\begin{array}{ll}{R}_{i,j}& \text{for i odd and j even,}\\ B_{i,j}& \text{for i even and j odd,}\\ G_{i,j}& \text{Otherwise,}\end{array}$$

$$\hat{I}(i,j)=\{\begin{array}{ll}(R_{i,j},{\hat{B}}_{i,j},{\hat{G}}_{i,j})& \text{for i odd and j even,}\\ ({\hat{R}}_{i,j},B_{i,j},{\hat{G}}_{i,j})& \text{for i even and j odd,}\\ ({\hat{R}}_{i,j},{\hat{B}}_{i,j},G_{i,j})& \text{Otherwise,}\end{array}$$

$$\mathrm{MSE}(k)=\frac{1}{MN}\sum _{i=1}^M\sum _{j=1}^N\{{\hat{I}}_k(i,j)-I_k(i,j){\}}^2,$$

$$\mathrm{CMSE}(I,\hat{I})=\frac{1}{3MN}\sum _{k\in \{\mathrm{R},\mathrm{G},\mathrm{B}\}}\sum _{i=1}^M\sum _{j=1}^N\{{\hat{I}}_k(i,j)-I_k(i,j){\}}^2,$$

$$\mathrm{PSNR}(k)=10{\log }_{10}\left(\frac{R^2}{\mathrm{MSE}}\right),$$

$$\underset{\mathbf{\theta }\in \Theta }{\mathrm{argmin}}\mathcal{L}(\mathbf{\theta })\ne \varnothing$$

$${\mathbf{\theta }}^{\ast }=\underset{\mathbf{\theta }\in \Theta }{\mathrm{argmin}}\mathcal{L}(\mathbf{\theta }),{\mathcal{L}}^{\ast }=\underset{\mathbf{\theta }\in \Theta }{\min }\mathcal{L}(\mathbf{\theta }).$$

$$\underset{k\to \infty }{\lim }{\mathcal{L}}_k^{\ast }={\mathcal{L}}^{\ast }.$$

$$\exists \delta >0,{\mathbf{\theta }}_k\in {\Theta }_k\mathrm{where}||{\mathbf{\theta }}_k-{\mathbf{\theta }}^{\ast }||<\delta ,|\mathcal{L}({\mathbf{\theta }}_k)-\mathcal{L}({\mathbf{\theta }}^{\ast })|<ϵ,$$

$${\mathcal{L}}^{\ast }\le {\mathcal{L}}_k^{\ast }\le \mathcal{L}({\theta }_k)\le {\mathcal{L}}^{\ast }+ϵ,$$

$$⟹\underset{k\to \infty }{\lim }{\mathcal{L}}_k^{\ast }={\mathcal{L}}^{\ast }.$$

$$\underset{k\to \infty }{\lim }\underset{i\to \infty }{\lim }{\mathbf{\theta }}_{ik}^{\ast }=\tilde{\mathbf{\theta }}.$$

$$\underset{k\to \infty }{\lim }\underset{i\to \infty }{\lim }\mathcal{L}({\mathbf{\theta }}_{ik}^{\ast })=\mathcal{L}(\tilde{\mathbf{\theta }}),$$

$${\Theta }_n=\{\mathbf{\theta }\mid \mathbf{\theta }=l_i+j{\delta }_i,j=0,\dots ,n-1,i=1,\dots ,d\}$$

$${\mathbf{\theta }}_n^{\ast }=\underset{\theta \in {\Theta }_n}{\mathrm{argmin}}\mathcal{L}(\mathbf{\theta }),{\mathcal{L}}_n^{\ast }=\underset{\theta \in {\Theta }_n}{\min }\mathcal{L}(\mathbf{\theta }).$$

$$\forall \theta ,{\theta }^{\prime }\in [a,b],|\mathcal{L}(\theta )-\mathcal{L}({\theta }^{\prime })|\le M|\theta -{\theta }^{\prime }|,$$

$${\mathcal{L}}_n^{\ast }-\frac{M\delta }{2}\le {\mathcal{L}}^{\ast }$$

$$|\mathcal{L}({\theta }_n^{\ast })-\mathcal{L}({\theta }^{\ast })|\le M|{\theta }_n^{\ast }-{\theta }^{\ast }|\le \frac{M\delta }{2}$$

$$⟹{\mathcal{L}}_n^{\ast }-\frac{M\delta }{2}\le {\mathcal{L}}^{\ast }.$$

Full text

Copyright Notice: This paper has been accepted in 16th International Conference of Image Analysis and Recognition (ICIAR 2019). Authors have assigned The Springer Verlage all rights under the Springer copyright. The revised version of the article is published in the Lecture Notes in Computer Science Springer series under the same title.

1 Introduction

Deep Learning is changing lives on a day to day basis. Overparametrization of deep models can have an impact in their deployment for variety of applications. For instance, model size could adversely affect the real-time updates in mobile phone implementation. Moreover, the training and inference time increases for a large model. For instance, ResNet152 takes 1.5 weeks to train on the Image-Net dataset using one Nvidia TitanX GPU (Han et al., 2015). In terms of hardware, much of the impact of such models is in the energy consumption. As an example, Dynamic Random Access Memory (DRAM) consumes most of the energy at inference. Therefore, it is imperative to reduce the model size (Han et al., 2016).

The success of deep learning models has been proven in many high-level vision tasks such as image classification and object detection. However, deep learning solutions are less explored for low-level vision problems where the edge implementation is an obligation. Previous work shows the effectiveness on other low-level vision problems such as image denoising and image demosaicing (Xie et al., 2012). Here we explore various architectures for demosaicing, with the aim of finding suitable models for deployment on the edge. It is imperative to make use of models that can be stored on edge devices.

2 Demosaicing

Image demosaicing involves interpolating full-resolution images from incomplete color samples produced by an image sensor. Limitations in camera sensor resolution and sensitivity leads to the problem of mosaicing. For a digital camera to capture or produce a full color image, it uses sensors to detect all 3 colors (RGB) at every pixel location. One way this is done involves using a beam-splitter to project the image onto three separate sensors, see Figure 1a, one for each color (RGB). Color filters are placed in front of each sensor to filter specific wavelengths such that three full-channel color images are obtained. This is a costly process that is generally used in scientific-grade microscopes. Most digital cameras, on the other hand, make use of just one sensor coated with a Color Filter Array (CFA). In this system, one color component is captured at every pixel location and the missing channels are reconstructed. The resultant image is often called a mosaic image, derived from the CFA’s mosaic pattern. This mosaic image is then subjected to software-based interpolation, resulting in a full-resolution image; a process termed as demosaicing.

Various existing CFA patterns are currently used, with the Bayer CFA being the most commonly used approach for mosaicing. The Bayer CFA measures the green image on quincinx grid, and the red/blue on the rectangular grids (Figure 1b). The green channel has higher sampling because the human retina has a higher sensitivity toward the green wavelength.

Let $I^{CFA}:\mathbb{Z}^{2}\rightarrow\mathbb{Z}^{3}$ denote an $M\times N$ Bayer CFA image. If we consider the sampling pattern shown in Figure 1b, then the mosaic image is as follows:

[TABLE]

where $R_{i,j}$, $G_{i,j}$, $B_{i,j}$ includes values between 0 and 255. To estimate the missing two missing channels for each pixel location using demosaicing,

[TABLE]

where $\hat{R}_{i,j}$,$\hat{G}_{i,j}$,$\hat{B}_{i,j}$ are the estimates for the channels at each pixel location.

Demosaicing approaches are divided into various categories:

1. Edge-sensitive methods, 2. Directional interpolation and decision methods, 3. Frequency domain approaches, 4. Wavelet-based methods, 5. Statistical reconstruction techniques, and 6. Deep learning (Menon and Calvagno, 2011). Early studies in bi-linear/bi-cubic (Yu, 2006) and spline interpolation make use of single-channel interpolation techniques that treat each channel separately, without using the inter-channel information and correlation. For instance, bi-linear interpolation computes the red value of a non-red pixel as the average of the 2/4 adjacent red pixels; this method is then repeated with the blue and green pixels. Many comparatively advanced algorithms use the results of bi-linear interpolation as a baseline. These algorithms make use of a weighted average of the neighboring pixels to fill the missing channel values.

Recently, neural network approaches like convolutional neural networks (CNN) have been effectively used for joint denoising and demosaicing (Gharbi et al., 2016). This method down-samples the mosaiced image using multiple convolutions and computes the residual at lower resolution; this is further concatenated with the input mosaic image. The final output is then calculated using another group of convolutions at full resolution to reconstruct the full resolution image. Studies (Tan et al., 2018) have also used Deep Residual Learning for image demosaicing for Bayer CFA. This neural network uses a two-stage architecture: 1) the first step recovers an intermediate result of the Green and Red/Blue channel separately as a guiding prior to use as input for the second stage, 2) the final demosaiced image is reconstructed.

The current state-of-the-art model, DMCNN-VD (Syu et al., 2018) architecture (Figure 2) consists of a deep network that uses same dimensions for the input and output layers. They use the Bayer CFA mosaic layer and 20 blocks (CNN, BN, SELU), instead of the 3 layers in the DMCNN model.

2.1 Artifacts

Although various approaches to the demosaicing problem have been discussed, most techniques have a propensity to introduce unwanted artifacts into the resulting images. A previous study shows that bi-linear interpolation of mosaic images results in various structural errors (Lanlan Chang, 2006), such as the two homogeneous areas with 2 gray levels of low and high intensity ($L$ and $H$, L$$<$$H) represented in Figure 3.1. Another artifact introduced is the mismatch in the green and blue/red channel intensity.

These 2 artifacts are generalized as zipper effect and false color. The zipper effect is an abrupt or sudden change in intensity over the neighboring pixels, generally seen along the edges (Figure 3.2a). False colors (Figure 3.2b), on the other hand, are pseudo-colors that are not an accurate representation of the original image. This generally occurs due to inconsistencies in color channels of the original mosaic image. Both these effects are categorized as misguidance color artifacts, generated as a result of wrong interpolation direction. Interestingly, these artifacts are observed even when the interpolation direction is appropriate. However, the effects in this case are less evident when compared to misguided color artifacts. Many deep learning demosaicing techniques outperform other classical techniques in reducing the impact of artifacts.

2.2 Performance Evaluation

A common demosaicing approach is to capture the ground truth images using a professional 3-sensor camera, convert them into a mosaic format using Bayer CFA, interpolate using a demosaicing algorithm, and then compare the results with the ground truth image. Widely used performance criteria are mean Squared Error (MSE) or the equivalent Peak Signal to Noise Ratio (PSNR). The Kodak / McMaster (Yu et al., 2018) dataset is generally used as a baseline because of the realistic scenes and varied complexities. The MSE in each color plane is:

[TABLE]

where $\hat{I}_{k}(i,j)$ is the predicted and $I_{k}(i,j)$ is the original pixel color component. Another measure is the Color Mean Squared Error (CMSE) (Menon and Calvagno, 2011) which averages over the color channel as well,

[TABLE]

The peak signal to noise ratio (PSNR) is defined as,

[TABLE]

where R is 1.0 (for float datatype), R is 255 (for integer datatype).

The Color Peak Signal to Noise Ratio (CPSNR) is computed using CMSE. Higher the PSNR value, better is the quality of the demosaiced image.

3 Architecture Search

For the exhaustive search of architectures, we made use of various parameters chosen based on state-of-the-art models to obtain the Pareto front between negative $\mathrm{CPSNR}$ as the loss and the number of parameters as the model complexity. To make the comparisons fair, we ran each model for 500 epochs and retrained the state-of-the-art architecture in this new setting. The various hyper-parameters for exhaustive search were mostly discrete and are as follows:

The use of depthwise separable convolutions (Sifre and Mallat, 2014), similar to MobileNet V2 architecture (Sandler et al., 2018), over the traditional convolutional layers as per the state-of-the-art models. 2. 2.

The number of filters used in the models (16, 32, 64, 128 and 256). 3. 3.

Number of blocks, each block consisted of sets of i) depthwise separable ii) standard convolution, and iii) activation function, chosen from (3,5,7 blocks). 4. 4.

We use a constant kernel size of 3 $\times$ 3 since much of the hardware has shown to be optimized for this convolution (Ma et al., 2018). 5. 5.

The length of skip connections (1,2). 6. 6.

The use of cosine annealing (Loshchilov and Hutter, ) or a fixed learning rate.

Some of the reasoning behind choosing the search space is as follows:

Primary reason for choosing this range of filter and block values was to obtain networks that have a smaller number of parameters and FLOPs as the current state-of-the-art networks. The filters and block size dominate the parameter density in a network. We defined an upper bound based on this. 2. 2.

Motivated from Ma et al. (2018), choosing equal channel width minimizes memory access cost (MAC). Hence, we retained the same channel width across all the layers. 3. 3.

Furthermore, (Ma et al., 2018) computed the relative costs of using group convolution, so we used the depth-wise convolution in all the layers. 4. 4.

Moreover, for deployment in edge devices, various restrictions are placed based on the platform for inference. In this problem, the model size, the number of FLOPs and MAC play a key role in inference time. For instance, the largest model we have has around $7\times 10^{5}$ parameters and $5.2\times 10^{6}$ multiplication FLOPs. Deploying such a model would have severe consequences in the overall usability of the edge device 5. 5.

We chose a lower bound on the number of filters, around the region, based on the smallest state-of-the-art model, i.e. DMCNN (Syu et al., 2018). 6. 6.

We used point-wise convolutions to avoid the use of fully-connected layers as the last layer, and therefore reduce the model size.

Our goal in this study was to explore architectures that improve the state of the art models by using fewer number of parameters and maintaining reasonable accuracy for edge implementation.

4 Methodology and Theory

Neural architecture search is a discrete optimization problem. Here, we used an exhaustive search over a small space, where the convergence to the optimum in the designed space is trivial and independent of the loss function, because all points are evaluated. However, most of these neural architecture search techniques are motivated by a very large design space. We show that as the designed space gets infinitely large, even simple methods such as exhaustive search and grid search, require some minimal properties of the loss function to converge to the optimum. Exhaustive search and grid search techniques are common optimization methods that rely only on function evaluation. Therefore, it is counter-intuitive that finding the optimum point using these techniques may require some mathematical assumptions about the search space, or the objective function. Motivated from the derivative-free optimization theory, we aim at clarifying some minimal conditions that the exhaustive search and grid search converges to the optimum.

Let us set the notation first. We denote a vector using bold ${\boldsymbol{\theta}}$ and an element of the this vector using unbold $\theta$. The search space can be a mix of continuous or discrete parameters and denoted by $\Theta$. Suppose a neural network is uniquely defined by its hyperparameters embedded in ${\boldsymbol{\theta}}\in\Theta$. Note that the complexity of each network might differ. Complexity may be defined as the number of parameters of the network, inference delay, training time, etc. Often the objective is to discover the neural network with minimum loss (or maximum accuracy) on a fixed validation set. Let’s denote the validation loss of the neural network with ${\mathcal{L}}({\boldsymbol{\theta}})$ and its complexity ${\mathcal{C}}({\boldsymbol{\theta}})$. It is preferable to minimize ${\mathcal{L}}({\boldsymbol{\theta}})$ and ${\mathcal{C}}({\boldsymbol{\theta}})$ at the same time, i.e. between two neural networks with the same validation loss ${\mathcal{L}}(.)$, the neural network with smaller complexity ${\mathcal{C}}(.)$ is preferred. To simplify the theoretical analysis we assume that ${\mathcal{C}}(.)$ is fixed and focus on minimizing the validation loss function ${\mathcal{L}}(.)$ regardless of ${\mathcal{C}}(.)$. In the context of demosaicing ${\mathcal{L}}(.)$ is the negative CPSNR and ${\mathcal{C}}(.)$ is the number of model parameters.

Assume the minimum exists within the search space $\Theta$

[TABLE]

Let’s define the exhaustive search algorithm to minimize ${\mathcal{L}}(.)$ as

Enumerate: $\{{\boldsymbol{\theta}}_{0},{\boldsymbol{\theta}}_{1},\ldots,{\boldsymbol{\theta}}_{K}\}.$ 2. 2.

Initialize: $k=0,{\boldsymbol{\theta}}^{*}={\boldsymbol{\theta}}_{0},{\mathcal{L}}^{*}={\mathcal{L}}({\boldsymbol{\theta}}_{0}).$ 3. 3.

Evaluate ${\mathcal{L}}({\boldsymbol{\theta}}_{k+1})$. 4. 4.

If ${\mathcal{L}}({\boldsymbol{\theta}}_{k+1})<{\mathcal{L}}^{*}$, then update ${\boldsymbol{\theta}}^{*}={\boldsymbol{\theta}}_{k+1},{\mathcal{L}}^{*}={\mathcal{L}}({\boldsymbol{\theta}}_{k+1})$. 5. 5.

If $k<K$ then $k=k+1$ go to 3, otherwise end.

It is trivial that after iterating the algorithm for $K$ iterations, the global minimum of the objective function is obtained $({\boldsymbol{\theta}}^{*},{\mathcal{L}}({\boldsymbol{\theta}}^{*}))$, for simplicity of notation we may denote $({\boldsymbol{\theta}}^{*},{\mathcal{L}}^{*})$ where

[TABLE]

Convergence to the optimum requires no restriction on the loss function ${\mathcal{L}}(.)$ except the existence of the minimum (6). However, if the algorithm is iterated only $k<K$ iterations there is no guarantee of convergence to the minimum.

Often the cardinality $||\Theta||=K$ is a large value because in a real scenario a grid search on the hyperparameter vector ${\boldsymbol{\theta}}$ is implemented, where the search becomes NP-hard, $K={\mathcal{O}}\left(e^{d}\right)$ where $d=\dim({\boldsymbol{\theta}})$, and the algorithm is run only $k<K$ iterations depending on computation budget.

In the following sections we explore under what conditions the exhaustive search convergence is assured. Suppose the search space is large but is countable, $K=\infty$. Infinitely many evaluations requires more conditions on ${\mathcal{L}}(.)$. For simplicity of the notation we may assume $\Theta\subset{\rm I\!R}^{d}$ where $d=\dim({\boldsymbol{\theta}})$. Suppose the minimum argument and the objective value at iteration $k$ of exhaustive search is denoted by $({\boldsymbol{\theta}}^{*}_{k},{\mathcal{L}}^{*}_{k})$

Theorem 1

If $\Theta$ is dense and ${\mathcal{L}}(.)$ is continuous, the exhaustive search converges to the minimum value

[TABLE]

Proof 2

Take the optimal solution ${\boldsymbol{\theta}}^{*}\in\Theta$ and because ${\mathcal{L}}(.)$ is continuous and $\Theta$ is dense

[TABLE]

or equivalently ${\mathcal{L}}(\theta_{k})<{\mathcal{L}}({\boldsymbol{\theta}}^{*})+\epsilon$. On the other hand $\forall k,{\mathcal{L}}({\boldsymbol{\theta}}^{*})\leq{\mathcal{L}}({\boldsymbol{\theta}}^{*}_{k})\leq{\mathcal{L}}({\boldsymbol{\theta}}_{k})$ and by definition ${\mathcal{L}}^{*}={\mathcal{L}}({\boldsymbol{\theta}}^{*}),{\mathcal{L}}^{*}_{k}={\mathcal{L}}({\boldsymbol{\theta}}^{*}_{k}),$ so

[TABLE]

[TABLE]

Theorem 3

If $\Theta$ is compact and ${\boldsymbol{\theta}}^{*}=\operatorname*{argmin~{}}_{{\boldsymbol{\theta}}\in\Theta}{\mathcal{L}}({\boldsymbol{\theta}})$ is unique, the exhaustive algorithm converges to the unique minimum.

Proof 4

$\forall k,{\boldsymbol{\theta}}_{k}\in\Theta$* which is compact, so there is a sub-sequence that converges to ${\boldsymbol{\theta}}^{*}_{k}$, call it ${\boldsymbol{\theta}}^{*}_{ik}$*

[TABLE]

By continuity of ${\mathcal{L}}(.)$

[TABLE]

but from Theorem 1 we know $\lim_{k\to\infty}{\mathcal{L}}({\boldsymbol{\theta}}^{*}_{k})$, therefore $\tilde{{\boldsymbol{\theta}}}\in\Theta.$ From the uniqueness of the minimum $\tilde{\boldsymbol{\theta}}={\boldsymbol{\theta}}^{*}.$

The convergence of exhaustive search relies on the continuity of ${\mathcal{L}}(.).$ It is easy to construct cases where the exhaustive search fails to converge if ${\mathcal{L}}(.)$ is non-continuous (Audet and Hare, 2017). Although exhaustive search is the first tool to explore the discrete space, in a continuous space exhaustive search is in-feasible. Therefore optimizing over a continuous parameter such as learning rate, annealing rate, regularization constant, etc requires other tools. Bayesian optimization (Snoek et al., 2012) is often used and some open source packages are available (Tsirigotis et al., 2018). However, grid search is still one of the most used tool to tune a continuous parameter.

Grid search on multiple variables is cumbersome as a large dimension of the search variable $d$ implies the evaluation set being ${\mathcal{O}}(n^{d})$. Multivariate grid search algorithm over the set $\prod_{i=1}^{d}[a_{i},b_{i}]\subset{\rm I\!R}^{d}$ is as follows:

Initialize number of grid evaluation $n\in{\rm I\!N}$. 2. 2.

Set the step size $\delta_{i}={b_{i}-a_{i}\over n-1},i=1,\ldots,d.$ 3. 3.

Create the grid search space with step iteration $j$

[TABLE] 4. 4.

Evaluate the objective function ${\mathcal{L}}(.)$ for all $n^{d}$ grid points $\theta\in\Theta_{n}$. 5. 5.

Find the grid search solution $({\boldsymbol{\theta}}^{*}_{n},{\mathcal{L}}^{*}_{n})$ where

[TABLE]

Grid search on $d$ variables is NP-hard, so it is commonly implemented only on one element of the vector ${\boldsymbol{\theta}}$ while keeping the other elements fixed to certain values. Therefore, it makes more sense to explore the convergence of the grid search on a uni-variate variable $\theta$.

Suppose the continuous uni-variate parameter $\theta\in[a,b]$ achieves its minimum at $(\theta^{*},{\mathcal{L}}^{*}).$ The uni-variate parameter $\theta$ is a continuous element, such as the learning rate, and is also an element of the vector ${\boldsymbol{\theta}}$, in which the vector ${\boldsymbol{\theta}}$ fully determines the neural network.

The grid search convergence result relies on creating a nearly dense set $\Theta_{n}$ by choosing a large number of grid evaluation $n$. The convergence also depends on ${\mathcal{L}}(\theta)$ being Lipschitz continuous, i.e.

[TABLE]

where $M$ is the Lipschitz constant.

Theorem 5

Suppose ${\mathcal{L}}(.)$ is Lipschitz continuous on $\theta\in[a,b]$ and a grid search is run over $n$ grids to create $\Theta_{n}$ with step size $\delta$, then

[TABLE]

Proof 6

Suppose $\theta^{*}\in\Theta$, take ${\boldsymbol{\theta}}_{n}\in\Theta_{n}$ as a grid search candidate. For any point in the grid $\Theta_{n}$, define the closest point to $\theta^{*}$ in the grid space $\theta_{n}^{*}=\operatorname*{argmin~{}}_{\theta_{n}\in\Theta_{n}}|\theta_{n}-\theta^{*}|.$ The distance between two consecutive points is $\delta$, so $|\theta^{*}_{n}-\theta^{*}|\leq{\delta\over 2}.$ By the Lipschitz continuity of ${\mathcal{L}}(.)$

[TABLE]

[TABLE]

Theorem 5 means for a finite $n$, ${\mathcal{L}}^{*}_{n}$ may never reach the true minimum ${\mathcal{L}}^{*}$ as ${\mathcal{L}}^{*}\leq{\mathcal{L}}^{*}_{n}$, but improves swiftly as $\delta$ gets smaller by evaluating a smaller step size or equivalently by increasing the number of evaluations $n$; the lower-bound improves by ${\mathcal{O}}(n^{-1})$.

5 Application

We first describe the dataset that was used for our experiments and elaborate on the results obtained.

Flickr500 data is used by (Syu et al., 2018) to generate around 3.5 million patches. The authors choose the images with some criteria such as i) being colorful enough to obtain the proper color distributions of real world, ii) high-frequency patterns in the images and iii) of high-quality. However, there are some key points missing in the description to re-generate the exact results such as:

Specifications on exactly how the patches are generated from the original Flickr500 images (related to the stride etc while cropping the patches). 2. 2.

Specifications on how to apply the Bayer CFA for creating the mosaic image.

Therefore, some assumptions were made while generating our data.

We generated random patches of $32\times 32\times 3$ to obtain approximately 1.5 million image patches from the 500 images. Since the method is not specified, we used random points in the image and extracted patches from the random points. 1382400 patches were used for the training set and 153600 patches for the validation set. Note: In the paper, they use patches of size $33\times 33\times 3$ but we used $32\times 32\times 3$ because the training is much faster on GPU due the dimensions being a power of 2 for tiling. 2. 2.

The Bayer CFA generation is also based on the python package. We used the default setting of $RGGB$ and filled the missing channels with zeros to maintain the channel dimensions. 3. 3.

All the results in the section were calculated against the McMaster dataset (MCM) containing 18 photos (58000 patches created), and the Kodak images containing 24 photos (78408 patches created).

The Pareto front of our exhaustive search is shown in Figure 4. The $y$-axis represents the validation loss ${\mathcal{L}}(.)$ which indicates the negative CPSNR and the $x$-axis represents the model complexity ${\mathcal{C}}(.)$, which is the total number of parameters in the model. The standard error for each experiment is calculated and as they are comparable, they produce a similar prediction interval.

Most of the architectures in the search space outperform the current state-of-the-art (SOTA) as in Figure 4 in a similar setting, see also Table 1. The total number of parameters in the current SOTA is approximately $7\times 10^{5}$. Architectures on the Pareto front maintain the same accuracy while significantly reducing the number of parameters. In our search the effect of cosine annealing was not obvious and would need further tuning on cycle rate. The ResNet skip connections length of 2 blocks worked better than 1.

Some of the models on the Pareto front are shown in the table below in comparison to the current state-of-the-art models for the MCM dataset. Once the initial search was completed, the models from the Pareto front architectures were fed through a Bayesian optimizer using the ORION package (Tsirigotis et al., 2018), to optimize the learning rate and regularization constant for $L_{2}$ regularization. Figure 4 indicates further improvement over the initial CPSNR obtained with a fixed learning rate $10^{-4}$ and regularization constant $10^{-8}$, so Bayesian optimization is recommended as an extra step after the architecture search as including it in the first step would further increase the complexity of the search. In our experiments, the PSNR was calculated separately for each image in the test dataset; the final CPSNR value was the mean of all the PSNRs of all images in the dataset.

6 Conclusion

Our designed space with a simple exhaustive search outperforms the state of the art and brings a range of loss versus complexity for edge implementation with varying resource constraints. Although in most of architecture search, the number of evaluations and complex search algorithm implementation is the bottleneck, here we showed using prior vision domain knowledge, we can overcome these drawbacks and a simple exhaustive search becomes an effective search tool. We only focused on demosaicing as an example of a low-vision problem where edge implementation is an obligation. We wonder whether a similar approach is useful in edge implementation of other low vision problems. A simple theoretical study suggests the continuity of the loss function and compactness of the search space for exhaustive and grid search to converge to the optimum.

Acknowledgement

We want to acknowledge Jingwei Chen, Yanhui Geng, and Heng Liao; executing this research project was impossible without their continuous support. We appreciate the assistance of Huawei media research lab of Japan that provided us details about demosaicing and camera sensors. We declare our deep gratitude to Charles Audet and Sébastien Le Digabel who introduced to us the theory behind derivative-free optimization.