Supervised Deep Kriging for Single-Image Super-Resolution

TL;DR

This paper introduces a supervised deep learning approach to single-image super-resolution based on kriging, enabling end-to-end training and statistical bias-variance estimation, achieving competitive results.

Contribution

It combines kriging with deep learning to learn weights end-to-end, providing a novel statistical framework for super-resolution.

Findings

Achieves competitive super-resolution performance.

Enables bias and variance estimation in deep networks.

Introduces a joint network for kriging-based super-resolution.

Abstract

We propose a novel single-image super-resolution approach based on the geostatistical method of kriging. Kriging is a zero-bias minimum-variance estimator that performs spatial interpolation based on a weighted average of known observations. Rather than solving for the kriging weights via the traditional method of inverting covariance matrices, we propose a supervised form in which we learn a deep network to generate said weights. We combine the kriging weight generation and kriging process into a joint network that can be learned end-to-end. Our network achieves competitive super-resolution results as other state-of-the-art methods. In addition, since the super-resolution process follows a known statistical framework, we are able to estimate bias and variance, something which is rarely possible for other deep networks.

Tables

Table 1. Table 1: Results of various super resolution techniques. The column 3th and 4th are unsupervised. The other columns are trained on the 291 data set  [ 28 , 29 ] . The Urban 100 set having no images at scale 3 we did not evaluate it.

Dataset	Scale	Bicubic	local kringing	SRCNN	VDSR	DRCN	DRRN B1U9	DRRN B1U25	ours
		PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM
Set 5	$\times 2$	33.66/0.930	35.68/0.923	36.66/0.953	37.53/0.959	37.63/0.959	37.66/0.959	37.74/0.959	37.65/0.960
	$\times 3$	30.39/0.868	31.72/0.864	32.75/0.909	33.66/0.921	33.82/0.923	33.93/0.923	34.04/0.924	33.94/0.923
	$\times 4$	28.42/0.810	29.83/0.814	30.48/0.862	31.35/0.883	31.53/0.885	31.58/0.886	31.68/0.889	31.56/0.886
Set 14	$\times 2$	30.24/0.869	31.26/0.874	32.45/0.907	33.03/0.912	33.04/0.912	33.19/0.913	33.23/0.914	33.10/0.912
	$\times 3$	27.55/0.774	27.94/0.781	29.30/0.822	29.77/0.831	29.76/0.831	29.94/0.831	29.96/0.834	29.81/0.831
	$\times 4$	26.00/0.703	26.43/0.715	27.50/0.751	28.01/0.767	28.02/0.767	28.02/0.767	28.18/0.770	28.08/0.771
B 100	$\times 2$	29.56/0.843	31.06/0.848	31.36/0.888	31.90/0.8960	31.85/0.894	32.01/0.897	32.05/0.897	32.01/0.896
	$\times 3$	27.21/0.8431	27.86/0.7357	31.36/0.8879	28.82/0.7976	28.80/0.7963	28.91/0.7992	28.95/0.800	28.82/0.800
	$\times 4$	25.96/0.668	26.55/0.665	26.90/0.711	27.29/0.725	27.23/0.723	27.35/0.726	27.38/0.728	27.36/0.727
Urban 100	$\times 2$	26.88/0.840	28.32/0.843	29.50/0.895	30.76/0.914	30.75/0.913	31.02/0.916	31.23/0.919	30.95/0.921
	$\times 3$	24.46/0.735	25.06/0.715	29.50/0.799	27.14/0.715	27.15/0.828	27.38/0.833	27.53/0.838
	$\times 4$	23.14/0.658	23.70/0.627	24.52/0.722	25.19/0.752	25.14/0.751	25.35/0.757	25.44/0.764	25.10/0.749

Equations

$${\hat{f}}_{\ast }=\sum _{i=1}^n{\omega }_i(x^{\ast })f_i,\text{with }\sum _{i=1}^n{\omega }_i(x^{\ast })=1.$$

$$\begin{gathered} R(\omega )=\mathbb{E}\left[(F_{\ast }-{\hat{F}}_{\ast }{)}^2\right] \\ =\text{mbox}var\left(F_{\ast }-{\hat{F}}_{\ast }\right). \end{gathered}$$

$$\mathcal{L}(w)=\text{var}\left(F_{\ast }-{\hat{F}}_{\ast }\right)+2\lambda \left(\sum _{i=1}^n{w}_i(x^{\ast })-1\right),$$

$$\left(\begin{array}{cccc}{C}_{11}& \dots & C_{1n}& 1\\ ⋮& \ddots & ⋮& ⋮\\ C_{1n}& \dots & C_{nn}& 1\\ 1& \dots & 1& 0\end{array}\right)\left(\begin{array}{c}{w}_1(x^{\ast })\\ ⋮\\ w_n(x^{\ast })\\ -\lambda \end{array}\right)=\left(\begin{array}{c}{C}_{1\ast }\\ ⋮\\ C_{n\ast }\\ 1\end{array}\right).$$

$${\hat{f}}_{\ast }^i=\sum _{k\in \{\Vert x_k-x^{\ast }{\Vert }_1\le \mathcal{K}\}}{\omega }_k(x^{\ast }){\tilde{f}}_k^i\text{with}\sum _{k\in \{\Vert x_k-x^{\ast }{\Vert }_1\le \mathcal{K}\}}{\omega }_k(x^{\ast })=1.$$

$$R(\omega )=\sum _{i=1}^{n_1}{\mathbb{E}}_{F^i\times {\tilde{F}}^i}\left[{\left(F_{\ast }^i-{\hat{F}}^i\right)}^2\right].$$

$$R(\omega )=\sum _{i=1}^{n_1}\sum _{j=1}^{N_i}\left[{\left(f_{\ast }^i-\sum _{k=1}^n{\omega }_k(x^{\ast }){\tilde{f}}_k^i\right)}^2\right],$$

$$\text{bias}(x)$$

$$\displaystyle=\mathbb{E}_{\tilde{F}|\omega_{1},\ldots,\omega_{n}}\big{[}\tilde{F}(x)\big{]}-\mathbb{E}_{\tilde{F}|\omega_{1},\ldots,\omega_{n}}\big{[}\hat{F}(x)\big{]}$$

$$={\mathbb{E}}_{\tilde{F}|{\omega }_1,\dots ,{\omega }_n}\left[\tilde{F}(x)\right]-\sum _{k_1=1}^n{\omega }_{k_1}(x_j){\mathbb{E}}_{\tilde{F}|{\omega }_1,\dots ,{\omega }_n}\left[\tilde{F}(x)\right].$$

$$V_{\tilde{F}}(x)={\mathbb{E}}_{\tilde{F}}\left[\hat{F}(x{)}^2\right]-{\left({\mathbb{E}}_{\tilde{F}}[\hat{F}(x)]\right)}^2.$$

$${\mathbb{E}}_{\tilde{F}}\left[\tilde{F}(x_k)\tilde{F}(x_{k^{\prime }})\right]=\tilde{C}(\Vert x_k-x_{k^{\prime }}{\Vert }_2)-{\mu }^2.$$

$$V_{\tilde{F}|{\omega }_1,\dots ,{\omega }_n}(x)=\sum _{\begin{array}{c}(k,k^{\prime })\in [1,(2\times \mathcal{K}+1{)}^2{]}^2\end{array}}{\omega }_k(x){\omega }_{k^{\prime }}(x)\tilde{C}(\Vert x_k-x_{k^{\prime }}{\Vert }_2)$$

$${\tilde{f}}_i(x^{\ast })=\sum _{k\in \{\Vert x_k-x^{\ast }{\Vert }_1\le \mathcal{K}\}}{\omega }_k(x^{\ast })\left({\tilde{f}}_i\ast h_k\right)(x_k)\text{ with }\sum _{k\in \{\Vert x_k-x^{\ast }{\Vert }_1\le \mathcal{K}\}}{\omega }_k(x^{\ast })=1$$

Full text

K****eywords Single image super resolution  $\cdot$ kriging  $\cdot$ deep learning

1 Introduction

Super-resolution aims to transform low resolution (LR) images into images with high resolution (HR). In computer vision, super-resolution is relevant for applications where high-frequency information and detailing is desirable yet not always captured, *e.g. *medical imaging, satellite imaging, surveillance, etc. Our interest is in single image super-resolution (SISR), a special case where only one LR image is available. SISR is an ill-posed inverse problem with multiple solutions, *i.e. *multiple HR images could lead to the same LR image after applying a low-resolution filter.

SISR been solved with many different approaches. Early approaches relied on spatial relationships present in the image and include nearest neighbours [1], bi-cubic interpolation [2]. Other techniques build upon the assumption that the underlying signal is sparse, *e.g. *compressive sensing [3] and dictionary learning [4]. These early approaches tend to yield HR images which are blurry and or noisy, due mainly to the fact that these methods are crafted heuristically. As a result, they cannot sufficiently recover enough information to yield sharp photo-realistic images.

Deep learning, and more specifically, convolutional neural networks (CNNs) [5], has afforded significant performance gains in SISR. The newest frameworks  [6, 7, 8, 9, 10] learn feature representations for super-resolution in a supervised end-to-end manner and are achieving ever-improving performance both in terms of speed and quantitative evaluation. However, despite their impressive results, most deep networks are being used as black boxes to make point estimates. The regressed outputs represent only a mean value and give no indication of model uncertainty. In the case of SISR, this means that we make (possibly very accurate) guesses for the unknown pixels, but we have no idea how good these guesses are. This can be highly problematic for applications such as microscopy or medical imaging, where model confidence is just as important as the actual output.

We would like to leverage the strong learning capabilities of deep networks and embed them within a known statistical framework for which we can derive model uncertainty. As such, we propose deep kriging to solve SISR. Kriging [11, 12] is a geostatistics method used for spatial interpolation of attributes such as topography and natural resources. It has close relations to Gaussian process (GP) regression, although some sources oversimplify and describe the two as being equivalent. The main difference is that kriging makes stationary and ergodic assumptions on the random field representing the data, while GP regression considers it as a Gaussian process.111We refer the reader to our supplementary materials for a more detailed comparison of the two.

Kriging interpolates unknown values by taking a weighted average of known values and results in an unbiased estimator with minimum variance. One can solve for the weights by inverting a covariance matrix computed on the known instances. Kriging makes for a natural extension to the application of SISR, where HR images are interpolated from pixels observed in the LR version of the image. The computational cost of the kriging is cubic with respect to the number of known instances; hence it is preferable to work locally. However, interpolated results are often overly smooth and depend on the choice of the covariance function.

Our contributions can be summarized as follows:

• •

We propose a novel deep learning-based statistical estimator with SISR results competitive with state-of-the-art.

• •

We propose a deep kriging approach that solves for kriging weights and performs the kriging in a single network; this allows us to perform a supervised form of spatial interpolation which can be applied not only to SISR, but other geostatistical computations such as field reconstructions.

• •

Our proposed deep kriging is a hybrid deep learning and statistical framework for which we can derive statistical bias and variance; such measures of model performance and uncertainty are not commonly available for other deep networks, making us the first to model and compute pixel uncertainty in deep SISR approaches.

2 Related work

Prior to the use of deep learning, SISR approaches applied variants of dictionary learning [13, 3, 4, 14, 15]. Patches were extracted from the low-resolution images and mapped to their corresponding high-resolution version which are then stitched together to increase the image resolution. Other learning-based approaches to increase image resolution include  [16, 17, 18].

State-of-the-art SISR methods are deep-learning-based [8, 6, 7, 10, 9]. The VDSR [6] and DRCN [7] approaches showed the benefits of working with image residuals for super-resolution. The DRRN approach [9] generalizes VDSR and concludes that the deeper the network, the better the super-resolved image. We also use a residual network in our approach, but unlike all the other deep SISR methods, we are solving for a set of filter weights to perform the super-resolution with our network rather than directly estimating the HR image itself.

Several unsupervised techniques have also been developed for SISR [19, 20], though their performance is usually poorer than supervised approaches. One work of note [21], uses GP regression to perform super-resolution and resembles ours in spirit in that we both model pixel intensities as a random process regressed from neighbouring pixels. However, [21] is unsupervised while our approach learn the weights from supervise examples.

Our proposed approach can be thought of as a form of local filtering [22], and is most similar in spirit to works which combine deep learning and GP regression [23, 24]. However, we differ from these techniques since we do not apply GP regression but a modified version of local kriging. These techniques are similar to us in that they also consider the data to follow a random process. However, we do not explicitly learn the covariance matrix, which offers us more flexibility in the relationships we wish to express. Furthermore, we treat each image as following a random process; the set of training images is then a set of random processes, while the aforementioned works consider all the training data to follow the same process.

3 Deep Kriging

We begin with a short overview on classical kriging in Section 3.1 before we introduce our proposed method of supervised kriging in Section 3.2 and elaborate on its statistical properties in Section 3.3. Finally, we show how the proposed form of deep kriging can be implemented in a deep network in Section 3.4.

3.1 Classical kriging

Consider an image $f$ as a realization of a random field $F$. This random field is a collection of random variables, *i.e. *$F=\{F_{i}\}$ where each $F_{i}$ is a random variable and $i$ is a position index. Unknown values on $f$ at position $x^{*}$, $\hat{f}_{*}$222For concise notation, we use $\hat{f}_{*}$ to denote $\widehat{f}(x^{*})$, $f_{i}$ to denote $f(x_{i})$ and $w_{i}^{*}=w_{i}(x^{*})$., can be interpolated linearly from $n$ known realizations $f_{i}$, with normalized weights $w_{i}$, i.e.

[TABLE]

In classical kriging, all known realizations are used in the interpolation and $n$ is the number of pixels in the image, though local variants have been proposed and studied in [25, 26].

The weights $\{\omega_{i}\}$ are found by minimizing the true risk:

[TABLE]

We can equate the $\mathbf{E}[\cdot]$ with the $\text{var}(\cdot)$ term in Eq. 2 because the constraint that the weights must sum up to $1$ implies that $\mathbb{E}[f_{*}-\hat{f}_{*}]\!\!=\!\!0$. However, since we do not have access to different realizations of the random field, this variance cannot be solved directly.

To infer the variance from a single event, the theory of geostatistics replaces the classical statistics assumption of having independent and identically distributed (iid) random variables with assumptions of stationarity and ergodicity on the random field. The random field is assumed to be first and second order stationary. This means that $\mathbb{E}_{F^{i}}(f^{i}(x))$ does not depend on $x$ and that $\mathbb{E}_{F^{i}}(f^{i}(x)\times f^{i}(x+\tau))$ depends only on $\tau$. In addition, the field is assumed to be second order ergodic, so a covariance or mean estimate in the probability domain is equivalent inthe spatial domain. This second assumption implies that an empirical covariance can be estimated and depends only on a distance $\tau$ between two realizations, *i.e. *$\tau=\|x_{i}-x_{j}\|$. However, it is difficult to work directly with the empirical covariance, since it can be noisy and also is not guaranteed to form a positive semi-definite matrix. As such, in kriging, we use the true covariance by fitting a parametric model. While different models can be used, the most common one is Gaussian, with covariance $C_{ij}$ between points $x_{i}$ and $x_{j}$ defined as $C_{ij}=C_{0}\exp{\left(-\tfrac{1}{\sigma^{2}}\|x_{i}-x_{j}\|^{2}\right)},$ where $C_{0}$ and $\sigma$ are parameters of the model.

A Lagrange multiplier (with constant $\lambda$) can be used to minimize the risk function with the constraints on the weights $w_{i}$, leading to the following cost function:

[TABLE]

and the associated solution expressed in matrix form as:

[TABLE]

The process of kriging then is reduced to solving for the weights $w_{i}$. This involves inverting the covariance matrix on the left. Since the matrix is of dimension $(n+1)$, where $n$ is the number of observed pixels in the image (patch), it can be very computationally expensive. Even though one can limit $n$ with local kriging, $n$ needs to be sufficiently large in order to accurately capture and represent the spatial relationships. We aim to bypass the matrix inversion and still maintain sufficient generalization power by using a deep network to directly learn the weights instead.

3.2 Supervised kriging

Consider $E$, a subset of the discrete space $\mathbb{Z}^{2}$, as a representation of the support space of a 2D image. We denote an image as $f\in\mathbb{R}$ and $f_{i}$ as its value at pixel $x_{i}\in E$. This assumes we work with greyscale images, which is a standard assumption made in most SISR algorithms [7, 6, 9].

We assume that we are given a set of $n_{1}$ training image pairs $\{(\tilde{f}^{i},f^{i})\},i\in[1,n_{1}]$, where $\tilde{f}^{i}$ is an up-sampled low resolution version of $f^{i}$ with the same size. Our objective is to super-resolve the low-resolution images of the test set $\mathcal{N}_{\mbox{test}}=\{\tilde{f}^{i}\}_{i=1}^{n_{2}}$. We further assume that $\tilde{f}^{i}$ is a realization of a first and second moment stationary random process $\tilde{F}^{i}$. For convenience we denote $\tilde{F}^{i}$ the distribution of the random process. In addition $f$ is a realization of a random field $F$. We denote the training set as $\{\tilde{f}^{i},f^{i}\}_{i=1}^{n_{1}}\sim\mathcal{M}$, where $\mathcal{M}$ is a joint meta-distribution where each training pair $(\tilde{f}^{i},f^{i})\sim\tilde{F^{i}}\times F^{i}$ follows its own random process.

In classical kriging, the estimated $\hat{f}(x)$ is expressed as a linear combination of all the observed samples *i.e. *pixels of $f$. In the case of super-resolution, the observations come from the low-resolution image $\tilde{f}(x)$. Furthermore, we estimate $\hat{f}(x)$ as a linear combination of only local observations of some window radius $\mathcal{K}$, leading to the estimator:

[TABLE]

When no confusion is possible we denote $\hat{F}_{*}^{i}=\sum_{k_{1}=1}^{n}\omega_{k}(x^{*})\tilde{F}_{k}^{i}$, with $n=(2\mathcal{K}+1)^{2}$. We found that a window of $7\times 7$ provided the best results. To find $\omega$, we want to minimize the true risk as given in Eq. 2, leading to:

[TABLE]

This risk is a compromise between the geostatistical model and the deep learning model, where the covariance of each field is learned through supervised learning. Furthermore, if we replace the true expectation with the empirical one and the random field with its realization, we arrive at

[TABLE]

where $N_{i}$ is the number of pixels in image $f^{i}$.

Now, rather than solving for the weights $\omega$ by inverting the covariance matrix in Eq. 4, we propose learning this set of weights directly with a CNN. This allow us to capture and model more complex spatial relationships between the data than the classical model with covariances. As such, the estimator becomes $\hat{f}(x,\tilde{f},g,\theta)$, where $w=g(\tilde{f},\theta)$, with $g$ and $\theta$ representing the CNN network function and its parameters respectively.

3.3 Statistical properties

We can derive statistical properties for our proposed estimator $\hat{F}$. Given $\{\omega_{1},\ldots,\omega_{n}\}$, the bias can be expressed as

[TABLE]

Since $\sum_{k_{1}=1}^{n}\omega_{k_{1}}(x_{j})=1$ and the random field is first-order stationary, we are left with zero bias according to the weights of the neural network, *i.e. *$\text{bias}(x)=0$. In other words, our network does not add bias to the field $\tilde{F}$. As such, from $F$ to $\tilde{F}$, if the two fields have the same first moment, then no bias added to $F$333Of course, we cannot account for the low resolution process that produced $\tilde{F}$. This point is critical since in classical statistics a good estimator should have zero bias and minimal variance.

By definition, the variance of our estimator $\hat{F}$ is

[TABLE]

Again, we estimate the covariance given $\{\omega_{1},\ldots,\omega_{n}\}$. Since we assume that $\tilde{F}$ is second-order stationary, the covariance of $\tilde{F}$ depends only on the distance between two points, i.e.

[TABLE]

By setting $\mu=\mathbb{E}_{\tilde{F}}[\tilde{F}(x)]$, we arrive at:

[TABLE]

where the covariance $\tilde{C}(\cdot)$ is estimated from the low resolution image $\tilde{f}$. The variance depends on the position $x$. Note, however, since we directly estimate the weights $\omega(x^{*})$ without making assumptions on the covariance function, we cannot compute the variance and an approximation is needed to estimate the covariance empirically from $\tilde{f}$. Eq. 10 however is particularly interesting since it shows that the estimator variance at $x$ is directly proportional to the variance of $\tilde{f}$ and the different values of $\omega$ near $x$. The bigger these values are, the more uncertain the estimator is.

3.4 Network Implementation and Training

So far, the theory which we have presented on supervised kriging is generic and does not indicate how one should minimize the risk in Eq. 7 to solve for the kriging weights. We solve for the weights with a CNN network which we show in Fig. 1. The weight estimation branch is composed of a residual network of 9 residual units; the residual unit itself has 2 convolutional layers, each preceded by a batch normalization [27] and a ReLU.

This branch follows a similar architecture as [9, 6], with the difference that they apply this architecture to directly estimate the HR image, while we use it to estimate our kriging weights, *i.e. *as a local dynamic filter [22]. In addition, contrary to [9] our network is not recurrent.

The network learns a total of 20 convolution layers to determine the kriging weights. The first 19 are of depth 128, the last prediction layer has depth $(2\times\mathcal{K}+1)^{2}$ and outputs the kriging weight vector $\omega(x)=[\omega_{1}(x),\ldots,\omega_{(2\times\mathcal{K}+1)^{2}}(x)]$. The weights are applied to the repeated input image via a point-wise multiplication and summation along the depth dimension to yield the HR output. Overall, the network with $\mathcal{K}=3$ is very lightweight, inference on a $320\times 480$ sized image takes only $0.10$ seconds with a titan X.

Note our network is actually learning how to estimate the kriging weights $\omega$, *i.e. *as a local dynamic filter. To derive this dynamic filter end-to-end, we adapt the following formulation of the Eq. 5 in convolution terms:

[TABLE]

with $h_{k}(x)=\delta(x-x_{k})$ or the Dirac function. The $h_{k}$ filter is denoted by “repeat input” in Fig. 1, since applying the Dirac functions in each position $x^{*}$ is a way to extract the pixel neighbourhood at position $x^{*}$. After having determined the weights, we normalize them so that they sum up to $1$, as given by the constraint in Eq. 11. We point-wise multiply the normalized weights to the output of the repeat filter; to arrive at the super-resolved image, we simply sum along the depth.

We implement our network in Tensorflow, minimizing the loss in Eq. 7 with the Adam optimizer, a batch size of 8 and a learning rate of $10^{-4}$. We apply gradient clipping and dropout between the last ReLU and the convolution layer that predicts $\omega$, with a dropout rate of 0.3. The effective depth of our network is $d=2+2\times U$, with $U$ the number of times the residual units are applied. In our case, we use $U=9$.

4 Experiments

4.1 Datasets, Pre-processing and Evaluation

For training, we follow [9, 6] and use the 291 images combined from [28] and the Berkeley Segmentation datasets [29] . We further augment the data by rotating ($90^{\circ}$, $180^{\circ}$ and $270^{\circ}$) and scaling the data ($\times 2$, $\times 3$, $\times 4$). We limit to these fixed grades of rotation and scaling to follow the same protocol as  [9]. Like [9, 6], we work with patches of size $31\times 31$, sampled from the augmented dataset with a stride of $21$. Testing is then performed on the four commonly used benchmarks: Set 5 [30], Set 14 [7], B100 [29] and Urban 100 [31]. To be consistent with [9, 6], we convert the RGB images to YCbCr and only resolved the Y component. This is then combined with an upsampled Cb and Cr component and converted back into RGB. Up-sampling is done via bi-cubic interpolation.

We evaluate the resulting super-resolved images quantitatively with the peak signal-to-noise ratio (PSNR): $\text{PSNR}(\tilde{f},f_{\text{ref}})=\log_{10}\left({\frac{255^{2}}{\sum_{i=1}^{N}(\tilde{f}(x_{i})-f_{\text{ref}}(x_{i}))^{2}}}\right),$ where $f_{\text{ref}}$ and $\tilde{f}$ are the ground truth and the super-resolved images respectively and $N$ refers to the number of pixels in the (high-resolution) image. A higher PSNR corresponds to better results. We also evaluate using Structural Similarity (SSIM) [32], which measures the similarity between two images. The closer the SSIM value is to $1$, the more similar the super-resolved image to the ground truth high resolution image.

4.2 Comparison to state-of-the-art

We compare our PSNR and SSIM measures for the different scales against state-of-the-art in Table 1. The first two methods, bi-cubic and local kriging are unsupervised, while all others are supervised approaches. We use the Matlab implementation of bicubic interpolation.

For the local kriging, we use a neighbourhood of $90\times 90$ and a stride of $81$. At this sparse setting, unsupervised kriging does very well in comparison to bicubic interpolation. However, it is already extremely slow, since for each patch, we need to compute the empirical covariance, true covariance, and then invert the covariance matrix. In total, it takes approximately 1 minute for an image of size $320\times 480$. In comparison, our proposed deep kriging, on the same image, applied densely is one magnitude faster, and takes only $0.25$ seconds with a titan x.

Looking at our approach (reported in the second last column in Table 1) with respect to the supervised methods, our performance is comparable to the DRRN [9] B1U9 setting, DRCN [7] and VSDR [6]; all three have networks with depth similar to ours. The best results are reported by the DRRN B1U25 [9], which has 50 convolution layers and is more than twice as deep as our network.

Given the trend of the community to work on deep learning-based approaches for super-resolution, as well as the fact that no labelled data is required for training, one can work with (infintely) large datasets [33, 8]. For fair comparison with state-of-the-art, however, we omit from this table the techniques which do not use the fixed 291 image training set as per the training protocol set by [9, 6].

4.3 Model uncertainty

One of the key strengths of our technique is the fact that we can estimate model uncertainty. We show in Figure 2 the estimated variance for each pixel based on Equation 10. To evaluate this equation, we need to apply covariance models for $f$ and $\tilde{f}$. We do this by first estimating the empirical covariance $c(\tau)=\sum_{i}f(x_{i})f(x_{i}+\tau)-\mu$ and then solving for $C_{0}$ and $\sigma$ of the Gaussian model in section 3.1 that is the closest to the empirical covariance.

The main advantage of our model of uncertainty is that we can have information on the black box CNN. It gives us feedback about the reliability of results provided by an unknown image. One can see in Figure  2 our uncertainty estimate. The estimated variance has a higher value than the real PSNR this is due to the fact that we have a noisy estimation of the PSNR from a low resolution image. Holistically, however, the are similar, in that areas with high variance corresponds to high PSNR. Quantitatively, we find for Set 5 that 91.9% of the super-resolved pixel values fall within 3 standard deviations based on the estimated variance. In addition, we evaluate the similarity between the images resulting from the PSNR and the one resulting from the variance in set 5. We use the correlation to measure the similarity and find that these images have high correlation up to 0.8.

5 Conclusions

In this paper, we have proposed a joint deep learning and statistical framework for single image super-resolution based on a form of supervised kriging. We solve for the kriging weights in the manner of a local dynamic filter and apply it directly to the low resolution image, all within a single network that can be learned end-to-end. Since we work within the known statistical framework of kriging, we can estimate model uncertainty, something typically not possible for deep networks. More specifically, we show through derivations that the statistical estimator generated by our network is unbiased and we calculate its variance.

Acknowledgement

This research was funded by the German Research Foun- dation (DFG) as part of the research training group GRK 1564 Imaging New Modalities.