Light Weight Color Image Warping with Inter-Channel Information

TL;DR

This paper introduces a low-cost color image warping method that leverages inter-channel correlations to improve quality, suitable for real-time mobile multimedia applications.

Contribution

The paper proposes a novel warping scheme that combines intra-channel interpolation with cross-channel information, enhancing quality with minimal computational overhead.

Findings

Significant improvement in warping quality using inter-channel correlation.

Method achieves high efficiency suitable for mobile devices.

Validated through extensive experiments.

Abstract

Image warping is a necessary step in many multimedia applications such as texture mapping, image-based rendering, panorama stitching, image resizing and optical flow computation etc. Traditionally, color image warping interpolation is performed in each color channel independently. In this paper, we show that the warping quality can be significantly enhanced by exploiting the cross-channel correlation. We design a warping scheme that integrates intra-channel interpolation with cross-channel variation at very low computational cost, which is required for interactive multimedia applications on mobile devices. The effectiveness and efficiency of our method are validated by extensive experiments.

Figures

Tables

Table 1. Table 1 : The average PSNR before and after refining bilinear and bicubic upsampling by sequential HQLI demosaicking, tested on benchmark datasets.

Dataset	Bilinear	Bilinear +HQLI	Bicubic	Bicubic +HQLI
Set5	28.86	30.43	28.64	29.73
Set14	26.55	27.50	26.23	26.85
BSD100	26.44	26.89	26.02	26.16
Urban100	23.57	24.12	23.17	23.45

Table 2. Table 2 : The integration weights learned by linear regression for S = 4 𝑆 4 S=4 and various interpolation functions f 𝑓 f .

	Bilinear	Bicubic	Lanczos
${\omega }_{g,r}$	0.094	0.045	0.032
${\omega }_{g,b}$	0.119	0.064	0.041
${\omega }_{r,g}$	0.195	0.096	0.058
${\omega }_{r,b}$	0.008	0.010	0.015
${\omega }_{b,g}$	0.180	0.089	0.054
${\omega }_{b,r}$	-0.003	0.003	0.008

Table 3. Table 3 : PSNR, SSIM and computation time of plain intra-channel warping and the proposed color warping, averaged over Set5, for various scaling factor S 𝑆 S and interpolation function f 𝑓 f .

$f$	$S$	Independent			Correlated
		PSNR	SSIM	Time	PSNR	SSIM	Time
Bilinear	$\times 2$	30.42	0.9566	0.0013	32.00	0.9673	0.0062
	$\times 3$	27.78	0.9274	0.0012	28.55	0.9368	0.0052
	$\times 4$	25.84	0.8966	0.0011	26.82	0.9096	0.0056
Bicubic	$\times 2$	31.81	0.9657	0.0020	32.37	0.9694	0.0075
	$\times 3$	28.63	0.9365	0.0019	29.07	0.9413	0.0065
	$\times 4$	26.70	0.9087	0.0018	27.13	0.9139	0.0068
Lanczos	$\times 2$	32.43	0.9687	0.0020	32.74	0.9706	0.0074
	$\times 3$	29.05	0.9402	0.0018	29.29	0.9427	0.0063
	$\times 4$	27.07	0.9129	0.0017	27.33	0.9157	0.0067

Table 4. Table 4 : Comparison of PSNR and computation time of GR [ 23 ] the proposed color warping (with the interpolation function fixed to be Lanczos), averaged over four benchmark datasets, for various scaling factor S 𝑆 S .

Dataset	$S$	GR		Ours
		PSNR	Time	PSNR	Time
Set5	$\times 2$	33.04	0.34	32.74	0.0074
	$\times 3$	29.50	0.22	29.29	0.0063
	$\times 4$	27.49	0.19	27.33	0.0067
Set14	$\times 2$	29.85	0.67	29.62	0.0138
	$\times 3$	26.92	0.46	26.75	0.0133
	$\times 4$	25.21	0.38	25.08	0.0124
BSD100	$\times 2$	28.90	0.45	28.97	0.0073
	$\times 3$	26.38	0.32	26.30	0.0070
	$\times 4$	25.03	0.26	24.97	0.0070
Urban100	$\times 2$	26.30	2.31	26.20	0.0480
	$\times 3$	23.69	1.46	23.50	0.0427
	$\times 4$	22.28	1.24	22.08	0.0428

Equations

$$\displaystyle\left[\begin{array}[]{c}x\\ y\end{array}\right]=\mathbf{H}^{-1}\left[\begin{array}[]{c}i\\ j\end{array}\right].$$

$$\begin{array}{rl}{\hat{I}}_1[x,y]& =(1-s)(1-t)I_1[m,n]+st{I}_1[m+1,n+1]\\ & +s(1-t)I_1[m+1,n]+(1-s)t{I}_1[m,n+1].\end{array}$$

$$G[m,n]\approx \bar{G}+\alpha \Delta R[m,n],$$

$$\begin{array}{rl}\Delta R[m,n]=R[m,n]-\frac{1}{4}& (R[m+2,n]+R[m-2,n]\\ & +R[m,n-2]+R[m,n+2]).\end{array}$$

$$G_1[x,y]$$

$$R_1[x,y]$$

$$B_1[x,y]$$

$$\displaystyle\mathbf{H}=\left[\begin{array}[]{cc}S&0\\ 0&S\end{array}\right]\quad\mathbf{H}^{-1}=\left[\begin{array}[]{cc}\frac{1}{S}&0\\ 0&\frac{1}{S}\end{array}\right],$$

$$\displaystyle\begin{split}&\left[\begin{array}[]{cc}\Delta R_{1}[x_{1},y_{1}]&\Delta B_{1}[x_{1},y_{1}]\\ \vdots&\vdots\\ \Delta R_{1}[x_{K},y_{K}]&\Delta B_{1}[x_{K},y_{K}]\end{array}\right]\left[\begin{array}[]{c}\omega_{g,r}\\ \omega_{g,b}\end{array}\right]\\ &=\left[\begin{array}[]{c}\tilde{G}_{1}[x_{1},y_{1}]-G_{2}[i_{1},j_{1}]\\ \vdots\\ \tilde{G}_{1}[x_{K},y_{K}]-G_{2}[i_{K},j_{K}]\end{array}\right]\end{split}$$

Full text

**Index Terms— ** Image warping, inter-channel correlation, Laplacian filtering, image enhancement

1 Introduction

Image warping is fundamental for a variety of multimedia applications such as image resizing (e.g, [1]), texture mapping (e.g., [2]), image-based rendering (e.g., [3]), panorama stitching (e.g., [4]), stereo reconstruction (e.g., [5]), optical flow computation (e.g., [6]), to name but a few. Image warping can be briefly described as transforming a source image $I_{1}$ into a target image $I_{2}$ under a geometric mapping $\mathbf{H}$, which can be linear, affine, perspective, non-parametric etc.[7]. Particularly, assuming $I_{2}$ has $M$ rows and $N$ columns, each pixel $(i,j)$ in the target image lattice $[1\quad M]\times[1\quad N]$ has a correspondence image point $[x,y]=\mathbf{H}^{-1}([i,j])$ in the source image, from which the intensity value $I_{1}[x,y]$ is assigned to $I_{2}[i,j]$. The central problem of image warping is to estimate $I_{1}[x,y]$ from pixels in the vicinity of $[x,y]$, which are generally not on the integer grid, and hence the true intensity $I_{1}[x,y]$ is not immediately available.

The most popular estimation techniques for image warping in real practice are nearest neighbour, bilinear and bicubic (spline) interpolation schemes [8], which do not adapt well to irregular image regions. Therefore, academic research effort has been mainly dedicated to edge guided interpolation, such as anisotropic filtering (e.g., [9], [10]) and variational regularization (e.g., [11]). Particularly on the sub-topic of Signal Image Super-resolution, i.e., in the special case that $\mathbf{H}$ degenerates to a scaling function, mathematical models such as compressive sensing (e.g., [12]), dictionary learning (e.g., [13]) and Convolutional Neural Networks (CNN, e.g., [1]) have also been intensively investigated.

Although numerous image warping schemes have been proposed, they are mostly designed for monochrome images. This means that on warping color images, these methods should be applied to each channel independently (e.g., [14]) or to the luminance channel only (e.g.[15]), without exploiting the cross-channel correlation. The only exception is end-to-end CNNs that take RGB images as input. In this situation, the color correlation is captured during the training process, but the learned correlation is CNN specific rather than being general. In contrast, cross-channel correlation is thoroughly studied for Image Demosaicking, which recovers the full RGB images from subsampled color channels, with only one primary color component available at each pixel [16]. Although each channel can be recovered by merely using the intra-channel interpolation, it is commonly known that inter-channel correlation information can significantly improve the demosaicking accuracy.

In this paper, we integrate intra-channel image warping with cross-channel correlation, and show that our approach significantly improve the warping accuracy at very low cost. The proposed integration scheme is inspired by the Malvar-He-Cutler High Quality Linear Interpolation (HQLI) demosaicking method [17], which is probably the fastest demosaicking algorithm in the literature (except the Nearest Neighbour and Bilinear interpolation), but its trade-off between accuracy and speed is superior to many sophisticated algorithms [18]. We first examine the potential of cross-channel correlation for image warping, by applying HQLI to postprocess color images upsampled with intra-channel interpolation. We then generalize this combination to formulate general color image warping. The performance of our method is assessed in the scenario of image super-resolution, as it is the most active sub-area of image warping. Despite its very low cost, the proposed algorithm achieves comparable accuracy to state of the art algorithms at about $40$ times faster speed. Compared to CNN-based methods, our model has the flexibility of resizing the image to arbitrary size. These features are desirable for real time interactive applications on multimedia devices that have limited computation power (e.g., mobile smart phones).

2 Inter-Channel Correlation Gain

Image warping is generally implemented by backward mapping. Given $\mathbf{H}$ the geometric mapping from the source image $I_{1}$ to the target image $I_{2}$, and $\mathbf{H}^{-1}$ be its inverse mapping, $\mathbf{H}^{-1}$ maps the coordinate frame of $I_{2}$ back to the coordinate frame of $I_{1}$. Image $I_{2}$ is generated in such a way that, for each pixel $(i,j)$ of $I_{2}$, the intensity value $I_{2}[i,j]$ is copied from $I_{1}[x,y]$, where

[TABLE]

As $x$ and $y$ are generally not integers, $I_{1}[x,y]$ has to be estimated from the surrounding pixels. Let $m=\left\lfloor x\right\rfloor$ and $n=\left\lfloor y\right\rfloor$ be the floor integer parts of $x$ and $y$, and define $s=x-m$, $t=y-n$. For example, the bilinear warping scheme estimates $I_{2}[i,j]$ by

[TABLE]

Today, multimedia images are usually chromatic rather than being grayscale. Therefore, $I_{1}$ and $I_{2}$ generally have RGB three channels. Typically, bilinear warping for color images is to replace $I_{1}$ in Eq.6 with its color channels $R_{1}$, $G_{1}$ and $B_{1}$, to estimate $R_{2}[i,j]$, $G_{2}[i,j]$ and $B_{2}[i,j]$, without cross-channel interaction.

The importance of cross-channel correlation is well investigated for the problem of Image Demosaicking, entailed in current consumer-level digital cameras, which commonly use Color Filter Arrays (CFA) to sample one of the RGB components for each pixel. Fig.1 shows an example of the Bayer pattern CFA, adopted for most digital cameras.

Define $\mathcal{R}$, $\mathcal{G}$ and $\mathcal{B}$ to be the sets that collect the pixels where the original red, green and blue values are captured respectively. Malvar-He-Cutler propose estimating the missing color values by a weighted combination of the local intra-channel average and the Laplacian of another channel [17]. For example, at a pixel $[m,n]\in\mathcal{R}$, the green value $G[m,n]$ is estimated by

[TABLE]

where constant $\alpha$ is learned from training images, $\bar{G}$ stands for the average of $G[m,n-1]$, $G[m,n+1]$, $G[m-1,n]$ and $G[m+1,n]$, and

[TABLE]

The other missing color values are recovered in the same fashion. This demosaicking method, namely HQLI, adds second-order details (the Laplacian) of one channel to the average of another channel, thereby it performs image enhancement.

To examine whether the inter-channel information supplements intra-channel recovery in image warping. We conduct a preliminary experiment using image super-resolution as an example application. We upsample test images by bilinear and bicubic interpolation, and then refine the upsampled image values by sequential HQLI demosaicking. In particular, we first apply the GRBG CFA pattern to filter the upsampled color image, and estimate the filtered-out color values by the remaining color values. Subsequently, we apply the RGGB CFA to the demosaicked full RGB image, and perform the second-round demosaicking. We then change the CFA pattern to BGGR, and repeat HQLI demosaicking for the final round. In this way, each color component of a pixel is refined by the intensity values in the local neighbourhood of the other two channels. Table1 compares the reconstruction accuracy, measured by Peak Signal to Noise Ratio (PSNR), before and after demosaicking (i.e., the cross-channel information) on benchmark datasets Set5 [19], Set14 [20], BSD100 [21] and Urban100 [22]. It can be seen that the inter-channel correlation introduced by demosaicking notably improves the intra-channel interpolation, on all test datasets. Numerically, the improvement is upto $1.57$ dB and $1.09$ dB for bilinear and bicubic intra-channel interpolation respectively.

Note that to simulate the real world applications, here the low resolution images are obtained by Nearest Neighbour downsampling of the high resolution test images. This is different from the common SISR algorithm evaluation routine, which utilizes Bicubic downsampling. The Nearest Neighbour sampling is known to suffer severe aliasing and blurring, whereas Bicubic downsampling implicitly performs low pass filtering before sampling, thereby largely avoids such artifacts [16]. However, using Bicubic downsampling also implicitly assumes that the high resolution images are known, which is not the case for real applications. Therefore we only apply bicubic downsampling in the comparison experiments for fair conditions. Due to the coarse downsampling, bicubic warping has lower accuracy than bilinear warping, which is seemingly counter intuitive.

3 Cross-Channel Color Image Warping

3.1 General Formulation

Based on the analysis of the gain achieved by the inter-channel information, we formulate general color image warping by

[TABLE]

where $\tilde{G}_{1}[x,y]$, $\tilde{R}_{1}[x,y]$ and $\tilde{B}_{1}[x,y]$ can be any intra-channel estimation of $G_{1}[x,y]$, $R_{1}[x,y]$ and $B_{1}[x,y]$; also, $\Delta G_{1}[x,y]$, $\Delta R_{1}[x,y]$ and $\Delta B_{1}[x,y]$ can be estimated by interpolating ${\Delta G_{1}[k,l]}_{[k,l]\in\Omega(m,n)}$, ${\Delta R_{1}[k,l]}_{[k,l]\in\Omega(m,n)}$ and ${\Delta B_{1}[k,l]}_{[k,l]\in\Omega(m,n)}$ respectively, where $\Omega[m,n]$ stands for the local neighbourhood of pixel $[m,n]$. The involved interpolation schemes can be either linear or non-linear, isotropic or anisotropic, depending on the particular requirements of the applications at hand. The weights $\omega_{(\cdot,\cdot)}$are to be learned from training images, as described in Section3.2.

3.2 Parameter Training

We illustrate the weight learning process using the Single Image Super Resolution (SISR) application as an example, for which the geometric mapping $\mathbf{H}$ and its inverse mapping $\mathbf{H}^{-1}$ are

[TABLE]

where $S$ in a real number larger than $1$. Note that $S$ is not restricted to be an integer. We downsample the groundtruth images as the source images $\mathbf{I}_{1}$ (i.e., the data) by bicubic interpolation, following the traditional experimental setting in the literature of SISR, as the learned weights are to be used for algorithm evaluation in comparison with state of the art. We use the groundtruth images (taken from benchmark training dataset BSD200 [21]) as target images $\mathbf{I}_{2}$ (i.e., the labels). For each pixel $[i,j]$ in the an image $\mathbf{I}_{2}$, the RGB values of its correspondence $(x,y)$ in $\mathbf{I}_{1}$ thus are known in the training process. $\tilde{G}_{1}[x,y]$, $\tilde{R}_{1}[x,y]$ and $\tilde{B}_{1}[x,y]$, as well as $\Delta G_{1}[x,y]$, $\Delta R_{1}[x,y]$ and $\Delta B_{1}[x,y]$ are computed on the pixels in each source image by interpolation function $f$. Subsequently, Eq.9 can be rewritten as linear systems of the weights $\omega_{(\cdot,\cdot)}$. For example,

[TABLE]

for Eq.9a. Here integer $K=10000$ is the total number of randomly selected training pixels, indexed by the subscripts of $x$, $y$, $i$ and $j$. Eq.9b and Eq.9c can be re-organized similarly. The Mean Square Error (MSE) solutions or Mean Absolute Error (MAE) solutions to the linear systems can be easily solved in closed-form. This yields the weights $\omega_{g,r}$, $\omega_{g,b}$,$\omega_{r,g}$,$\omega_{r,b}$,$\omega_{b,r}$ and $\omega_{b,g}$.

Seemingly, the learned weight values should be specific to the scaling factor $S$ and interpolation function $f$. However, we observe that weights learned for large $S$ actually generalizes well to small $S$. Therefore, we suggest conducting the training for a large scaling factor (e.g., $S=4$), and apply the learned weights directly to applications whose scaling factor is smaller.

Table2 lists the weights we learned from Eq.15 by linear regression for $S=4$, with the interpolation function $f$ being Bilinear, Bicubic and Lanczos (with $5\times 5$ footprint).

4 Experimental Results

We conduct two sets of experiments to evaluate the proposed color image warping technique. The first experiment measures the SISR accuracy before and after incorporating the cross-channel terms to intra-channel warping, at various scaling factors and using different real time interpolation functions. The second experiments compares our technique with the state of the art SISR method Global Regression (GR) proposed by Timofte-De Smet-Van Gool in [23], using the released Matlab source code. The GR method is the fastest SISR method proposed in recent years to the best of our knowledge, and hence is chosen for comparison in our experiment. It should be mentioned that, the weight GR is significantly higher than our linear isotropic scheme used for testing.

For comparison in equal conditions, the low resolution test images $I_{1}$ are obtained by bicubicly downsampling the ground truth image $I_{2}$ at sampling factor $S$. That is, the low resolution images have $\left\lfloor\frac{M}{S}\right\rfloor$ rows and $\left\lfloor\frac{N}{S}\right\rfloor$ columns. Laplacians are computed at each pixel in each channel of the low resolution images. Both the Laplacian maps and low resolution images are upsampled by the same interpolation function $f$. In our experiments, $S$ is set to be 2, 3 or 4; and $f$ is set to be Bilinear, Bicubic or Lanczos (of kernel size $5\times 5$).

Evaluation is carried out on benchmark datasets Set5, Set14, BSD100 and Urban100. Beside the PSNR accuracy measure, we also report the Structure Similarity (SSIM) measure [24] and running time (in seconds) of the SISR algorithms. All experiments are conducted on an Intel Core i7-7700 3.60GHz CPU with 8GB RAM, using Matlab.

4.1 Improvement on Popular Warping Methods

We compare the performance of popular real time warping methods and the proposed method. Bilinear warping probably has the best trade-off between performance and speed in the literature, as pointed out by Zitová-Flusser [8]. Hence it is recommended for video and animation applications, such as frame registration for motion estimation in OpenCV and texture mapping in OpenGL. Bicubic warping is another frequently adopted option, especially for still image processing. For example, image editing softwares Photoshop and GIMP employ bicubic interpolation for image resizing and perspective view rectification.

Table3 presents the PSNR, SSIM and running time averaged over Set5 for various $S$ and $f$. In this table, the plain intra-channel warping is referred to as “Independent”, and the proposed inter-channel color warping is referred to as “Correlated”. In all situations, the proposed color warping scheme achieves evidential improvement, either in terms of PSNR or SSIM, in real time computation. The most remarkable quantitative improvement is obtained for bilinear interpolation, the PSNR values of which are increased by 1.58dB, 0.77dB and 0.98dB for 2, 3 and 4 times upscaling respectively. It can be observed that the improvement decreases with the complexity of the interpolation function. However, in the case of Lanczos interpolation (the most complex function here), the accuracy of our color warping scheme for $2\times$ upscaling is probably sufficient in many applications, and its computation is merely in several milliseconds.

4.2 Comparison with State of the Art

To the best of our knowledge, among the state of the art SISR algorithms that have Matlab source code released, GR [23] is the fastest. We compare the accuracy and speed of our method with GR. In this experiment, we fix the interpolation function $f$ to be Lanczos. Table 4 shows the comparison, and Fig.2 shows a visual assessment. Relative to GR, the PSNR of our method is generally around $0.2$dB lower for various scaling factors. However, on BSD100 the accuracy performance are very close. This might be attributed to the integration weights learned from BSD200, and suggests the potential for higher performance by using more sophisticated training strategy. It should be noted that the proposed warping model has merely 6 parameters, and the computation is tens of times faster.

5 Conclusion

In this paper, we have proposed a light weight color image warping technique, which integrates intra-channel interpolation with cross-channel details. Extensive experiments on four benchmark datasets have validated that, the proposed technique substantially improves the most popular image warping algorithms, in very simple form and at trivial computational cost. Our algorithm has also been shown to push the performance of traditional Lanczos upsampling scheme to be comparable with state of the art methods, while being real time.

The proposed technique can be readily applied to interactive multimedia applications that require both fast image warping and realistic visual effects, such as Computer Gaming and Image Editing. As our integration of cross-channel information is of light weight, it is suitable for multimedia devices with limited computation power. It should be noted that our baseline interpolation, integration formulation and weight training are all very basic. Actually, each step can be extended for higher performance, if the computation cost budget allows. Moreover, our method can also be used as an high quality initialization to many CNN-based methods.