Joint Regression and Ranking for Image Enhancement

TL;DR

This paper introduces a novel end-to-end joint regression and ranking method using Gaussian processes for automated image enhancement, effectively modeling the relationship between images and enhancement parameters.

Contribution

It presents a new approach that explicitly models the structure of enhancement parameters and images, improving parameter search and image ranking in enhancement tasks.

Findings

Effective parameter prediction using only image features

Improved image enhancement quality demonstrated on benchmark datasets

The ranking method correlates well with subjective quality assessments

Abstract

Research on automated image enhancement has gained momentum in recent years, partially due to the need for easy-to-use tools for enhancing pictures captured by ubiquitous cameras on mobile devices. Many of the existing leading methods employ machine-learning-based techniques, by which some enhancement parameters for a given image are found by relating the image to the training images with known enhancement parameters. While knowing the structure of the parameter space can facilitate search for the optimal solution, none of the existing methods has explicitly modeled and learned that structure. This paper presents an end-to-end, novel joint regression and ranking approach to model the interaction between desired enhancement parameters and images to be processed, employing a Gaussian process (GP). GP allows searching for ideal parameters using only the image features. The model naturally leads to a ranking technique for comparing images in the induced feature space. Comparative evaluation using the ground-truth based on the MIT-Adobe FiveK dataset plus subjective tests on an additional data-set were used to demonstrate the effectiveness of the proposed approach.

Figures

Equations

$$p({\bar{y}}_{\ast m}^{+}|{\mathbf{f}}_{\ast },\mathbf{F},\mathbf{Y})=\mathcal{N}({\bar{y}}_{\ast m}^{+}|{\mathbf{k}}_{\ast }^T{\mathbf{K}}_y^{-1}{\bar{\mathbf{y}}}_m^{+},k_{\ast \ast }-{\mathbf{k}}_{\ast }^T{\mathbf{K}}_y^{-1}{\mathbf{k}}_{\ast })$$

$$\log p({\bar{\mathbf{y}}}_m^{+}|\mathbf{F})$$

$$0.5Nlog(2\pi )$$

$$\kappa ({\mathbf{f}}_i,{\mathbf{f}}_j)={\sigma }_f^2\exp (-\frac{1}{2}({\mathbf{f}}_i-{\mathbf{f}}_j{)}^T\cdot \mathbf{\Lambda }\cdot ({\mathbf{f}}_i-{\mathbf{f}}_j))+{\sigma }_y^2{\delta }_{ij}$$

$$\underset{w,{\xi }_{ij}}{\min }\frac{1}{2}{\mathbf{w}}^T\mathbf{w}+C\sum _{i,j}{\xi }_{ij},\text{subject to:}{\mathbf{u}}_i\succ {\mathbf{u}}_j\forall (i,j)$$

$$\underset{w,{\xi }_{ij}}{\min }\frac{1}{2}{\mathbf{w}}^T\mathbf{w}+C_1\sum _{i,j}{\xi }_{ij}+C_2\sum _{i,k}{\xi }_{ik}^{\prime },$$

$$\text{subject to:}{\mathbf{w}}^T{\mathbf{f}}_{ij}^{+}\ge {\mathbf{w}}^T{\mathbf{f}}_i+1-{\xi }_{ij},$$

$$\text{subject to:}{\mathbf{w}}^T{\mathbf{f}}_i\ge {\mathbf{w}}^T{\mathbf{f}}_{ik}^{-}+1-{\xi }_{ik}^{\prime },$$

$$\text{subject to:}{\mathbf{w}}^T{\mathbf{f}}_{ij}^{+}\ge {\mathbf{w}}^T{\mathbf{f}}_{ik}^{-}+1-{\xi }_{ik}^{\prime \prime },{\xi }_{ij},{\xi }_{ik}^{\prime },{\xi }_{ik}^{\prime \prime }\ge 0$$

$$\forall i=\{1,\dots ,N\},\forall j=\{1,\cdots ,p\},\forall k=\{1,\cdots ,p\}.$$

$$\underset{w,{\xi }_i}{\min }\frac{1}{2}{\mathbf{w}}^T\mathbf{w}+C\sum _i{\xi }_i,$$

$$\text{subject to:}{\mathbf{w}}^T{\mathbf{D}}_i+1-{\xi }_i\le 0,{\xi }_i\ge 0,\forall i=\{1,\dots ,N^{\prime }\}.$$

$$L(\mathbf{w},\mathbf{\alpha },\mathbf{\beta })$$

$$\sum _i{\alpha }_i({\mathbf{w}}^T{\mathbf{D}}_i+1-{\xi }_i)-\sum _i{\beta }_i{\xi }_i$$

$${\nabla }_{w}L(\mathbf{w},\mathbf{\alpha },\mathbf{\beta })=0\Rightarrow \mathbf{w}=-\sum _i{\alpha }_i{\mathbf{D}}_i$$

$${\nabla }_{\xi }L(\mathbf{w},\mathbf{\alpha },\mathbf{\beta })=C-{\alpha }_i-{\beta }_i=0\Rightarrow {\alpha }_i\le C.$$

$$\underset{\mathbf{\alpha }}{\max }\sum _i{\alpha }_i-\frac{1}{2}\sum _i\sum _j{\alpha }_i{\alpha }_j{\mathbf{D}}_i^T{\mathbf{D}}_j,\text{subj. to:}0\le {\alpha }_i\le C.$$

$$\underset{\mathbf{\alpha }}{\max }{1}^T\mathbf{\alpha }-\frac{1}{2}{\mathbf{\alpha }}^T{\mathbf{K}}_y\mathbf{\alpha }.$$

$$\underset{h}{\min }\left(\sum _i||{\mathbf{K}}_y^{{\mathbf{F}}_i^{+}}|{|}_F^2-||{\mathbf{K}}_y^{{\mathbf{F}}_i^{+},{\mathbf{F}}_i^{-}}|{|}_F^2\right),$$

$${\mathbf{K}}_y^{{\mathbf{F}}_i^{+},{\mathbf{F}}_i^{-}}=\left[\begin{array}{ccc}\kappa ({\mathbf{f}}_{i1}^{+},{\mathbf{f}}_{i1}^{-})& \cdots & \kappa ({\mathbf{f}}_{i1}^{+},{\mathbf{f}}_{ip}^{-})\\ \kappa ({\mathbf{f}}_{i2}^{+},{\mathbf{f}}_{i1}^{-})& \cdots & \kappa ({\mathbf{f}}_{i2}^{+},{\mathbf{f}}_{ip}^{-})\\ ⋮& \ddots & ⋮\\ \kappa ({\mathbf{f}}_{ip}^{+},{\mathbf{f}}_{i1}^{-})& \cdots & \kappa ({\mathbf{f}}_{ip}^{+},{\mathbf{f}}_{ip}^{-})\end{array}\right]$$

$$\underset{h}{\min }Z=$$

$$\frac{1}{2}{\mathbf{\alpha }}^T{\mathbf{K}}_y\mathbf{\alpha }+\sum _i\left(||{\mathbf{K}}_y^{{\mathbf{F}}_i^{+}}|{|}_F^2-||{\mathbf{K}}_y^{{\mathbf{F}}_i^{+},{\mathbf{F}}_i^{-}}|{|}_F^2\right)$$

$$\frac{\partial Z}{\partial {\mathbf{K}}_y}=-\frac{1}{2}{\mathbf{K}}_y^{-1}$$

$$2\sum _i\left({\mathbf{K}}_y^{{\mathbf{F}}_i^{+}}-{\mathbf{K}}_y^{{\mathbf{F}}_i^{+},{\mathbf{F}}_i^{-}}\right),$$

$${\left[\frac{\partial {\mathbf{K}}_y}{\partial {\theta }_q}\right]}_{ij}=-\frac{1}{2}{\sigma }_f^2$$

$$({\mathbf{f}}_i^{(q)}-{\mathbf{f}}_j^{(q)}{)}^2,$$

$$\frac{\partial {\mathbf{K}}_y}{\partial {\sigma }_f^2}={\sigma }_f^2\exp$$

$${\left[\frac{\partial {\mathbf{K}}_y}{\partial {\sigma }_y^2}\right]}_{ij}={\delta }_{ij},$$

$$\frac{\partial Z}{\partial {\theta }_q}=\text{tr}\left[{\left(\frac{\partial Z}{\partial {\mathbf{K}}_y}\right)}^T\left(\frac{\partial {\mathbf{K}}_y}{\partial {\theta }_q}\right)\right]\forall q\in \{1,\dots ,D\},$$

Full text

1 Introduction

The corpus of images on the Web is exponentially increasing in size with close to two billion photos being added or circulated each day111http://www.kpcb.com/internet-trends. Image sharing has become an integral part of daily life for many people, and they want their photos to look good without doing too much manual editing. Some tools for easy image enhancement have already been deployed on popular social networking platforms, such as Instagram, or mobile devices such as iPhone. However, most such tools are essentially based on some pre-defined image filters for obtaining certain visual effects. Recent research efforts on automated image enhancement employing machine learning techniques for improved functionalities such as content adaptivity and personalization. Such solutions range from learning a tone mapping between the spaces of low-quality and high-quality images222We call the images before enhancement as low-quality and those after the enhancement as high-quality in the rest of this article. We also refer to the process of enhancing a new picture as “the testing stage”. to building a ranking relation between these two spaces [2, 5, 13, 9, 12, 14, 6], although we are yet to see such techniques being deployed on a popular platform.

Many recent approaches follow the pipeline shown in Fig. 1. During training, these approaches learn a model to assign a score for a given image quantifying its visual appeal. To enhance a new image, a nearest training image is found. Then a dense sampling around the parameters333The brightness, saturation and contrast are referred to as “parameters” of an image in this article of the high-quality counterpart of that training image gives the candidate set of enhancement parameters for the new image. A set of candidate images is then generated by applying these enhancement parameters to the new image. The next step is extracting features of all candidate images and use the learned model to select the highest-quality image.

There are two major drawbacks in the above processing flow. First, it is computationally expensive at the testing phase since a search through the entire training data is needed. The training data for such applications could be of huge size and is usually hosted on a server. Thus hundreds of thousands people querying the server per second is undesirable and uncalled-for. Second, the set of candidate parameters which would enhance the original image is found in a sub-optimal manner by doing a $k$NN search. It does not provide any structured way to search for the optimal parameters and thus it becomes necessary to search the entire training set and create a lot of candidate images, resulting a computational bottleneck for the testing phase.

In this paper, we develop a joint regression and ranking approach to address the above drawbacks. Our approach employs GP regression to predict the mean and variance of the candidate parameters only from the feature vector of a low-quality image. We also simultaneously train a ranking model on the GP-covariance-kernel-induced feature space. To achieve this, we derive and use the dual-form of ranking SVM [10] with the GP kernel integrated into it. Thus the kernel builds a relation between the image feature space and its corresponding enhancement parameter space. Along with that, the kernel learns to give more weight to the image features which are highly responsible for making an image to be of higher quality. Finally, we learn the GP kernel in such a way that all the high-quality counterparts of a low-quality image form a cluster. This allows exploration of the image parameter space in a structured manner for obtaining the optimal solution.

In the testing stage (i.e., while enhancing a new image), the model provides the expected value and variances of the enhancement parameters, drastically reducing required computation since there is no need to sift through the training set. We can generate/show some enhanced images by applying parameters that are $k$ standard deviations away from the expected values of the parameters, where $k$ is a user-defined and can be changed on-the-fly, so that user can choose from some good candidate images. The same kernel can be used to rank images if the user wants to see a single image. Through extensive experiments, we show that our approach is computationally efficient at the testing phase, and that it predicts parameters correctly for new images and that it also predicts the ranking relations between the new images and its enhanced counterparts.

2 Related Work

Automated image enhancement has recently been an active research area. Various solutions have been proposed for this task. We review those works which aim to improve the visual appeal of an image using automated techniques. A novel tone-operator was proposed to solve the tone reproduction problem [18]. A database named MIT-Adobe FiveK of corresponding low and high-quality images was published in [5]. They also proposed algorithm to solve the problem of global tonal adjustment. The tone adjustment problem only manipulates the luminance channel. In [11], an approach was presented, focusing on correcting images containing faces. They built a system to align faces between a “good” and a “bad” photo and then use the good faces to correct the bad ones.

Content-aware enhancement approaches have been developed which aim to improve a specific image region. Some examples of such approaches are [2, 14]. A drawback of these is the reliance on obtaining segmented regions that are to be enhanced, which itself may prove difficult. Pixel-level enhancement was performed by using local scene descriptors. First, images similar to the input are retrieved from the training set. Then for each pixel in the input, a set of pixels was retrieved from the training set and they were used to improve the input pixel. Finally, Gaussian random fields are used to maintain the spatial smoothness in the enhanced image. This approach does not take the global information of an image into account and hence the local adjustments may not look right when viewed globally. A deep-learning based approach was presented in [26]. In [12], users were required to enhance a small amount of images to augment the current training data.

Two closely related and recent works involve training a ranking model from low and high-quality image pairs [25, 6]. In a recent state-of-art method [25], a dataset of $1300$ corresponding image pairs was reported, where even the intermediate enhancement steps are recorded. A ranking model trained with this information can quantify the (enhancement) quality of an image. In [6], non-corresponding low and high-quality image pairs were used to train a ranking model. Both the approaches use $k$NN search at the test time to create a pool of candidate images first. After extracting features and ranking all of them, the best image is presented to the user.

Now we briefly review Gaussian process based methods which are relevant in this context. GP has been effectively used to obtain good performance for applications where complex relationships have to be learned using a small amount of data (in the order of several hundreds) [23]. In [7], it was used for view-invariant facial recognition. A GP latent variable model was used to learn a discriminative feature space using LDA prior where examples from similar classes are project nearby. In [20], GP regression was used to map the non-frontal facial points to the frontal view. Then facial expression methods can be used using these projected frontal view points. Coupled GP have been used to capture dependencies between the mappings learned between non-frontal and frontal poses, which improves the facial expression recognition performance [19].

Our effort in this paper deals with enhancement considering contrast, saturation and brightness of an image. We attempt to explicitly model interactions between parameters controlling these factors and features extracted from the underlying image, employing GP. Our approach of joint regression and ranking allows us to learn the complex mapping from the image features to the regions corresponding to desired enhancement in the parameters space, without actually generating several hundreds of enhanced candidate images. The expected value of the parameters and their standard deviations provide us with a way to systematically explore the parameters space. In the next section, we detail our proposed approach. To the best of our knowledge, this is the first attempt on incorporating GP regression and ranking for image enhancement.

3 Proposed Approach

Our problem is to predict the set of image parameters which would enhance a given image. Our proposed approach consists of two objectives: 1. Given a low-quality image feature, probabilistically estimate the parameters that could generate the enhanced counterpart. 2. Produce a ranking in the GP-kernel-induced feature space and thereby discover the features responsible for making an image of higher-quality.

We have pairs of low and high-quality images along with their parameters for training. The feature representation of an image will be discussed in detail in section 3.6. Features of $N$ low-quality images are represented by $\bm{F}=\{\bm{f}_{1},\bm{f}_{2},\ldots,\bm{f}_{N}\}$444We represent vectors by lower-case bold letters. Matrices are represented by upper-case bold letters. Scalars are denoted by non-bold letters.. We have $p$ high-quality versions for each low-quality image. Its features are represented by $\bm{F}^{+}=\{\bm{F}_{1}^{+},\ldots,\bm{F}_{N}^{+}\}$, where $\bm{F}_{i}^{+}=\{\bm{f}_{i1}^{+},\ldots,\bm{f}_{ip}^{+}\}$, and $\bm{f}_{i},\bm{f}_{ij}^{+}\in\mathbb{R}^{D\times 1}\ \forall\ i,j$. We also have $p$ sets of high-quality parameters for a low-quality image. However, for illustration, we predict parameters only for the first set. Note that we still use all the $p$ sets of high-quality images to train a ranking model. The parameters of low and high-quality images are represented by $\bm{Y}=\{\bm{y}_{1},\ldots,\bm{y}_{N}\}$ and $\bm{Y}^{+}=\{\bm{y}_{1}^{+},\ldots,\bm{y}_{N}^{+}\}$ respectively. We use three image parameters, namely, brightness, contrast and saturation and hence $\bm{y}_{i},\bm{y}_{i}^{+}\in\mathbb{R}^{3\times 1}\ \forall\ i$. Our task is to obtain $\bm{y}_{i}^{+}$ using only $\bm{f}_{i}$ and $\bm{y}_{i}$. We predict each parameter using a separate GP. To that end, we collect the $m^{th}$ parameter of all low and high-quality images into, $\bar{\bm{y}}_{m}=(y_{1m},\ldots,y_{Nm})^{T}$ and $\bar{\bm{y}}_{m}^{+}=(y_{1m}^{+},\ldots,y_{Nm}^{+})^{T}$, respectively and train a separate GP model to predict each parameter. We now outline the proposed joint GP regression and ranking.

3.1 GP Regression

GPs define a prior distribution over functions which becomes a posterior over functions after observing the data. GPs assume that this distribution over functions is jointly Gaussian with a mean and a positive definite covariance kernel function. GPs provide well-calibrated, probabilistic outputs which are particularly useful and necessary in our application [15]. If we let the prior on regression function be a GP, then it can be denoted as $GP(m(\bm{f}),\kappa(\bm{f},\bm{f}^{\prime}))$ where $\bm{f}$ and $\bm{f^{\prime}}$ are image features $\in\mathbb{R}^{D\times 1}$ as defined previously, $m(\bm{f})$ is a mean function and $\kappa(\bm{f},\bm{f}^{\prime})$ is a covariance function. It is well-known that GPs are flexible enough to model an arbitrary mean. It can be shown that the posterior predictive density for a single test input is:

[TABLE]

where $\bm{k}_{*}=[\kappa(\bm{f}_{*},\bm{f}_{1}),\ldots,\kappa(\bm{f}_{*},\bm{f}_{N})]$, $N$ is the number of samples, $k_{**}=\kappa(\bm{f}_{*},\bm{f}_{*})$ and $\bm{K}_{y}=\bm{K}+\sigma_{y}^{2}\bm{I}_{N}$. $\bm{K}$ is a kernel function between all training inputs $\bm{f}$, and $(\cdot)_{*}$ denotes a new data sample. The noise or uncertainty in the output is modeled by the noise variance, $\sigma_{y}^{2}$ .

It can be further shown that the log-likelihood function for a GP regression model is easily obtained by using a standard multivariate Gaussian distribution as follows,

[TABLE]

We choose a standard squared exponential kernel for our application. It is as follows,

[TABLE]

Here, $\sigma_{f}^{2}$ controls the vertical scale of the regression function, $\sigma_{y}^{2}$ models uncertainty, $\bm{\Lambda}$ is a diagonal matrix with entries $\{\theta_{1},\ldots,\theta_{D}\}$ and $\delta_{pq}$ is a Kronecker delta function, which takes the value $1$ if $p=q$, it is zero everywhere else. We call $\bm{h}=\{\sigma_{f}^{2},\bm{\Lambda},\sigma_{y}^{2}\}$ as hyper-parameters. It is easy to see that the prediction in Equation 1 is dependent on the kernel and in turn on the hyper-parameters: $\sigma_{f},\bm{\Lambda}$ and $\sigma_{y}^{2}$. We will see later how to obtain optimal hyper-parameters. Now, we explain inclusion of ranking into our formulation.

3.2 GP Ranking

We build a ranking relation in the GP-kernel-induced feature space. Thus the GP kernel discovers the features responsible for making an image of higher-quality and assigns higher weight to them by adjusting the hyper-parameters. The primal form of rank SVM [10] is given by:

[TABLE]

where $\bm{u}_{i}\succ\bm{u}_{j}$ indicates that $\bm{u}_{i}$ is ranked higher than $\bm{u}_{j}$.

For ranking, we observe that only building a relation between low and high-quality images does not provide good ranking accuracy on new images. The reason is that the enhanced images often possess high saturation, brightness and/or contrast. Thus the ranking model sometimes assigns a higher score to over-saturated and over-exposed images. This would not be a problem if one had intermediate information about the enhancement steps being performed [25]. Thus we deteriorate our original low-quality images by shifting the image parameters to both extremes. The amount of shifting for an image is decided by first deteriorating 20 images manually and then heuristically defining a relation between existing image parameters and the amount of shifting needed to significantly deteriorate the image. Let’s call these images as poor-quality images. We also generate $p$ poor-quality images for every low-quality image. We now have features for poor, low and high quality images, denoted by $\bm{F}^{-},\bm{F}$ and $\bm{F}^{+}$ respectively. Primal form for our ranking model can be written as follows,

[TABLE]

Now we derive the dual form to incorporate the GP-kernel, $\kappa$. It would be cumbersome to derive the dual of Equation 5 as it stands. Instead, we can define a new set of data $\bm{D}$ consisting of $\bm{f}_{i}-\bm{f}_{ij}^{+}$, $\bm{f}_{ik}^{-}-\bm{f}_{i}$ and $\bm{f}_{ik}^{-}-\bm{f}_{ij}^{+}\ \forall\ i,j,k$. Now, $\bm{D}$ has $N^{\prime}=N(2p+p^{2})$ elements, we can write the primal form as follows:

[TABLE]

We use Lagrangian multipliers to convert the above equation into an unconstrained optimization problem.

[TABLE]

Differentiating with respect to $\bm{w}$ and $\xi$ and equating them to zero, we get,

[TABLE]

Substituting $\bm{w}$ back into Equation 6 and doing some algebraic manipulation, we get a dual maximization problem as,

[TABLE]

The inner product in the above equation can be replaced with GP kernel by employing the kernel trick. Thus the final optimization problem to get $\bm{\alpha}$ becomes,

[TABLE]

Here, $\bm{1}$ is a column vector of ones. The length of both $\bm{\alpha}$ and $\bm{1}$ is $N(2p+p^{2})$. The dimensions of $\bm{K}_{y}$ are $N(2p+p^{2})\times N(2p+p^{2})$. The $(i,j)^{th}$ element of $\bm{K}_{y}$ is $\kappa(\bm{D}_{i},\bm{D}_{j})$.

3.3 Clustering high-quality images together

We turn our attention to the third constraint. Given a low-quality image: 1. it forces all its high-quality counterparts to form a cluster and 2. it tries to maximize the distance between poor-quality and high-quality images in the GP-kernel-induced feature space. The intuition behind this is as follows. Ultimately, given a new image, we would not only like to predict the parameters for its enhanced counterpart, but we also wish to traverse the parameter space and explore more of such enhancement parameters. The traversing of the parameter space is, in our opinion, essential since the choices of people vary by a great amount and no model would do justice with just one set of predicted parameters. Note that this constraint tries to minimize distance between $\bm{f}_{i}$ and $\bm{f}_{ij}^{+}\ \forall j$, so by definition of GP, the distance between the corresponding output parameters, $\bm{y}_{ij}^{+}\ \forall j$, will be reduced, which in turn achieves the aforementioned effective traversal. The second part of the constraint tries to push the predicted parameters away from the parameters of the poor-quality images. The details of traversing the parameter space after getting the GP predictions are discussed later. The constraint can be formulated as follow.

[TABLE]

where $||\cdot||_{F}^{2}$ indicates squared Frobenius norm. The term $\bm{K}_{y}^{\bm{F}_{i}^{+},\bm{F}_{i}^{-}}$ is a $p\times p$ matrix defined as follows,

[TABLE]

The term $\bm{K}_{y}^{\bm{F}_{i}^{+}}$ is equal to $\bm{K}_{y}^{\bm{F}_{i}^{+},\bm{F}_{i}^{+}}$.

We combine Equations 2, 10 and 11 to form our objective function. It is as follows,

[TABLE]

Note that we have removed the constant term. We now focus on how to solve Equation 10 and 13 to get $\bm{\alpha}$ and $\bm{h}$.

3.4 Optimization

Our optimization problem is separable in $\bm{\alpha}$ and $\bm{h}$. First we optimize $\bm{\alpha}$, which can be done by using a standard rank-SVM solver. It could also be solved by using quadratic programming, however, that would be memory inefficient. In particular, we use a rank-SVM implementation which uses the LASVM algorithm proposed in [4]. LASVM employs active example selection to significantly reduce the accuracy after just one pass over the training examples.

After optimizing $\bm{\alpha}$, we turn our attention to Equation 13. We find its local minimizer, $\bm{h}^{*}$, by using scaled conjugate gradient descent (SCG) algorithm. SCG is chosen due to its ability to handle tens of thousands of variables. SCG has also been widely used in previous approaches involving GPs [17, 7, 19]. We use chain rule to compute $\frac{\partial Z}{\partial\bm{h}}$ by evaluating first $\frac{\partial Z}{\partial\bm{K}_{y}}$ and then $\frac{{\partial\bm{K}_{y}}}{\partial\bm{h}}$. The matrix calculus identities from [16] are used while computing the following expressions.

[TABLE]

where tr denotes matrix trace. Similarly, $\frac{\partial Z}{\partial\sigma_{f}^{2}}$ and $\frac{\partial Z}{\partial\sigma_{y}^{2}}$ are computed to construct $\frac{\partial Z}{\partial\bm{h}}\in\mathbb{R}^{D+2}$. This derivative can now be used to obtain the optimal set of hyper-parameters, $\bm{h}$. In practice, all the matrix inverses are implemented using Cholesky decomposition. We alternately optimize for $\bm{\alpha}$ and $\bm{h}$ till Equation 13 converges or the maximum cycles are reached. We set the convergence criterion to be $10^{-3}$ and the maximum cycles to 20.

3.5 Testing

Once we get the optimal $\bm{\alpha}$ and $\bm{h}$, we can predict the parameters, $\{\bar{y}_{*1}^{+},\bar{y}_{*2}^{+},\bar{y}_{*3}^{+}\}$, for the enhanced counterpart by using three trained GP models in Equation 1. Let us call the mean and variances of the predicted parameters as $\bm{m}=\{m_{1},m_{2},m_{3}\}$ and $\bm{s}=\{s_{1},s_{2},s_{3}\}$ respectively. With their availability, we now explain our parameter space traversal.

As mentioned before, people’s choices vary a lot in such applications. Thus, it is essential to explore the parameter space to generate additional enhancement parameters. The first advantage of our approach is that we can generate such parameters without referring to the training set. Since we explore the parameter space in a structured manner (with a certain mean and variance), we can afford to generate only $32$ parameters per image instead of hundreds as done in conventional $k$NN-based heuristic methods.

The First step in parameter space traversal is to determine lower and upper bounds. Those can be decided heuristically. For example, we decrease the saturation, brightness and contrast at most by an amount of $\{15\%,15\%,5\%\}$ and increase it at most by $\{35\%,35\%,20\%\}$ of the original image parameter values. We observed that these limits are not absolutely critical to the quality since the generated images will be ranked later using the learned $\bm{\alpha}$ and the images with extreme parameter settings will usually be filtered out.

Now, we change (increase and decrease) the mean value of the parameters by $\bm{\mu}\bm{s}$ till it reaches the pre-specified thresholds. Intuitively, we think that $\bm{s}$ gives us the direction of our stride in the parameter space and $\bm{\mu}$ gives us the length of that stride. The value of $\bm{\mu}$ is determined by the number of enhanced counterparts the user wants to generate for each low-quality image. We set that value to be $30$. This value could be decreased if the user is on a mobile device with a smaller screen and similarly increased when operating on a desktop. These settings can be changed on-the-fly.

3.6 Image feature representation

We extract $432$-D color histogram with 12 bins for hue, 6 bins each for saturation and value, which acts as a global feature. We then divide the image into a $12\times 12$ grid. For each grid, we calculate its saturation, value by taking the mean values of those image blocks in the HSV color space. We also calculate RMS contrast on that grid. These act as localized features of $144$-D each. We finally append the image parameters, which are average saturation, value and RMS contrast. Appending the image parameters allows GP to express the parameters of the enhanced counterparts as a function of both, the low-quality parameters and its feature vector. We finally get a $867$-D $(=432+3\times 144+3)$ representation for every image.

3.7 Implementation Details and Efficiency

GPs are known to be computationally-intensive. They take about $O(N^{3})$ time for training, where $N$ are the number of training examples. The matrix inversion of an $N\times N$ matrix and the computation of the derivative of the kernel are the bottlenecks in the GP training procedure. We train a GP model using about $1200$ low-quality images and six counterparts per image in about $18$ hours on an Intel Xeon **@**2.4 GHz $\times$ 16. The computational efficiency can be improved by using GP regression techniques proposed for large data [8, 1] or using efficient data-structures such as kd-trees [22]. During testing, our approach is extremely fast. We tested it on two systems, Intel Xeon and a modern desktop system with Intel i7 **@**3.7GHz. It can predict all the three parameters for $3150$ and $1287$ images per second using Intel Xeon and i7 systems respectively. A built-in $k$NN-search function processes only $224$ images per second when asked to find one nearest neighbor in $5000$ image data-set on the Intel Xeon system. All the implementations are done in MATLAB. Since our approach need not query the training database, it could be portable and potentially allow for enhancements being performed on mobile devices.

4 Data-sets and Experimental Setup

In this section, we present describe the data and experimental setup. Results of these experiments are presented in the Section 5. We perform four kinds of experiments. The first experiment provides a weak quantitative measure of the accuracy of our approach. We use the MIT-Adobe FiveK [5] data-set for this experiment. This data-set has $5000$ low-quality images with $5$ expert-enhanced counterparts for each image. This is the largest such data-set available. We use $1200$ images and 6 counterparts (three each for poor and high-quality) per low-quality image to train our GP models. We use $1500$ and $800$ images for validation and testing respectively. We predict the parameters (i.e. brightness, contrast and saturation) for the first enhanced counterpart of all the images in the test set. Then we calculate the root mean square error (RMSE) and a more stringent criterion - Pearson’s correlation - between the ground-truth parameters computed from expert-enhanced image and our predicted parameters. We compare our quantitative results against twin Gaussian processes (TGP) [3]. TGP is a strutured prediction method which considers correlation between both input and output to produce predictions. Though a low RMSE between ground truth and predicted parameters does not guarantee that the enhancement will be visually appealing (unless the RMSE tends to zero), it gives us a confirmation that the prediction is lying near the ground-truth in the parameter space. Also, this experiment validates the effectiveness of the GP regressor.

Second experiment is a qualitative measure of the image quality produced by the proposed and the competing algorithms - $k$NN, Picasa and [25]. The metric of $L2$ error in the L*ab space was adopted in [25]. We believe that it is a poor indicator of the enhancement quality and instead opt for Visual Information Fidelity (VIF) metric [21]. This metric has the ability to predict whether the visual quality of the other image has been enhanced in comparison with the reference image by producing a value greater than one. This is unlike other quality metrics such as SSIM [24], FSIM [28], VSI [27] etc. We use the publicly available implementation of VIF555available at live.ece.utexas.edu/research/quality/. We calculate the VIF between the proposed enhancement (reference image) and the enhancement by 1. $k$NN 2. Picasa and 3. the approach of [25] (other image). Thus VIF $<1$ implies that the proposed enhancement is better than the one produced by the competing algorithm and vice-versa. This comparison is done for 60 pairs where 15 images each are enhanced using Picasa and [25], whereas the remaining 30 images are enhanced using $k$NN approach.

Third experiment is aimed towards evaluating the effectiveness of GP ranking. For each image we generate only 32 enhanced versions. Our GP ranker selects the highest ranked image out of those 32 and presents it to the user. The highest ranked image is supposed to have the best quality. We compute the VIF metric between the best image selected by the ranker (reference image) and the other 31 images (other images). Ideally, for all these 31 images, we should get values less than one indicating that GP ranker has indeed selected the best image.

We also carry out a subjective evaluation test to assess if people prefer the enhanced counterparts generated by our approach. We compare our approach against three other methods. First one is the $k$NN-based approach. Given a low-quality image, we search for the nearest non-duplicate image from the $5000$ images of MIT-Adobe dataset. The parameters of the expert-enhanced counterparts of the nearest image are applied to the given low-quality image. In this manner, we generate $5$ enhanced counterparts per low-quality image. Note that, $k$NN utilizes all other $4999$ images whereas we only use the model trained on $1200$ images for prediction. Then we compare against Picasa’s one-touch-enhance tool. The third approach is from [25], which also is a learning-to-rank based image enhancement approach that uses the pipeline shown at the top in Fig. 1.

We use $60$ images for the subjective test which was performed by 15 people. Thirty images are selected from our testing set of the MIT-Adobe data-set. The rest of the images are from the data-set used in the paper [25]. Since we only have access to their testing set, we use that data-set solely for subjective test purposes. It contains $124$ images out of which we randomly select $30$ images. We enhance all the $60$ images using our approach. The comparison against other methods is done as follows.

The first $30$ images from the MIT-Adobe data is split into two halves. The first half is enhanced using the $k$NN approach and the second half is enhanced using Picasa. The remaining 30 images from [25] are split into two halves. The first half is enhanced using the $k$NN approach and for the second half, we directly use the high-resolution results of the test data-set of [25]. Thus each person compares 60 image pairs. One of the image in that pair has been enhanced using our approach and the other image has been enhanced using either $k$NN approach, Picasa or the approach of [25]. The subject has to choose the image which he/she finds “visually-appealing”. If the subject feels that both images have almost the same visual appeal, a third option of preferring neither image is provided. The order in which the images appear in front of a subject is always randomized. The pairing order is also randomized. The subjects do the evaluation test in standard lighting conditions and at a comfortable and constant distance from the screen.

5 Results

We present results of our quantitative analysis first. We train three GP models to predict saturation, brightness and contrast for $800$ images from the test set of MIT-Adobe data-set. When compared with the parameters of expert-enhanced counterparts, we achieve RMSEs of $0.0057,0.0022,0.0037$ and correlations of $0.5359,0.5553,0.8023$ respectively, for the above three parameters. TGP gets an average RMSE of $0.0022$ but it suffers while producing an average correlation of only $0.3326$. We can see that it is relatively easier for a GP to relate the contrast to the image quality, which is intuitive since contrast variation changes the image drastically and it also makes the image look vibrant or dull. This in turn contributes most to the visual appeal of an image.

The left bar chart in Fig. 3 shows the results of second experiment. VIF between our enhancement and competing enhancements produces values which are, in most cases, less than one. Thus according to VIF metric, our approach produces better enhancements than Picasa, $k$NN-based heuristics and [25]. For third experiment, we get 32 VIF values for each image, which correspond to 32 enhanced versions generated by our approach. The GP ranker selects one, as mentioned earlier. We compute the average VIF value and its standard deviation over 31 other images. This process is repeated for all the 60 images and the VIF values are shown in the right bar chart of Fig. 3.

We now analyze the results of our subjective tests. We provide the following five metrics about our subjective test in Fig. 2. 1. we count votes gathered by our approach and by all other competing approaches bundled into one. This is a coarse measure of how much preference people have towards enhancements generated by our approach. 2. We count votes gathered by our approach and by $k$NN approach on the MIT-Adobe data-set. 3. comparison of votes gathered by our approach and by the $k$NN approach on the data-set of [25]. 4. comparing our approach against the results of [25] on their data. 5. Lastly, we compare our approach versus Picasa on the MIT-Adobe data. Fig. 2 shows all these metrics. On top of each bar, we indicate the mean and standard deviation for that particular approach and metric. For example, the second set of bars denote that for the MIT-Adobe data, our approach gathered $133$ votes against $104$ votes gathered by the $k$NN approach. The average number of votes obtained per user for our and the $k$NN approach were $8.9$ and $6.9$ with the standard deviations of $2$ and $1.6$, respectively. Fig. 2 shows that people consistently prefer our approach over other state-of-art approaches.

Fig. 4 shows some of the results obtained by ours and the approach of [25], $k$NN and Picasa’s auto-enhance tool. The first and the third row illustrate that the $k$NN approach is not always effective and sometimes may give over(under)-exposed results due to its dependence on the nearest training image parameters. The second row shows three representative versions generated by GP. We can see that the image in the fourth column is over-exposed. However, our ranking model successfully filters out that image and selects the one in the third column. In general, we observed that $k$NN can only get comparable results to Picasa and our approach if it finds a good match in the training set. Thus $k$NN is unlikely to scale to large-scale enhancement tasks.

6 Conclusion

We presented a novel approach to image enhancement using joint regression and ranking by employing GPs. We train our GP models on the pairs formed from poor, low and high-quality images. The learned GP models predict the desired parameters for a low-quality image from its features, which may produce its enhanced counterparts. We also described a strategy to traverse the parameter space without referring to the training images, which makes our approach efficient during testing. The GP prediction is defined by the covariance kernel, on which we impose two constraints. The first one enables the kernel to learn the feature dimensions responsible for making an image of higher-quality. The other constraint clusters all the enhancement parameters corresponding to a low-quality image, thereby allowing for effective parameter traversal. We perform quantitative and subjective evaluation experiments on two-data sets to assess the effectiveness of our approach, first one being the MIT-Adobe data [5] and the another one proposed in [25]. Quantitative experiments show that our predictions produce a low RMSE when compared with the ground-truth parameters of the MIT-Adobe data. The results show that people consistently prefer the enhancements produced by the proposed approach over the other state-of-art approaches.

Acknowledgement: The work was supported in part by ONR grant N00014-15-1-2344 and ARO grant W911NF1410371, and a gift from Nokia. Any opinions expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors.