Bridging the Gap Between Computational Photography and Visual Recognition

TL;DR

This paper introduces the UG^2 dataset and evaluation metrics to assess how image restoration and enhancement algorithms impact visual recognition, highlighting the early stage of integrating computational photography with recognition tasks.

Contribution

The paper presents a new large-scale benchmark dataset, novel evaluation metrics, and six innovative algorithms for joint image enhancement and recognition improvement.

Findings

Algorithms improve visual quality and recognition accuracy under challenging conditions.

Deep learning approaches outperform classic methods in joint tasks.

Early evidence suggests potential for better integration of photography and recognition.

Abstract

What is the current state-of-the-art for image restoration and enhancement applied to degraded images acquired under less than ideal circumstances? Can the application of such algorithms as a pre-processing step to improve image interpretability for manual analysis or automatic visual recognition to classify scene content? While there have been important advances in the area of computational photography to restore or enhance the visual quality of an image, the capabilities of such techniques have not always translated in a useful way to visual recognition tasks. Consequently, there is a pressing need for the development of algorithms that are designed for the joint problem of improving visual appearance and recognition, which will be an enabling factor for the deployment of visual recognition tools in many real-world scenarios. To address this, we introduce the UG^2 dataset as a large-scale benchmark composed of video imagery captured under challenging conditions, and two enhancement tasks designed to test algorithmic impact on visual quality and automatic object recognition. Furthermore, we propose a set of metrics to evaluate the joint improvement of such tasks as well as individual algorithmic advances, including a novel psychophysics-based evaluation regime for human assessment and a realistic set of quantitative measures for object recognition performance. We introduce six new algorithms for image restoration or enhancement, which were created as part of the IARPA sponsored UG^2 Challenge workshop held at CVPR 2018. Under the proposed evaluation regime, we present an in-depth analysis of these algorithms and a host of deep learning-based and classic baseline approaches. From the observed results, it is evident that we are in the early days of building a bridge between computational photography and visual recognition, leaving many opportunities for innovation in this area.

Equations

$$\phi =m_e-m_b\ge \epsilon ,\epsilon \in [-1,1]$$

$$\tilde{I}={\left(\frac{I}{V}\right)}^{\gamma }(V),$$

Full text

1 Introduction

The advantages of collecting imagery from autonomous vehicle platforms such as small UAVs are clear. Man-portable systems can be launched from safe positions to penetrate difficult or dangerous terrain, acquiring hours of video without putting human lives at risk during search and rescue operations, disaster recovery, and other scenarios where some measure of danger has traditionally been a stumbling block. Similarly, cars equipped with vision systems promise to improve road safety by more reliably reacting to hazards and other road users compared to humans. However, what remains unclear is how to automate the interpretation of what are inherently degraded images collected in such applications — a necessary measure in the face of millions of frames from individual flights or road trips. A human-in-the-loop cannot manually sift through data of this scale for actionable information in real-time. Ideally, a computer vision system would be able to identify objects and events of interest or importance, surfacing valuable data out of a massive pool of largely uninteresting or irrelevant images, even when that data has been collected under less than ideal circumstances. To build such a system, one could turn to recent machine learning breakthroughs in visual recognition, which have been enabled by access to millions of training images from the Internet [1, 2]. However, such approaches cannot be used as off-the-shelf components to assemble the system we desire, because they do not take into account artifacts unique to the operation of the sensor and optics configuration on an acquisition platform, nor are they strongly invariant to changes in weather, season, and time of day.

Whereas deep learning-based recognition algorithms can perform on par with humans on good quality images [3, 4], their performance on distorted samples is degraded. It has been observed that the presence of imaging artifacts can severely impact the recognition accuracy of state-of-the-art approaches [5, 6, 7, 8, 9, 10, 11]. Having a real-world application such as a search and rescue drone or autonomous driving system fail in the presence of ambient perturbations such as rain, haze or even motion induced blur could have unfortunate aftereffects. Consequently, developing and evaluating algorithms that can improve the object classification of images captured under less than ideal circumstances is fundamental for the implementation of visual recognition models that need to be reliable. And while one’s first inclination would be to turn to the area of computational photography for algorithms that remove corruptions or gain resolution, one must ensure that they are compatible with the recognition process itself, and do not adversely affect the feature extraction or classification processes (Fig. 1) before incorporating them into a processing pipeline that corrects and subsequently classifies images.

The computer vision renaissance we are experiencing has yielded effective algorithms that can improve the visual appearance of an image [12, 13, 14, 15, 16], but many of their enhancing capabilities do not translate well to recognition tasks as the training regime is often isolated from the visual recognition aspect of the pipeline. In fact, recent works [5, 17, 18, 19] have shown that approaches that obtain higher scores on classic quality estimation metrics (namely Peak Signal to Noise Ratio), and thus, would be expected to produce high-quality images, do not necessarily perform well at improving or even maintaining the original image classification performance. Taking this into consideration, we propose to bridge the gap between traditional image enhancement approaches and visual recognition tasks as a way to jointly increase the abilities of enhancement techniques for both scenarios.

In line with the above objective, in this work, we introduce UG2: a large-scale video benchmark for assessing image restoration and enhancement for visual recognition. It consists of a publicly available dataset (https://goo.gl/AjA6En) composed of videos captured from three difficult real-world scenarios: uncontrolled videos taken by UAVs and manned gliders, as well as controlled videos taken on the ground. Over $200,000$ annotated frames for hundreds of ImageNet classes are available. From the base dataset, different enhancement tasks can be designed to evaluate improvement in visual quality and automatic object recognition, including supporting rules that can be followed to execute such evaluations in a precise and reproducible manner. This article describes the creation of the UG2 dataset as well as the advances in visual enhancement and recognition that have been possible as a result.

Specifically, we summarize the results of the IARPA sponsored UG2 Challenge workshop held at CVPR 2018. The challenge consisted of two specific tasks defined around the UG2 dataset: (1) image restoration and enhancement to improve image quality for manual inspection, and (2) image restoration and enhancement to improve the automatic classification of objects found within individual images. The UG2 dataset contains manually annotated video imagery (including object labels and bounding boxes) with an ample variety of imaging artifacts and optical aberrations; thus it allows for the development and quantitative evaluation of image enhancement algorithms. Participants in the challenge were able to use the provided imagery and as much out-of-dataset imagery as they liked for training and validation purposes. Enhancement algorithms were then submitted for evaluation and results were revealed at the end of the competition period.

The competition resulted in six new algorithms, designed by different teams, for image restoration and enhancement in challenging image acquisition circumstances. These algorithms included strategies to dynamically estimate corruption and choose the appropriate response, the simultaneous targeting of multiple artifacts, the ability to leverage known image priors that match a candidate probe image, super-resolution techniques adapted from the area of remote sensing, and super-resolution via Generative Adversarial Networks. This was the largest concerted effort to-date to develop new approaches in computational photography supporting human preference and automatic recognition. We look at all of these algorithms in this article.

Having a good stable of existing and new restoration and enhancement algorithms is a nice start, but are any of them useful for the image analysis tasks at hand? Here we take a deeper look at the problem of scoring such algorithms. Specifically, the question of whether or not researchers have been doing the right thing when it comes to automated evaluation metrics for tasks like deconvolution, super-resolution and other forms of image artifact removal is explored. We introduce a visual psychophysics-inspired assessment regime, where human perception is the reference point, as an alternative to other forms of automatic and manual assessment that have been proposed in the literature. Using the methods and procedures of psychophysics that have been developed for the study of human vision in psychology, we can perform a more principled assessment of image improvement than just a simple A/B test, which is common in computer vision. We compare this human experiment with the recently introduced Learned Perceptual Image Patch Similarity (LPIPS) metric proposed by Zhang et al. [21]. Further, when it comes to assessing the impact of restoration and enhancement algorithms on visual recognition, we suggest that the recognition performance numbers are the only metric that one should consider. As we will see from the results, much more work is needed before practical applications can be supported.

In summary, the contributions of this article are:

• •

A new video benchmark dataset representing both ideal conditions and common aerial image artifacts, which we make available to facilitate new research and to simplify the reproducibility of experimentation.

• •

A set of protocols for the study of image enhancement and restoration for image quality improvement, as well as visual recognition. This includes a novel psychophysics-based evaluation regime for human assessment and a realistic set of quantitative measures for object recognition performance.

• •

An evaluation of the influence of problematic conditions on object recognition models including VGG16 and VGG19 [29], InceptionV3 [30], ResNet50 [31], MobileNet [32], and NASNetMobile [33].

• •

The introduction of six new algorithms for image enhancement or restoration, which were created as part of the UG2 Challenge workshop held at CVPR 2018. These algorithms are pitted against eight different classical and deep learning-based baseline algorithms from the literature on the same benchmark data.

• •

A series of recommendations on specific aspects of the problem that the field should focus its attention on so that we have a better chance at enabling scene understanding with less than ideal acquisition circumstances.

2 Related Work

Datasets. The areas of image restoration and enhancement have a long history in computational photography, with associated benchmark datasets that are mainly used for the qualitative evaluation of image appearance. These include very small test image sets such as Set5 [13] and Set14 [12], the set of blurred images introduced by Levin et al. [34], and the DIVerse 2K resolution image dataset (DIV2K) [35] designed for super-resolution benchmarking. Datasets containing more diverse scene content have been proposed including Urban100 [15] for enhancement comparisons and LIVE1 [36] for image quality assessment. While not originally designed for computational photography, the Berkeley Segmentation Dataset has been used by itself [15] and in combination with LIVE1 [37] for enhancement work. The popularity of deep learning methods has increased demand for training and testing data, which Su et al. provide as video content for deblurring work [16]. Importantly, none of these datasets were designed to combine image restoration and enhancement with recognition for a unified benchmark.

Most similar to the dataset we employ in this paper are various large-scale video surveillance datasets, especially those which provide a “fixed" overhead view of urban scenes [38, 39, 40, 41]. However, these datasets are primarily meant for other research areas (e.g., event/action understanding, video summarization, face recognition) and are ill-suited for object recognition tasks, even if they share some common imaging artifacts that impair recognition.

With respect to data collected by aerial vehicles, the VIRAT Video Dataset [42] contains “realistic, natural and challenging (in terms of its resolution, background clutter, diversity in scenes)” imagery for event recognition, while the VisDrone2018 Dataset [26] is designed for object detection and tracking. Other datasets including aerial imagery are the UCF Aerial Action Data Set [43], UCF-ARG [44], UAV123 [25], UAVDT [24], Campus [28], and the multi-purpose dataset introduced by Yao et al. [45]. However, existing datasets in this area contain a limited number of frames and object categories. Table I provides a comparison of our UG2 dataset to relevant aerial datasets. As with the computational photography datasets, none of these sets have protocols for image restoration and enhancement coupled with object recognition.

Visual Quality Enhancement. There is a wide variety of enhancement methods dealing with different kinds of artifacts, such as deblurring (where the objective is to recover a sharp version $x^{\prime}$ of a blurry image $y$ without knowledge of the blur parameters) [46, 47, 34, 48, 49, 50, 51, 52], denoising (where the goal is the restoration of an image $x$ from a corrupted observation $y=x+n$, where $n$ is assumed to be noise with variance $\sigma^{2}$) [48, 49, 53, 54], compression artifact reduction (which focuses on removing blocking artifacts, ringing effects or other lossy compression-induced degradation) [55, 56, 57, 58], reflection removal [59, 60], and super-resolution (which attempts to estimate a high-resolution image from one or more low-resolution images) [61, 62, 63, 64, 65, 66, 20, 67, 68]. Other approaches designed to deal with atmospheric perturbations include dehazing (which attempts to recover the scene radiance $J$, the global atmospheric light $A$ and the medium transmission $t$ from a hazy image $I(x)=J(x)t(x)+A(1-t(x))$) [69, 70, 71, 72, 73], and rain removal techniques [74, 75, 76, 77].

Most of these approaches are tailored to address a particular kind of visual aberration, and the presence of multiple problematic conditions in a single image might lead to the introduction of artifacts by the chosen enhancement technique. Recent work has explored the possibility of handling multiple degradation types [78, 79, 80, 81].

Visual Enhancement for Recognition. Intuitively, if an image has been corrupted, then employing restoration techniques should improve the performance of recognizing objects in the image. An early attempt at unifying a high-level task like object recognition with a low-level task like deblurring was performed by Zeiler et al. through deconvolutional networks [82, 83]. Similarly, Haris et al. [18] proposed an end-to-end super-resolution training procedure that incorporated detection loss as a training objective, obtaining superior object detection results compared to traditional super-resolution methods for a variety of conditions (including additional perturbations on the low-resolution images such as the addition of Gaussian noise).

Sajjadi et al.  [19] argue that the use of traditional metrics such as Peak Signal to Noise Ratio (PSNR), Structural Similarity Index (SSIM), or the Information Fidelity Criterion (IFC) might not reflect the performance of some models, and propose the use of object recognition performance as an evaluation metric. They observed that methods that produced images of higher perceptual quality obtained higher classification performance despite obtaining low PSNR scores. In agreement with this, Gondal et al. [84] observed the correlation of the perceptual quality of an image with its performance when processed by object recognition models. Similarly, Tahboub et al. [85] evaluate the impact of degradation caused by video compression on pedestrian detection. Other approaches have used visual recognition as a way to evaluate the performance of visual enhancement algorithms for tasks such as text deblurring [86, 87], image colorization [88], and single image super-resolution [89].

Also similar to our work, Li et al. [90] perform an in-depth analysis of diverse de-raining models where they compare them using a wide variety of metrics, including their impact on object detection. It is important to note that while similar in spirit to our work, the main purpose of their task-driven evaluation is to provide a complementary perspective on the performance of their visual enhancement method, rather than to improve machine learning-based object recognition.

While the above approaches employ object recognition in addition to visual enhancement, there are approaches designed to overlook the visual appearance of the image and instead make use of enhancement techniques to exclusively improve the object recognition performance. Sharma et al. [91] make use of dynamic enhancement filters in an end-to-end processing and classification pipeline that incorporates two loss functions (enhancement and classification). The approach focuses on improving the performance of challenging high-quality images. In contrast to this, Yim et al. [10] propose a classification architecture (comprised of a pre-processing module and a neural network model) to handle images degraded by noise. Li et al. [92] introduced a dehazing method that is concatenated with Faster R-CNN and jointly optimized as a unified pipeline. It outperforms traditional Faster R-CNN and other non-joint approaches. Singh et al. [93] propose the use of a dual directed capsule network with a dual high-resolution and targeted reconstruction loss to reconstruct very low-resolution images (16x16 pixels) in order to improve digit and face recognition.

Additional work has been undertaken in using visual enhancement techniques to improve high-level tasks such as face recognition [94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106] (through the incorporation of deblurring, super-resolution, and hallucination techniques) and person re-identification [107] algorithms for video surveillance data.

3 A Framework for Improving Object Recognition in Degraded Imagery

All images acquired in real environments are degraded in some way (Fig. 2.a). A detailed discussion of how this happens can be found in Supp. Sec. 1. The first step towards solving the problem at hand is to define a specific framework that algorithms will operate in. Ideally, the framework should be able to jointly optimize the dual tasks of image restoration and enhancement, and object recognition. The input to the system is a naturally corrupted image and the output is the corresponding improved image, optimized for classification performance. During training, a learning objective $\varphi$ can be defined that compares the reconstructed output from the system to that derived from baseline input. The result is then used to make further adjustments to the model, if needed, in order to improve performance over the baseline. This process is shown in Fig. 2.b.

For example, if $m_{e}$ represents the evaluation metric for the enhanced image, and $m_{b}$ represents the baseline metric for the corresponding unaltered image, then a possible learning objective for this setup can be defined as:

[TABLE]

where $\varepsilon$ is a threshold applied over the difference between scores obtained for the enhanced and unaltered noisy images. The metric $m$ in Eq. 1 can be the class probabilities obtained from the recognition model for the enhanced image and its corresponding unaltered image. Even when operating with probability scores, $\varepsilon$ lies between $[-1,1]$ to account for situations when the enhanced image is worse than the unaltered image, leading to worse object recognition performance (e.g., accuracy is [math] for the enhanced image and $1$ for the baseline).

The learning objective can be used alone or in conjunction with other loss functions (e.g., reconstruction loss for restoration). It can be optimized using gradient descent. If ground-truth is available for both image artifacts and objects present in images, then the learning setting is fully supervised. If only ground-truth object annotations are available, then the setting is semi-supervised (this is what is considered for the UG2 dataset). All of the novel algorithms we discuss in this paper operate within this framework.

4 A New Evaluation Regime for Image Restoration and Enhancement

To assess the interplay between restoration and enhancement and visual recognition, we designed two evaluation tasks: (1) enhancement to facilitate manual inspection, where algorithms produce enhanced images to facilitate human assessment, and (2) enhancement to improve object recognition, where algorithms produce enhanced images to improve object classification by state-of-the-art neural networks.

4.1 Enhancement to Facilitate Manual Inspection

The first task is an evaluation of the qualitative enhancement of images. Through this task, we wish to answer two questions: Did the algorithm produce an enhancement that agrees with human perceptual judgment? And to what extent was the enhancement an improvement or deterioration? Widely used metrics such as SSIM have attempted to answer these questions by estimating human perception but have often failed to accurately imitate the nuances of human visual perception, and at times have caused algorithms to deteriorate perceptual quality [21].

While prior work has successfully adapted psychophysical methods from psychology as a means to directly study perceptual quality, these methods have been primarily posed as real-vs-fake tests [108]. With respect to qualitative enhancement, these real-vs-fake methods can only indicate if an enhancement has caused enough alteration to cause humans to cross the threshold of perception, and provides little help in answering the two questions we are interested in for this task. Zhang et al. [21] came close to answering these questions when they proposed LPIPS for evaluating perceptual similarity. However, this metric lacks the ability to measure whether the enhancement was an improvement or a deterioration (see the analysis in Sec. 7).

In light of this, for our task we propose a new procedure, grounded in psychophysics, to evaluate the visual enhancement of images by answering both of our posed questions. The procedure for assessing image quality enhancement is a non-forced-choice procedure that allows us to take measurements of both the threshold of perceived change and the suprathresholds, which represent the degree of perceived change [109]. Specifically, we employ a bipolar labeled Likert Scale to estimate the amount of improvement or deterioration the observer perceives, once the threshold has been crossed. The complete procedure is as follows.

An observer is presented with an image $x$ positioned on the left-hand side of a screen and the output of the enhancement algorithm $y$ on the right. The observer is informed that $x$ is the original image and $y$ is the enhanced image. Below the image pair, five labels are provided and the observer is asked to select the label that most applies (see Fig. 3 for labels and layout). To capture as much of the underlying complexities in human judgment as possible, no criteria is provided for making a selection. To reduce any dependence on the subsystems in the visual cortex that specialize in memory, the pair of images is displayed until the observer selects their perceived label. An observer is given unlimited time and is informed that providing accurate labels is most important. For images larger than $480\times 480$ pixels, the observer has the option to enlarge the images and examine them in finer detail.

The label that is selected by the observer is then converted to an assigned ordinal value $1-5$ where $[1,3)$ and $(3,5]$ are the suprathreshold measurements of improvement and deterioration, respectively. A rating of $3$ is the superficial threshold imposed on the observer, which indicates that the enhancement was imperceptible. In our proposed procedure, there is no notion of accuracy in the measurement of qualitative enhancement, as it is left entirely up to the observer’s subjective discretion. However, when there are $\geq 2$ sampled observers, the perception of quality to the average observer can be estimated to provide a reliable metric for the evaluation of qualitative enhancement. We verified this holds true even when $x$ and $y$ are swapped (i.e., responses are symmetric).

To perform a large scale evaluation, we used Amazon’s Mechanical Turk (AMT) service, which is widely deployed for many related tasks in computer vision [21, 110, 1]. AMT allows a Requester (i.e., researcher) to hire Workers to perform a task for payment. Our task was for a Worker to participate in the rating procedure for 100 image pairs. An additional three sentinel images pairs were given to ensure that the Worker was actively participating in the rating procedure. The ratings of Workers who failed to correctly respond to at least two of the three sentinel image pairs were discarded. In total, we had over $600$ Workers rating each image enhancement approximately $20$ times. Out of that pool, $\sim 85.4\%$ successfully classified a majority of the sentinel image pairs (the ratings provided by the remaining Workers were discarded). See Sec. 7 for results and analysis.

4.2 Enhancement to Improve Object Recognition

The second task is an evaluation of the performance improvement given by enhanced images when used as input to state-of-the-art image classification networks. When considering a fixed dataset for training (as in Fig. 2.b), the evaluation protocol allows for the use of some within dataset training data (the training data provided by the UG2 dataset, described below, contains frame-level annotations of the object classes of interest), and as much out of dataset data as needed for training and validation purposes. In order to establish good baselines for classification performance before and after the application of image enhancement and restoration algorithms, this task makes use of a selection of deep learning approaches to recognize annotated objects and then scores results based on the classification accuracy. The Keras [111] versions of the pre-trained networks VGG16 and VGG19 [29], InceptionV3 [30], and ResNet50 [31] are used for this purpose. We also look at two lightweight mobile networks: MobileNet [32] and NASNetMobile [33].

Each candidate algorithm is treated as an image pre-processing step to prepare sequestered test images to be submitted to all six networks. After pre-processing, the objects of interest are cropped out of the images based on verified ground-truth coordinates. The cropped images are then used as input to the networks. Algorithms are evaluated based on any improvement observed over the baseline classification result (i.e., the classification scores of the un-altered test images). The work flow of this evaluation pipeline is shown in Fig. 2.c. To avoid introducing further artifacts due to down-sampling, algorithms are required to have consistent input and output frame sizes.

4.2.1 Classification Metrics

The networks used for the classification task return a list of the ImageNet synsets (ImageNet provides images for “synsets" or “synonym sets" of words or phrases that describe a concept in WordNet [112]) and the probability of the object belonging to each of the synset classes. However, (as will be discussed in Sec. 5), in many cases it is impossible to provide fine-grained labeling for the annotated objects. Consequently, most of the super-classes in our dataset are composed of more than one ImageNet synset. That is, each annotated image $i$ has a single super-class label $L_{i}$ which is defined by a set of ImageNet synsets $L_{i}=\{s_{1},...,s_{n}\}$.

To measure accuracy, we observe the number of correctly identified synsets in the top-five predictions made by each pre-trained network. A prediction is considered to be correct if its synset belongs to the set of synsets in the ground-truth super-class label. We use two metrics for this. The first measures the rate of achieving at least one correctly classified synset class (M1). In other words, for a super-class label $L_{i}=\{s_{1},...,s_{n}\}$, a network is able to place at least one correctly classified synset in the top-five predictions. The second measures the rate of placing all the correct synset classes in the super-class label synset set (M2). For example, for a super-class label $L_{i}=\{s_{1},s_{2},s_{3}\}$, a network is able to place three correct synsets in the top-five predictions.

4.2.2 Scoring

Each image enhancement or restoration algorithm’s performance on the classification task is then calculated by applying one of the two metrics defined above for each of the four networks and each collection within the UG2 dataset. This results in up to $18$ scores for each metric (i.e., M1 or M2 scores from VGG16, VGG19, Inception, ResNet, MobileNet, and NASNetMobile for the UAV, Glider, and Ground collections). For the image enhancement and restoration algorithms we consider in this article, each is ranked against all other algorithms based on these scores. A score for an enhancement algorithm is considered “valid" if it was higher than that of the scores obtained by evaluating the classification performance of the un-altered images. In other words, we only consider a score valid if it improves upon the baseline classification task of classifying the original images. The number of valid scores in which an algorithm excels over the others being evaluated is then counted as its score in points for the task (for a maximum of $36$ points, achievable if an algorithm obtained the highest improvement — compared to all other competitors — in all possible configurations).

5 The UG2 Dataset

As a basis for the evaluation regime described above in Sec. 4, we collected a new dataset called UG2 (UAV, Glider, and Ground). The UG2 dataset contains videos from challenging imaging scenarios containing mobile airborne cameras and ground-based captures. While we provide frame-level annotations for the purpose of object classification, the annotations can be re-purposed for other high-level tasks such as object detection and tracking [113]. The video files are provided to encourage further annotation for other vision tasks. The training and test datasets employed in the evaluation are composed of annotated frames from three different video collections. The annotations provide bounding boxes establishing object regions and classes, which were manually annotated using the VATIC tool for video annotation [114]. For running classification experiments the objects were cropped from the frames in a square region of at least $224\times 224$ pixels (a common input size for many deep learning-based recognition models), using the annotations as a guide. For details on annotation, see Supp. Sec. 2.

Each annotation in the dataset indicates the position, scale, visibility, and super-class of an object in a video. The need for high-level classes (super-classes) arises from the challenge of performing fine-grained object recognition using aerial collections, which have high variability in both object scale and rotation. These two factors make it difficult to differentiate some of the more fine-grained ImageNet categories. For example, while it may be easy to recognize a car from an aerial picture taken from hundreds (if not thousands) of feet above the ground, it might be impossible to determine whether that car is a taxi, a jeep or a sports car. Thus we defined super-classes that encompass multiple visually similar ImageNet synsets, as well as evaluation metrics that allow for a coarse-grained classification evaluation of such cases (see Sec. 4.2.1). The three different video collections consist of:

(1) UAV Video Collection: Composed of clips recorded from small UAVs in both rural and urban areas, the videos in this collection are open-source content tagged with a Creative Commons license, obtained from YouTube (Supp. Fig. 3.a). Because of the source, they have different video resolutions (from $600\times 400$ to $3840\times 2026$), objects of interest sizes (cropped objects with sizes ranging from $224\times 224$ to $800\times 800$), and frame rates (from $12$ to $59$ FPS). This collection has distortions such as glare/lens flare, compression artifacts, occlusion, over/under exposure, camera shaking, sensor noise, motion blur, and fish-eye lens distortion. Videos with problematic scene/weather conditions such as night/low light video, fog, cloudy conditions and occlusion due to snowfall are also included.

(2) Glider Video Collection: Consists of videos recorded by licensed pilots of fixed-wing gliders in both rural and urban areas (Supp. Fig. 3.b). The videos have frame rates ranging from $25$ to $50$ FPS, objects of interest sizes ranging from $224\times 224$ to $900\times 900$, and different types of compression such as MTS, MP4, and MOV. The videos mostly present imagery taken from thousands of feet above the ground, further increasing the difficulty of object recognition. Additionally, the scenes contain artifacts such as motion blur, camera shaking, noise, occlusion (which in some cases is pervasive throughout the videos, showcasing parts of the glider that partially occlude the objects of interest), glare/lens flare, over/under exposure, interlacing, and fish-eye lens distortion. Videos with problematic weather conditions such as fog, clouds, and rain are also present.

(3) Ground Video Collection: In order to provide some ground-truth with respect to problematic image conditions, this collection contains videos captured at ground level with intentionally induced artifacts (Supp. Fig. 3.c). These videos capture static objects (e.g., flower pots, buildings) at a wide range of distances ($30$, $40$, $50$, $60$, $70$, $100$, $150$, and $200$ft), and motion blur induced by an orbital shaker to generate horizontal movement at different rotations per minute ($120$rpm, $140$rpm, $160$rpm, and $180$rpm). Additionally, this collection includes videos under different weather conditions (sun, clouds, rain, snow) that can affect object recognition. We used a Sony Bloggie hand-held camera (with $1280\times 720$ resolution and a frame rate of 60 FPS) and a GoPro Hero 4 (with $1920\times 1080$ resolution and a frame rate of 30 FPS), whose fish-eye lens introduced further distortion. Furthermore, we provide an additional class of videos (resolution-chart) showcasing a $9\times 11$ inch $9\times 9$ checkerboard grid exhibiting all the aforementioned distances at all intervals of rotation. The motivation for including this additional class is to provide a reference for camera calibration and to aid participants in finding the distortion measures of the cameras used.

Training Dataset. The training dataset is composed of $629$ videos with $3,535,382$ frames, representing $228$ ImageNet [1] classes extracted from annotated frames from the three different video collections. These classes are further categorized into $37$ super-classes encompassing visually similar ImageNet categories and two additional classes for pedestrian and resolution chart images. The dataset contains a subset of $152,083$ object-level annotated frames where $160,160$ objects are fully visible (out of $200,694$ total annotated frames) and the videos are tagged to indicate problematic conditions. Table II summarizes this dataset.

Testing Dataset. The testing dataset is composed of $55$ videos with frame-level annotations. Out of the annotated frames, $8,910$ disjoint frames were selected among the three different video collections, from which we extracted $9,187$ objects. These objects are further categorized into $42$ super-classes encompassing visually similar ImageNet categories. While most of the super-classes in the testing dataset overlap with those in the training dataset, there are some classes unique to each. Table III summarizes this dataset.

6 Novel and Baseline Algorithms

Six competitive teams participated in the 2018 UG2 Workshop held at CVPR, each submitting a novel approach for image restoration and enhancement meant to address the evaluation tasks we described in Sec. 4. In addition, we assessed eight different classical and deep learning-based baseline algorithms from the literature.

6.1 Challenge Workshop Entries

The six participating teams were Honeywell ACST, Northwestern University, Texas A&M and Peking University, National Tsing Hua University, Johns Hopkins University, and Noblis. Each team had a unique take on the problem, with an approach designed for one or both of the tasks.

6.1.1 Camera and Conditions-Relevant Enhancements (CCRE)

Honeywell ACST’s algorithmic pipeline was motivated by a desire to closely target image enhancements based on image quality assessment. Of the wide range of image enhancement techniques, there is a smaller subset of enhancements which may be useful for a particular image. To find this subset, the CCRE pipeline considers the intersection of camera-relevant enhancements with conditions-relevant enhancements. Examples of camera-relevant enhancements include de-interlacing, rolling shutter removal (both depending on the sensor hardware), and de-vignetting (for fisheye lenses). Example conditions-relevant enhancements include de-hazing (when imaging distant objects outdoors) and raindrop removal. To choose among the enhancements relevant to various environmental conditions and the camera hardware, CCRE makes use of defect-specific detectors (Supp. Fig. 4) and takes $\sim 12$ seconds to process each image.

This approach, however, requires a measure of manual tuning. For the evaluation task targeting human vision-based image quality assessment, manual inspection revealed severe interlacing in the glider set. Thus a simple interlacing detector was designed to separate each frame into two fields (comprised of the even and odd image rows, respectively) and compute the horizontal shift needed to register the two. If that horizontal shift was greater than $0.16$ pixels, then the image was deemed interlaced, and de-interlacing was performed by linearly interpolating the rows to restore the full resolution of one of the fields.

For the evaluation task targeting automated object classification, de-interlacing is also performed with the expectation that the edge-type features learned by the VGG network will be impacted by jagged edges from interlacing artifacts. Beyond this, a camera and conditions assessment is partially automated using a file analysis heuristic to determine which of the collections a given video frame came from. While interlacing was the largest problem with the glider images, the ground and UAV collections were degraded by compression artifacts. Video frames from those collections were processed with the Fast Artifact Reduction CNN [58].

6.1.2 Multiple Artifact Removal CNN (MA-CNN)

The Northwestern team focused their attention on three major causes of artifacts in an image: (1) motion blur, (2) de-focus blur and (3) compression algorithms. They observed that in general, traditional algorithms address inverse problems in imaging via a two-step procedure: first by applying proximal algorithms to enforce measurement constraints and then by applying natural image priors (sometimes denoisers) on the resulting output [115, 116]. This process takes about 1 second per image. Recent trends in inverse imaging algorithms have focused on developing a single algorithm or network to address multiple imaging artifacts [117]. These networks are alternately applied to denoise and deblur the image. Building on the principle of using image quality as prior knowledge, the MA-CNN learning-based approach was developed to remove multiple artifacts in an image. A training dataset was created by introducing motion, de-focus and compression artifacts into $126,000$ images from ImageNet. The motion-blur was introduced by using a kernel with a fixed length $l$ and random direction ${\theta}$ for each of the images in the training dataset. The defocus blur was introduced by using a Gaussian kernel with a fixed standard variance ${\sigma}$. The parameters {$l,{\sigma}$} were tuned to create a perceptually improved result.

MA-CNN is a fully convolutional network architecture with residual skip connections to generate the enhanced image (Supp. Fig. 5). To achieve better visual quality, a perceptual loss function that uses the first four convolutional layers of a pre-trained VGG-16 network is incorporated.

By default, the output of the MA-CNN contains checkerboard artifacts. Since these checkerboard artifacts are periodic, they can be removed by suppressing the corresponding frequencies in the Fourier-domain. Moreover, all images (of the same size) generated with the network have artifacts in a similar region in the Fourier domain. For images of differing sizes, the distance of the center of the artifact from the origin is proportional to its size.

6.1.3 Cascaded Degradation Removal Modules (CDRM)

The Texas A&M team observed that independently removing any single type of degradation could, in fact, undermine performance in the object recognition evaluation task since other degradations were not simultaneously considered and those artifacts might be amplified during this process. Consequently, they proposed a pipeline that consists of sequentially cascaded degradation removal modules to improve recognition. Further, they observed that different collections within the UG2 dataset had different degradation characteristics. As such, they proposed to first identify the incoming images as belonging to one of the three collections as a form of quality estimation, and then deploy a specific processing model for each collection. The entire pipeline process an image in $\sim 14$ seconds. In their model, they adopted six different enhancement modules (Supp. Fig. 6).

(1) Histogram Equalization balances the distribution of pixel intensities and increases the global contrast of images. To do this, Contrast Limited Adaptive Histogram Equalization (CLAHE) is adopted [118]. The image is partitioned into regions and the histogram of the intensities in each is mapped to a more balanced distribution. As the method is applied at the region level, it is more robust to locally strong over-/under-exposures and can preserve edges better. (2) Given that removing blur effects is widely found to be helpful in fast-moving aerial cameras, and/or in low light filming conditions, Deblur GAN [119] is employed as an enhancement module in which, with adversarial training, the generator in the network is able to transform a blurred image to a visually sharper one. (3) Recurrent Residual Net for Super-Resolution was previously proposed in [120]. Due to the large distance between objects and aerial cameras, low-resolution is a bottleneck for recognizing most objects from UAV photos. This model is a recurrent residual convolutional neural network consisting of six layers and skip-connections. (4) Deblocking Net [121] is an auto-encoder-based neural network with dilation convolutions to remove blocking effects in videos, which was fine-tuned using the VGG-19 perceptual loss function, after training using JPEG-compressed images. Since lossy video coding for on-board sensors introduced blocking effects in many frames, the adoption of the deblocking net was found to suppress visual artifacts. (5) RED-Net [122] is trained to restore multiple mixed degradations, including noise and low-resolution together. Images with various noise levels and scale levels are used for training. The network can improve the overall quality of images. (6) HDR-Net [123] can further enhance the contrast of images to improve the quality for machine and human analysis. This network learns to produce a set of affine transformations in bilateral space to enhance the image while preserving sharp edges.

6.1.4 Tone Mapping Deep Image Prior (TM-DIP)

The main idea of the National Tsing Hua University team’s approach was to derive deep image priors for enhancing images that are captured from a specific scene with certain poor imaging conditions, such as the UG2 collections. They consider the setting that the high-quality counterparts of the poor-quality input images are unavailable, and hence it is not possible to collect pairwise input/output data for end-to-end supervised training to learn how to recover the sharp images from blurry ones. The method of Deep Image Prior presented by Ulyanov et al. [124] can reconstruct images without using information from ground-truth sharp images. However, it usually takes several minutes to produce a prior image by training an individual network for each image. Thus a new method was designed to replace the per-image prior model of [124] by a generic prior network. This idea is feasible since images taken in the same setting, e.g., the UG2 videos, often share similar features. It is not necessary to have a distinct prior model for each image. One can learn a generic prior network that takes every image as input and generates its corresponding prior as output.

At training time, the method from [124] is used to generate image pairs $\{(I,V)\}$ for training a generic prior network, where $I$ is an original input image and $V$ is its corresponding prior image. The generic prior network adopts an encoder-decoder architecture with skip connections as in [125]. At inference time, given a new image, its corresponding prior image is efficiently obtained from the learned generic prior network, with tone mapping then applied to enhance the details.

It was observed that the prior images obtained by the learned generic prior network usually preserve the significant structure of the input images but exhibit fewer details. This observation, therefore, led to a different line of thought on the image enhancement problem. By comparing the prior image with the original input image, details for enhancement may be extracted. Thus, the tone mapping technique presented in [126] was used to enhance the details:

[TABLE]

where the ratio $I/V$ can be considered the details, and $\gamma$ is a factor for adjusting the degree of detail-enhancement. With the tone-mapping function in Eq. (2), the local details are detached from the input image, and the factor $\gamma$ is subsequently adjusted to obtain an enhanced image $\widetilde{I}$.

6.1.5 Satellite Images Super-Resolution (SSR)

The team from Johns Hopkins University proposed a neural network-based approach to apply super-resolution on images. They trained their model on satellite imagery, which has an abundance of detailed features. Their network is fully convolutional, and takes as input an image of any resolution and outputs an image that is exactly double the original input in width and height in $\sim 3$ seconds.

The network is constrained to $32\times 32$ pixel patches of the image with an “apron" of $2$ pixels for an overlap. This results in a $64\times 64$ output where the outer $4$ pixels are ignored, as they are the apron — they mirror the edge to “pad" the image. These segments are then stitched together to form the final image. The network consists of five convolutional layers (Supp. Table 1). Most of the network’s layers contain $1\times 1$ kernels, and hence are just convolutionalized fully connected layers. This network structure is appropriate for a super-resolution task because it can be equated to a regression problem where the input is a $25$ dimension ($5\times 5$) vector leading to a $4$ dimensional ($2\times 2$) vector. The first convolutional layer is necessary to maintain the spatial relationships of the visual features through the $5\times 5$ kernel.

The SpaceNet dataset[127] is used to train this network and is derived from satellite-based images. Images were downsampled and paired with the originals. Training took place for $20$ epochs using an L2 + L1 combined loss and the Adam optimizer in Keras/Tensorflow [111].

6.1.6 Style-Transfer Enhancement Using GANs (ST-GAN)

Noblis attempted a style-transfer approach for improving the quality of the UG2 imagery. Since the classification networks used in the UG2 evaluation protocol were all trained on ImageNet, a CycleGAN [128] variant was trained to translate between the UAV and drone collections and ImageNet, using LSGAN [129] losses. The architecture was based on the original CycleGAN paper, with modified generators adding skip connections between same spatial resolution convolutional layers on both sides of the residual blocks (in essence a U-Net [125] style network), which appeared to improve retention of details in the output images. This algorithm was able to process each image in less than $4$ seconds. The UG2 to ImageNet generator was also made to perform $4\times$ upscaling (by adding two $0.5$ strided convolutional layers after the first convolutional layer), and it was also made to perform $4\times$ downscaling (by adding two stride $2$ convolutional layers after the first convolutional layer). The discriminators were left unmodified. The networks were trained using $128\times 128$ patches selected from the UG2 images, and ImageNet images cropped and resized to $512\times 512$. UG2 patches were selected by randomly sampling regions around the ground-truth annotation bounding boxes to avoid accidentally sampling flat-colored patches.

However, several problems were initially encountered when optimizing the network. Optimization would fail outright, unless it employed some form of normalization. Adding the identity mapping losses (i.e., loss terms for $G(X)-X$, and $F(Y)-Y$) discussed in the original CycleGAN paper proved effective in avoiding these kinds of failures. Since the UG2 evaluation protocol specifies the enhancement of full video frames, either a larger input to the generator must be used (which seemed feasible considering a fully-convolutional architecture), or the input image must be divided into tiles. To address this, an operation that performed normalization independently down the channels of each pixel was used. This stabilized convergence and did not cause problems when tiling out very large images.

6.2 Baseline Algorithms

The following algorithms serve as canonical references or baselines against which the algorithms in Sec. 6.1 were tested. We used both classical methods and state-of-the-art deep learning-based methods for image interpolation [130], super-resolution [20, 67], and deblurring [131, 132]

Classical Methods. For image enhancement, we used three different interpolation methods (bilinear, bicubic and nearest neighbor) [130] and a single restoration algorithm (blind deconvolution [131]). The interpolation algorithms attempt to obtain a high-resolution image by up-sampling the source low-resolution image and by providing the best approximation of a pixel’s color and intensity values depending on the nearby pixels. Since they do not need any prior training, they can be directly applied to any image. Nearest neighbor interpolation uses a weighted average of the nearby translated pixel values in order to calculate the output pixel value. Bilinear interpolation increases the number of translated pixel values to two and bicubic interpolation increases it to four. Different from image enhancement, in image restoration, the degradation, which is the product of motion or depth variation from the object or the camera, is modeled. The blind deconvolution algorithm can be used effectively when no information about the degradation (blur and noise) is known [133]. The algorithm restores the image and the point-spread function (PSF) simultaneously. We used Matlab’s blind deconvolution algorithm, which deconvolves the image using the maximum likelihood algorithm, with a $3\times 3$ array of $1$s as the initial PSF.

Deep Learning-Based Methods. With respect to state-of-the-art deep learning-based super-resolution algorithms, we tested the Super-Resolution Convolutional Neural Network (SRCNN) [20] and Very Deep Super Resolution (VDSR) [67]. The SRCNN method employs a feed-forward deep CNN to learn an end-to-end mapping between low-resolution and high-resolution images. The network was trained on $5$ million “sub-images" generated from $395,909$ images of the ILSVRC 2013 ImageNet detection training partition [1]. The VDSR algorithm [67] outperforms SRCNN by employing a deeper CNN inspired by the VGG architecture [29] and decreases training iterations and time by employing residual learning with a very high learning rate for faster convergence. Unlike SRCNN, the network is capable of handling different scale factors.

With respect to deep learning-based image restoration algorithms, we tested Deep Dynamic Scene Deblurring [132], which was designed to address camera shake blur. However, the results presented in [132] indicated that this method can obtain good results for other types of blur. The algorithm employs a CNN that was trained with video frames containing synthesized motion blur such that it receives a stack of neighboring frames and returns a deblurred frame. The algorithm allows for three types of frame-to-frame alignment: no alignment, optical flow alignment, and homography alignment. For our experiments, we used optical flow alignment, which was reported to have the best performance with this algorithm. We had originally evaluated an additional video deblurring algorithm proposed by Su et al. [16]. However, this algorithm employs information across multiple consecutive frames to perform its deblurring operation. Given that the training and testing partitions of the UG2 dataset consist of disjoint video frames, we omitted this method to provide a fair comparison.

With respect to image enhancement specifically to improve classification, we tested the recently released algorithm by Sharma et al. [91]. This approach learns a dynamic image enhancement network with the overall goal to improve classification, but not necessarily the human perception of the image. The proposed architecture enhances image features selectively, in such a way that the enhanced features provide a valuable improvement for a classification task. High quality (i.e., free of visual artifacts) images are used to train a CNN to learn a configuration of enhancement filters that can be applied to an input image to yield an enhanced version that provides better classification performance.

7 Results & Analysis

In the following analysis, we review the results that came out of the UG2 Workshop held at CVPR 2018, and discuss additional results from the slate of baseline algorithms.

7.1 Enhancement to Facilitate Manual Inspection (UG2 Evaluation Task 1)

There is a recent trend to use deep features, as measured by the difference in activations from the higher convolutional layers of a pre-trained network for the original and reconstructed images, as a perceptual metric — the motivation being deep features somewhat mimic human perception. Zhang et al. [21] evaluate the usability of these deep features as a measure of human perception. Their goal is to find a perceptual distance metric that resembles human judgment. The outcome of their work is the LPIPS metric, which measures the perceptual similarity between two images ranging from [math], meaning exactly similar, to $1$, which is equivalent to two completely dissimilar images. Here we compare this metric (i.e., similarity between the original image and the output of the evaluated enhancement algorithms) directly to human perception, which we argue is the better reference point for such assessments.

We used the most current version (v.0.1) of the LPIPS metric with a pre-trained, linearly calibrated AlexNet model. As can be observed in Fig. 4, the four novel algorithms that were submitted by participants for the first UG2 evaluation task have very heterogeneous effects on different images of the dataset, with LPIPS scores ranging all the way from [math] (no perceptual dissimilarity) to $0.4$ (moderate dissimilarity). This effect is accentuated for the images in the UAV Collection (Fig. 4a), which yields more variance in LPIPS scores, whereas LPIPS scores for the remaining two collections (Fig. 4b, 4c) remained between [math] and $0.15$ for most of these algorithms. We observed a similar effect for some of our baseline algorithms (Fig. 5a), particularly for Blind Deconvolution (BD), which sharpened images, but also amplified artifacts that were already present.

The images human raters considered as having a high-level of improvement tended to also have low LPIPS scores (usually between [math] and $0.15$), while images with higher LPIPS scores tended to be rated negatively by human observers (see Figs. 4 and 5a). Similar behavior was observed by Zu et al. [134]. They suggest that high LPIPS scores might indicate the presence of unnatural looking images. Contrasting the results of the two best participant and baseline algorithms (Fig. 5b), we observed that for the baselines and CCRE the LPIPS and human rating distributions were more tightly grouped than MA-CNN. Nevertheless, their changes to images tended to be considered small by both human raters and the LPIPS metric, with a user rating closer to $3.0$ (no change) and LPIPS scores between [math] and $0.1$. In contrast, changes induced by the MA-CNN method reached extremes of $0.36$ and $2.35$ for LPIPS score and human rating respectively, flagging the presence of very noticeable, but in some cases detrimental, changes. This is a further constraint of the LPIPS metric.

It is important to note that while we calculated the mean user rating of all the workshop participant submissions and baseline algorithms, it was not possible to obtain the LPIPS scores for any of the super-resolution approaches. This would have required us to down-sample enhanced images to be of the same size as that of the original images, which would have negated the improvement of such methods.

Focusing just on the image improvement / deterioration as perceived by human raters, we can turn to Fig. 6a for the performance of all algorithms, including the super-resolution approaches, submitted by each team. It is important to note that while most of the algorithms tended to improve the visual quality of the images they were presented with, a large fraction of the images they enhanced tended to have an average score between $3$ and $3.25$. In such cases, humans could not detect any meaningful change.

The best performing algorithm submitted for this task, CCRE, was able to improve the visual quality of $126$ images, even though the algorithm did not appear to perform any significant changes to most of the images ($144$ images had a rating between $3$ and $3.25$). The enhancement applied was considered a subtle improvement in most scenarios: $114$ images had a score between $2.75$ and $3$, with the remaining $12$ having a higher improvement score between $2.5$ and $2.75$. However, only $10\%$ of the modified images were considered to degrade the image quality, and even then they had a rating of between $3.25$ and $3.5$, which means that the degradation was very small.

As mentioned previously, the visual changes generated by the runner-up enhancement algorithm MA-CNN were more explicit than those present in CCRE. While the number of images that were considered to be improved was smaller ($59$ improved images), $13$ of them were between the range of $2.25$ and $2.75$, indicating a good measure of higher visual quality. Nevertheless, the sharp changes introduced by this algorithm also seemed to increase the perceived degradation on a larger portion of the images, with almost $25\%$ having a score between $3.5$ and $3.75$ (thus indicating a significant deterioration of the image quality).

With respect to the baseline algorithms, we observed a less dramatic perception of quality degradation. Given that most of the algorithms tested were focused on enhancing the image resolution (by performing image interpolation or super-resolution), they were more prone to perform very subtle changes in the structure of the image. This is reflected in Fig. 6b. Fig. 6c shows a side by side comparison of the two best baselines (VDSR and Nearest Neighbor interpolation) and the two best performing participant submissions.

7.2 Evaluation of Object Recognition Performance (UG2 Evaluation Task 2)

The results for this evaluation task fill a gap in our knowledge on the effects certain restoration and enhancement algorithms have on visual recognition. In the following section, we introduce results for various algorithms ranging from image interpolation to deep learning-based approaches that are designed solely for the purpose of improving the “perceived” visual quality of an image. We then compare the performance of such techniques to the enhancement approaches intentionally designed to improve object classification.

For this evaluation task, the participants were expected to provide enhancement techniques catering to machine vision rather than human perception. Supp. Table 2 and Fig. 7 depict the baseline classification results for the UG2 training and testing datasets, without any restoration or enhancement algorithm applied, at rank 5.

Given the very poor quality of its videos, the UAV Collection proved to be the most challenging for all networks in terms of object classification, leading to the lowest classification performance out of the three collections. While the Glider Collection shares similar problematic conditions with the UAV Collection, the images in this collection lead to a higher classification rate than those in the UAV Collection in terms of identifying at least one correctly classified synset class (metric M1). This improvement might be caused by the limited degree of movement of the gliders, since it ensures that the displacement between frames was kept more stable over time, as well as a higher recording quality (taking into consideration the camera weight limitations present in small UAVs are no longer a limiting factor for this collection’s videos). The controlled Ground Collection yielded the highest classification rates, which, in an absolute sense, are still low (the highest classification rate being $65.23\%$ for metric M1 and $52.06\%$ for metric M2 for the testing dataset; see Supp. Table 2). The participants of the UG2 workshop were expected to develop algorithms to improve upon the baseline scores.

Additionally, we were interested in assessing the performance of the participants’ enhancement methods when applied to light-weight classification models. As such, we employed two additional networks designed for mobile and embedded vision applications: MobileNet (v2) and NASNetMobile. The performance follows similar patterns to the ones obtained from larger classification models: (1) the UAV Collection remains the most challenging for both networks, while the Ground Collection obtained the best results out of the three collections; (2) the sharp difference in the performance on the two metrics remained. However, both mobile networks obtained poor results (when compared to the larger networks) when classifying the UG2 data. It is likely that these models are specialized towards ImageNet-like images, and have more trouble generalizing to new data without further fine-tuning.

While there are some correlations between the improvement of visual quality as perceived by humans and high-level tasks such as object classification performed by networks, research by Sharma et al. [91] suggests that image enhancement focused on improving image characteristics valuable for object recognition can lead to an increase in classification performance. Thus we tried such a technique as an initial experiment and compared it to the baseline results. Fig. 8 shows the performance of the five enhancement filters proposed by Sharma et al. on the UG2 dataset. It is important to note that said filters were the un-altered filters Sharma et al. trained making use of good quality images. This is because their focus was on improving the classification performance of images with few existing perturbations. As such, the effect on improving highly corrupted images is much different from that obtained on a standard dataset of images crawled from the web. The results in Fig. 8 establish that even existing deep learning networks designed for this task cannot achieve good classification rates for UG2 due to the domain shift in training.

As can be observed in Supp. Table 3 and Fig. 9, the four novel algorithms submitted by participants for the second UG2 evaluation task excelled in the processing of certain collections while falling short in others. Most of the submitted algorithms were able to improve the classification performance of the images in the Ground Collection, but they struggled in improving the classification for the aerial collections, whose scenes tend to have a higher degree of variability than those present in the Ground Collection. Only the CCRE algorithm was able to improve the performance of one of the metrics for the UAV Collection (the M2 metric for the ResNet network, with a $0.20\%$ improvement over the baseline). The MA-CNN algorithm was able to improve two of the Glider Collection metrics (the M2 metric for the VGG16 and VGG19 networks with $2.23\%$ and $3.10\%$ improvement respectively over the baselines). For the most part, the algorithms tended to improve the metrics for the Ground Collection, with the highest classification improvement being provided by the CDRM method, with an improvement of $5.30\%$ and $5.21\%$ over the baselines for the Inception M1 and M2 metrics.

Further along these lines, while both metrics saw some improvement, the M2 metric benefited the most from these enhancement algorithms. This behavior is more pronounced when examined in the context of classic vs. state-of-the-art algorithms (Supp. Fig. 7). While the baseline enhancement algorithms had moderate improvements in both metrics, the participant algorithms seemed to favor the M2 metric over the M1 metric. For example, while the highest improvement (on the Ground Collection) for the VGG19 network in M1 was $0.03\%$, the improvement for the same network in M2 was $2.95\%$. This leads us to conclude that the effect these algorithms have on automatic object recognition would be that of increasing within-class classification certainty. In other words, they would make an object belonging to a super-class $L_{i}=\{s_{1},...,s_{n}\}$ become a better representation of such class features, such that the networks are able to detect more members of that class in their top 5 predictions.

8 Discussion

The area of computational photography has a long-standing history of generating algorithms to improve image quality. However, the metrics those algorithms tend to optimize do not correspond to human perception. And as we found out, this makes them unreliable for the task of object recognition. The UG2 dataset paves the way for this new research through the introduction of a benchmark that is well matched to the problem area. The first iteration of the challenge workshop making use of this dataset saw the participation of teams from around the globe and introduced six unique algorithms to bridge the gap between computational photography and recognition, which we have described in this article. As noted by some participants and in accordance with our initial results, the problem is still not solved — improving image quality using existing techniques does not necessarily solve the recognition problem.

The results of our experiments led to some surprises. Even though the restoration and enhancement algorithms tended to improve the classification results for the diverse imagery included in our dataset, no approach was able to uniformly improve the results for all of the candidate networks. Moreover, in some cases, performance degraded after image processing, particularly for frames with higher amounts of image aberrations. This highlights the often single focus nature of image enhancement algorithms, which tend to specialize in removing a specific kind of artifact from an otherwise good quality image, which might not always be the present. Some of the algorithmic advancements (e.g., MA-CNN and CDRM) developed as a product of this challenge seek to address the problem by incorporating techniques such as deblurring, denoising, deblocking, and super-resolution into a single pre-processing pipeline. This practice pointed out the fact that while the individual implementation of some of these techniques might be detrimental to the visual quality or visual recognition task, when applied in conjunction with other enhancement techniques their effect turned out to be beneficial for both of these objectives.

We also found out that image quality is a subjective assessment and better left to humans who are physiologically tuned to notice higher variations and artifacts in images as a result of evolution. Based on this observation, we developed a psychophysics-based evaluation regime for human assessment and a realistic set of quantitative measures for object recognition performance. The code for conducting such studies will be made publicly available following the publication of this article.

Inspired by the success of the UG2 challenge workshop held at CVPR 2018, we intend to hold subsequent iterations with associated competitions based on the UG2 dataset. These workshops will be similar in spirit to the PASCAL VOC and ImageNet workshops that have been held over the years and will feature new tasks, extending the reach of UG2 beyond the realm of image quality assessment and object classification.

Acknowledgments

Funding for this work was provided under IARPA contract #2016-16070500002, and NSF DGE #1313583. This workshop is supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA). The views and conclusions contained herein are those of the organizers and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

Hardware support was generously provided by the NVIDIA Corporation, and made available by the National Science Foundation (NSF) through grant #CNS-1629914. We thank Drs. Adam Czajka, and Christopher Boehnen for conducting an impartial judgment for the challenge tracks to determine the winners, Mr. Vivek Sharma for executing his code on our data and providing us with the result, Kelly Malecki for her tireless effort in annotating the test dataset for the UAV and Glider collections, and Sandipan Banerjee for assistance with data collection.