Deep Bilateral Learning for Real-Time Image Enhancement

TL;DR

This paper presents a neural network architecture inspired by bilateral grid processing that enables real-time, high-quality image enhancement on mobile devices by learning to approximate complex image transformations from data.

Contribution

The authors introduce a novel neural network design that efficiently predicts local affine transformations in bilateral space for real-time image enhancement without needing the original operator at runtime.

Findings

Processes high-resolution images in milliseconds on smartphones

Achieves real-time 1080p viewfinder performance

Matches quality of state-of-the-art approximation techniques

Abstract

Performance is a critical challenge in mobile image processing. Given a reference imaging pipeline, or even human-adjusted pairs of images, we seek to reproduce the enhancements and enable real-time evaluation. For this, we introduce a new neural network architecture inspired by bilateral grid processing and local affine color transforms. Using pairs of input/output images, we train a convolutional neural network to predict the coefficients of a locally-affine model in bilateral space. Our architecture learns to make local, global, and content-dependent decisions to approximate the desired image transformation. At runtime, the neural network consumes a low-resolution version of the input image, produces a set of affine transformations in bilateral space, upsamples those transformations in an edge-preserving fashion using a new slicing node, and then applies those upsampled transformations to the full-resolution image. Our algorithm processes high-resolution images on a smartphone in milliseconds, provides a real-time viewfinder at 1080p resolution, and matches the quality of state-of-the-art approximation techniques on a large class of image operators. Unlike previous work, our model is trained off-line from data and therefore does not require access to the original operator at runtime. This allows our model to learn complex, scene-dependent transformations for which no reference implementation is available, such as the photographic edits of a human retoucher.