Learning to Reduce Defocus Blur by Realistically Modeling Dual-Pixel Data

TL;DR

This paper introduces a method to generate realistic dual-pixel data synthetically, enabling improved defocus deblurring models without the need for extensive real DP data collection.

Contribution

We propose a synthetic data generation process for dual-pixel images and a recurrent convolutional network for defocus deblurring applicable to single and multi-frame data.

Findings

Synthetic DP data improves deblurring performance.

The RCN architecture outperforms existing methods.

Synthetic data is effective for training video deblurring models.

Abstract

Recent work has shown impressive results on data-driven defocus deblurring using the two-image views available on modern dual-pixel (DP) sensors. One significant challenge in this line of research is access to DP data. Despite many cameras having DP sensors, only a limited number provide access to the low-level DP sensor images. In addition, capturing training data for defocus deblurring involves a time-consuming and tedious setup requiring the camera's aperture to be adjusted. Some cameras with DP sensors (e.g., smartphones) do not have adjustable apertures, further limiting the ability to produce the necessary training data. We address the data capture bottleneck by proposing a procedure to generate realistic DP data synthetically. Our synthesis approach mimics the optical image formation found on DP sensors and can be applied to virtual scenes rendered with standard computer software. Leveraging these realistic synthetic DP images, we introduce a recurrent convolutional network (RCN) architecture that improves deblurring results and is suitable for use with single-frame and multi-frame data (e.g., video) captured by DP sensors. Finally, we show that our synthetic DP data is useful for training DNN models targeting video deblurring applications where access to DP data remains challenging.