Deep Stacked Hierarchical Multi-patch Network for Image Deblurring

TL;DR

This paper introduces a deep hierarchical multi-patch network for image deblurring that achieves real-time performance at 720p resolution and surpasses previous methods in speed and quality, with scalable depth for different applications.

Contribution

The paper proposes a novel stacked multi-patch network inspired by Spatial Pyramid Matching, enabling faster and more effective image deblurring with scalable depth.

Findings

Achieves state-of-the-art performance on GoPro dataset.

Runs at 30ms for 1280x720 images, enabling real-time deblurring.

Significantly improves deblurring quality with increased network depth.

Abstract

Despite deep end-to-end learning methods have shown their superiority in removing non-uniform motion blur, there still exist major challenges with the current multi-scale and scale-recurrent models: 1) Deconvolution/upsampling operations in the coarse-to-fine scheme result in expensive runtime; 2) Simply increasing the model depth with finer-scale levels cannot improve the quality of deblurring. To tackle the above problems, we present a deep hierarchical multi-patch network inspired by Spatial Pyramid Matching to deal with blurry images via a fine-to-coarse hierarchical representation. To deal with the performance saturation w.r.t. depth, we propose a stacked version of our multi-patch model. Our proposed basic multi-patch model achieves the state-of-the-art performance on the GoPro dataset while enjoying a 40x faster runtime compared to current multi-scale methods. With 30ms to process an image at 1280x720 resolution, it is the first real-time deep motion deblurring model for 720p images at 30fps. For stacked networks, significant improvements (over 1.2dB) are achieved on the GoPro dataset by increasing the network depth. Moreover, by varying the depth of the stacked model, one can adapt the performance and runtime of the same network for different application scenarios.