Semantic-aware Image Deblurring

TL;DR

This paper introduces a novel semantic-aware image deblurring method that leverages image captioning to guide the deblurring process, significantly improving results on severely blurred images by integrating semantic content understanding.

Contribution

The paper proposes the S3E-Deblur model with a Structured-Spatial Semantic tree that jointly learns image deblurring and captioning, bridging the gap between semantic understanding and image restoration.

Findings

Outperforms state-of-the-art methods on MSCOCO and GoPro datasets

Effectively captures semantic content to improve deblurring quality

Demonstrates the benefit of multi-task learning in image restoration

Abstract

Image deblurring has achieved exciting progress in recent years. However, traditional methods fail to deblur severely blurred images, where semantic contents appears ambiguously. In this paper, we conduct image deblurring guided by the semantic contents inferred from image captioning. Specially, we propose a novel Structured-Spatial Semantic Embedding model for image deblurring (termed S3E-Deblur), which introduces a novel Structured-Spatial Semantic tree model (S3-tree) to bridge two basic tasks in computer vision: image deblurring (ImD) and image captioning (ImC). In particular, S3-tree captures and represents the semantic contents in structured spatial features in ImC, and then embeds the spatial features of the tree nodes into GAN based ImD. Co-training on S3-tree, ImC, and ImD is conducted to optimize the overall model in a multi-task end-to-end manner. Extensive experiments on severely blurred MSCOCO and GoPro datasets demonstrate the significant superiority of S3E-Deblur compared to the state-of-the-arts on both ImD and ImC tasks.