LDP: Language-driven Dual-Pixel Image Defocus Deblurring Network

TL;DR

This paper introduces a novel unsupervised framework that leverages CLIP's language-image understanding to estimate blur maps from dual-pixel image pairs, significantly improving deblurring performance.

Contribution

It is the first to utilize CLIP with carefully designed prompts for blur map estimation in dual-pixel deblurring without fine-tuning.

Findings

Achieves state-of-the-art deblurring results.

Effectively estimates blur maps using CLIP without fine-tuning.

Improves all-in-focus image recovery quality.

Abstract

Recovering sharp images from dual-pixel (DP) pairs with disparity-dependent blur is a challenging task.~Existing blur map-based deblurring methods have demonstrated promising results. In this paper, we propose, to the best of our knowledge, the first framework that introduces the contrastive language-image pre-training framework (CLIP) to accurately estimate the blur map from a DP pair unsupervisedly. To achieve this, we first carefully design text prompts to enable CLIP to understand blur-related geometric prior knowledge from the DP pair. Then, we propose a format to input a stereo DP pair to CLIP without any fine-tuning, despite the fact that CLIP is pre-trained on monocular images. Given the estimated blur map, we introduce a blur-prior attention block, a blur-weighting loss, and a blur-aware loss to recover the all-in-focus image. Our method achieves state-of-the-art performance in extensive experiments (see Fig.~\ref{fig:teaser}).