AsyncDSB: Schedule-Asynchronous Diffusion Schr\"odinger Bridge for Image Inpainting

TL;DR

AsyncDSB introduces a novel asynchronous pixel scheduling strategy for diffusion Schr"odinger bridge-based image inpainting, addressing schedule-restoration mismatches and improving restoration quality by prioritizing high-frequency pixels.

Contribution

The paper proposes a schedule-asynchronous approach that better aligns the theoretical and practical processes in diffusion Schr"odinger bridge models for image inpainting.

Findings

Achieves 3%-14% FID improvement over baselines.

Effectively models pixel-wise asynchronous restoration.

Enhances image quality in real-world datasets.

Abstract

Image inpainting is an important image generation task, which aims to restore corrupted image from partial visible area. Recently, diffusion Schr\"odinger bridge methods effectively tackle this task by modeling the translation between corrupted and target images as a diffusion Schr\"odinger bridge process along a noising schedule path. Although these methods have shown superior performance, in this paper, we find that 1) existing methods suffer from a schedule-restoration mismatching issue, i.e., the theoretical schedule and practical restoration processes usually exist a large discrepancy, which theoretically results in the schedule not fully leveraged for restoring images; and 2) the key reason causing such issue is that the restoration process of all pixels are actually asynchronous but existing methods set a synchronous noise schedule to them, i.e., all pixels shares the same noise schedule. To this end, we propose a schedule-Asynchronous Diffusion Schr\"odinger Bridge (AsyncDSB) for image inpainting. Our insight is preferentially scheduling pixels with high frequency (i.e., large gradients) and then low frequency (i.e., small gradients). Based on this insight, given a corrupted image, we first train a network to predict its gradient map in corrupted area. Then, we regard the predicted image gradient as prior and design a simple yet effective pixel-asynchronous noise schedule strategy to enhance the diffusion Schr\"odinger bridge. Thanks to the asynchronous schedule at pixels, the temporal interdependence of restoration process between pixels can be fully characterized for high-quality image inpainting. Experiments on real-world datasets show that our AsyncDSB achieves superior performance, especially on FID with around 3% - 14% improvement over state-of-the-art baseline methods.