Time-variant Image Inpainting via Interactive Distribution Transition Estimation

TL;DR

This paper introduces a new task called Time-variant Image Inpainting (TAMP), addressing the challenge of restoring damaged images using reference images captured at different times with significant content differences, and proposes a novel method with a new dataset.

Contribution

The paper proposes the InDiTE module and TAMP-Diff method for effective TAMP, and creates the first benchmark dataset TAMP-Street for this task.

Findings

Our method outperforms state-of-the-art reference-guided inpainting methods on TAMP tasks.

The proposed approach effectively handles significant content differences between images.

Experimental results demonstrate the robustness of our method across different settings.

Abstract

In this work, we focus on a novel and practical task, i.e., Time-vAriant iMage inPainting (TAMP). The aim of TAMP is to restore a damaged target image by leveraging the complementary information from a reference image, where both images captured the same scene but with a significant time gap in between, i.e., time-variant images. Different from conventional reference-guided image inpainting, the reference image under TAMP setup presents significant content distinction to the target image and potentially also suffers from damages. Such an application frequently happens in our daily lives to restore a damaged image by referring to another reference image, where there is no guarantee of the reference image's source and quality. In particular, our study finds that even state-of-the-art (SOTA) reference-guided image inpainting methods fail to achieve plausible results due to the chaotic image complementation. To address such an ill-posed problem, we propose a novel Interactive Distribution Transition Estimation (InDiTE) module which interactively complements the time-variant images with adaptive semantics thus facilitate the restoration of damaged regions. To further boost the performance, we propose our TAMP solution, namely Interactive Distribution Transition Estimation-driven Diffusion (InDiTE-Diff), which integrates InDiTE with SOTA diffusion model and conducts latent cross-reference during sampling. Moreover, considering the lack of benchmarks for TAMP task, we newly assembled a dataset, i.e., TAMP-Street, based on existing image and mask datasets. We conduct experiments on the TAMP-Street datasets under two different time-variant image inpainting settings, which show our method consistently outperform SOTA reference-guided image inpainting methods for solving TAMP.