Image Inpainting via Iteratively Decoupled Probabilistic Modeling
Wenbo Li, Xin Yu, Kun Zhou, Yibing Song, Zhe Lin, Jiaya Jia
TL;DR
This paper introduces a novel pixel spread model (PSM) that combines the efficiency of GANs with probabilistic models to improve image inpainting quality and speed, especially for large missing regions.
Contribution
The paper proposes a new iterative decoupled probabilistic model that enhances image inpainting by efficiently spreading informative pixels, achieving state-of-the-art results.
Findings
Achieves new state-of-the-art performance on multiple benchmarks.
Enhances inpainting quality with fewer iterations.
Reduces computational cost compared to existing probabilistic methods.
Abstract
Generative adversarial networks (GANs) have made great success in image inpainting yet still have difficulties tackling large missing regions. In contrast, iterative probabilistic algorithms, such as autoregressive and denoising diffusion models, have to be deployed with massive computing resources for decent effect. To achieve high-quality results with low computational cost, we present a novel pixel spread model (PSM) that iteratively employs decoupled probabilistic modeling, combining the optimization efficiency of GANs with the prediction tractability of probabilistic models. As a result, our model selectively spreads informative pixels throughout the image in a few iterations, largely enhancing the completion quality and efficiency. On multiple benchmarks, we achieve new state-of-the-art performance. Code is released at https://github.com/fenglinglwb/PSM.
Peer Reviews
Decision·ICLR 2024 spotlight
The proposed pixel spread model makes use of the merits of GANs’ efficient optimization and the tractability of probabilistic models. Good performance is achieved on several benchmark datasets.
There are several unclear statements listed as follows.
- The idea is novel to gradually fill the regions through the computed mean and variance. - The method is efficient by utilizing the optimized computational resource of GAN model and the thoughts of autoregressive model.
- Some parts of the paper is not very clear. For example, - What's the meaning of $y^i$ in equation (1)? Is it the groundtruth? - More explanation is necessary for equation (5). What happens when $m_t=1$ and $m_0=0$ or vise versa? - Is the uncertainty map computed only in the mask region or the entire image? - Why $\sigma_t$ is in the range $[0,1]$? - Comparison to Controlnet is necessary. - It seems that sometimes the model is hard to grasp some structure of the image. For exam
1. A very simple method that gives really good results. 2. Authors experimented with large-scale data sets. 3. Results are very impressive. The presented comparisons with state-of-the-art methods shows that the proposed iterative scheme yields better in-painting than all the rest. Furthermore, it does it in a fraction of the time compared to the most recent methods. I believe these results are extremely promising, and if they can be reproduced they may have a substantial impact.
1. The training strategy is not well explained. Authors should clarify whether and how the iterations are taken into account during the training. 2. There are some suspiciously accurate in-painting results in the appendix - particularly in Fig. M7. Can authors explain how the proposed model can be so accurate? I can understand being realistic, however, the third column shows that the generation is pretty much the same. Similar in-apinting results are given in Figure M8.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis · Advanced Image Processing Techniques · Image and Signal Denoising Methods
MethodsInpainting · Diffusion