Generative Modeling in Structural-Hankel Domain for Color Image Inpainting

TL;DR

This paper introduces a novel low-rank structural-Hankel matrix-based score generative model for color image inpainting, requiring minimal samples and leveraging internal patch statistics for high-quality, diverse restorations.

Contribution

It proposes a new approach that constructs low-rank Hankel matrices from few samples and integrates them into a score-based generative model for effective image inpainting.

Findings

Achieves high-quality inpainting with minimal samples

Demonstrates superior diversity and performance over existing methods

Effectively models internal patch statistics for image restoration

Abstract

In recent years, some researchers focused on using a single image to obtain a large number of samples through multi-scale features. This study intends to a brand-new idea that requires only ten or even fewer samples to construct the low-rank structural-Hankel matrices-assisted score-based generative model (SHGM) for color image inpainting task. During the prior learning process, a certain amount of internal-middle patches are firstly extracted from several images and then the structural-Hankel matrices are constructed from these patches. To better apply the score-based generative model to learn the internal statistical distribution within patches, the large-scale Hankel matrices are finally folded into the higher dimensional tensors for prior learning. During the iterative inpainting process, SHGM views the inpainting problem as a conditional generation procedure in low-rank environment. As a result, the intermediate restored image is acquired by alternatively performing the stochastic differential equation solver, alternating direction method of multipliers, and data consistency steps. Experimental results demonstrated the remarkable performance and diversity of SHGM.