Diffusion-based Holistic Texture Rectification and Synthesis

TL;DR

This paper introduces a diffusion-based framework that effectively rectifies occlusions and distortions in degraded natural textures, enabling holistic texture synthesis from partial samples using a novel occlusion-aware latent transformer.

Contribution

The paper proposes a new diffusion model with an occlusion-aware transformer for texture synthesis from degraded samples, extending exemplar-based methods to natural images with occlusions.

Findings

Outperforms existing texture synthesis methods quantitatively and qualitatively.

The framework effectively handles large occlusions and distortions in natural textures.

Ablation studies confirm the importance of each component in the model.

Abstract

We present a novel framework for rectifying occlusions and distortions in degraded texture samples from natural images. Traditional texture synthesis approaches focus on generating textures from pristine samples, which necessitate meticulous preparation by humans and are often unattainable in most natural images. These challenges stem from the frequent occlusions and distortions of texture samples in natural images due to obstructions and variations in object surface geometry. To address these issues, we propose a framework that synthesizes holistic textures from degraded samples in natural images, extending the applicability of exemplar-based texture synthesis techniques. Our framework utilizes a conditional Latent Diffusion Model (LDM) with a novel occlusion-aware latent transformer. This latent transformer not only effectively encodes texture features from partially-observed samples necessary for the generation process of the LDM, but also explicitly captures long-range dependencies in samples with large occlusions. To train our model, we introduce a method for generating synthetic data by applying geometric transformations and free-form mask generation to clean textures. Experimental results demonstrate that our framework significantly outperforms existing methods both quantitatively and quantitatively. Furthermore, we conduct comprehensive ablation studies to validate the different components of our proposed framework. Results are corroborated by a perceptual user study which highlights the efficiency of our proposed approach.