From Inpainting to Layer Decomposition: Repurposing Generative Inpainting Models for Image Layer Decomposition

TL;DR

This paper introduces a novel approach that repurposes diffusion-based inpainting models for image layer decomposition, enabling independent editing of image components with high detail preservation.

Contribution

The authors adapt a diffusion inpainting model for layer decomposition using lightweight finetuning and introduce a multi-modal context fusion module to enhance detail preservation.

Findings

Achieves superior object removal and occlusion recovery

Operates effectively with synthetic training data

Enables advanced image editing and creative applications

Abstract

Images can be viewed as layered compositions, foreground objects over background, with potential occlusions. This layered representation enables independent editing of elements, offering greater flexibility for content creation. Despite the progress in large generative models, decomposing a single image into layers remains challenging due to limited methods and data. We observe a strong connection between layer decomposition and in/outpainting tasks, and propose adapting a diffusion-based inpainting model for layer decomposition using lightweight finetuning. To further preserve detail in the latent space, we introduce a novel multi-modal context fusion module with linear attention complexity. Our model is trained purely on a synthetic dataset constructed from open-source assets and achieves superior performance in object removal and occlusion recovery, unlocking new possibilities in downstream editing and creative applications.