Stable Diffusion-Based Approach for Human De-Occlusion

TL;DR

This paper introduces a novel two-stage diffusion-based method for human de-occlusion, reconstructing occluded body parts and appearances by leveraging prior models, explicit spatial cues, and textual features, outperforming existing approaches.

Contribution

The work presents a new two-stage diffusion approach combining body priors, spatial cues, and textual features for improved human de-occlusion, with fine-tuned diffusion models to reduce artifacts.

Findings

Outperforms existing de-occlusion methods in mask and RGB reconstruction.

Enhances downstream tasks like pose estimation and 3D reconstruction.

Effective reconstruction even under severe occlusions.

Abstract

Humans can infer the missing parts of an occluded object by leveraging prior knowledge and visible cues. However, enabling deep learning models to accurately predict such occluded regions remains a challenging task. De-occlusion addresses this problem by reconstructing both the mask and RGB appearance. In this work, we focus on human de-occlusion, specifically targeting the recovery of occluded body structures and appearances. Our approach decomposes the task into two stages: mask completion and RGB completion. The first stage leverages a diffusion-based human body prior to provide a comprehensive representation of body structure, combined with occluded joint heatmaps that offer explicit spatial cues about missing regions. The reconstructed amodal mask then serves as a conditioning input for the second stage, guiding the model on which areas require RGB reconstruction. To further enhance RGB generation, we incorporate human-specific textual features derived using a visual question answering (VQA) model and encoded via a CLIP encoder. RGB completion is performed using Stable Diffusion, with decoder fine-tuning applied to mitigate pixel-level degradation in visible regions -- a known limitation of prior diffusion-based de-occlusion methods caused by latent space transformations. Our method effectively reconstructs human appearances even under severe occlusions and consistently outperforms existing methods in both mask and RGB completion. Moreover, the de-occluded images generated by our approach can improve the performance of downstream human-centric tasks, such as 2D pose estimation and 3D human reconstruction. The code will be made publicly available.