OccFusion: Rendering Occluded Humans with Generative Diffusion Priors

TL;DR

OccFusion is a novel method that leverages 3D Gaussian splatting and diffusion models to render occluded humans with high fidelity, overcoming limitations of previous approaches that require full visibility.

Contribution

This work introduces a three-stage pipeline combining mask generation, Gaussian optimization, and in-context inpainting for occluded human rendering.

Findings

Achieves state-of-the-art performance on ZJU-MoCap and OcMotion datasets.

Effectively handles partial visibility and occlusions in human rendering.

Improves rendering quality of less observed human body parts.

Abstract

Most existing human rendering methods require every part of the human to be fully visible throughout the input video. However, this assumption does not hold in real-life settings where obstructions are common, resulting in only partial visibility of the human. Considering this, we present OccFusion, an approach that utilizes efficient 3D Gaussian splatting supervised by pretrained 2D diffusion models for efficient and high-fidelity human rendering. We propose a pipeline consisting of three stages. In the Initialization stage, complete human masks are generated from partial visibility masks. In the Optimization stage, 3D human Gaussians are optimized with additional supervision by Score-Distillation Sampling (SDS) to create a complete geometry of the human. Finally, in the Refinement stage, in-context inpainting is designed to further improve rendering quality on the less observed human body parts. We evaluate OccFusion on ZJU-MoCap and challenging OcMotion sequences and find that it achieves state-of-the-art performance in the rendering of occluded humans.