Contact-Aware Amodal Completion for Human-Object Interaction via Multi-Regional Inpainting

TL;DR

This paper introduces a contact-aware amodal completion method for human-object interactions that leverages physical priors and multi-regional inpainting within diffusion models to improve plausibility and realism in dynamic scenarios.

Contribution

It proposes a novel multi-regional inpainting approach incorporating physical contact constraints, enhancing amodal completion in HOI beyond existing diffusion-based methods.

Findings

Outperforms existing methods in HOI amodal completion

Improves shape and visual detail accuracy of completions

Robust even without ground-truth contact annotations

Abstract

Amodal completion, which is the process of inferring the full appearance of objects despite partial occlusions, is crucial for understanding complex human-object interactions (HOI) in computer vision and robotics. Existing methods, such as those that use pre-trained diffusion models, often struggle to generate plausible completions in dynamic scenarios because they have a limited understanding of HOI. To solve this problem, we've developed a new approach that uses physical prior knowledge along with a specialized multi-regional inpainting technique designed for HOI. By incorporating physical constraints from human topology and contact information, we define two distinct regions: the primary region, where occluded object parts are most likely to be, and the secondary region, where occlusions are less probable. Our multi-regional inpainting method uses customized denoising strategies across these regions within a diffusion model. This improves the accuracy and realism of the generated completions in both their shape and visual detail. Our experimental results show that our approach significantly outperforms existing methods in HOI scenarios, moving machine perception closer to a more human-like understanding of dynamic environments. We also show that our pipeline is robust even without ground-truth contact annotations, which broadens its applicability to tasks like 3D reconstruction and novel view/pose synthesis.