HOComp: Interaction-Aware Human-Object Composition

TL;DR

HOComp is a novel method for human-object image composition that ensures interaction-aware, seamless blending by leveraging multi-level pose guidance and appearance preservation, validated on a new IHOC dataset.

Contribution

This paper introduces HOComp, the first approach to interaction-aware human-object composition, with innovative pose guidance and appearance preservation techniques, along with a new IHOC dataset.

Findings

HOComp outperforms existing methods in generating natural human-object interactions.

The approach achieves high visual harmony and appearance consistency in composite images.

Experimental results demonstrate the effectiveness of the proposed techniques on the IHOC dataset.

Abstract

While existing image-guided composition methods may help insert a foreground object onto a user-specified region of a background image, achieving natural blending inside the region with the rest of the image unchanged, we observe that these existing methods often struggle in synthesizing seamless interaction-aware compositions when the task involves human-object interactions. In this paper, we first propose HOComp, a novel approach for compositing a foreground object onto a human-centric background image, while ensuring harmonious interactions between the foreground object and the background person and their consistent appearances. Our approach includes two key designs: (1) MLLMs-driven Region-based Pose Guidance (MRPG), which utilizes MLLMs to identify the interaction region as well as the interaction type (e.g., holding and lefting) to provide coarse-to-fine constraints to the generated pose for the interaction while incorporating human pose landmarks to track action variations and enforcing fine-grained pose constraints; and (2) Detail-Consistent Appearance Preservation (DCAP), which unifies a shape-aware attention modulation mechanism, a multi-view appearance loss, and a background consistency loss to ensure consistent shapes/textures of the foreground and faithful reproduction of the background human. We then propose the first dataset, named Interaction-aware Human-Object Composition (IHOC), for the task. Experimental results on our dataset show that HOComp effectively generates harmonious human-object interactions with consistent appearances, and outperforms relevant methods qualitatively and quantitatively.