PersonaHOI: Effortlessly Improving Personalized Face with Human-Object Interaction Generation

TL;DR

PersonaHOI is a novel framework that combines general and personalized diffusion models to generate realistic, identity-consistent human-object interaction images, maintaining facial details and full-body coherence without additional training.

Contribution

It introduces a training- and tuning-free method that fuses StableDiffusion with a personalized face model using cross-attention and spatial merging, improving HOI image generation.

Findings

Outperforms existing methods in realism and scalability

Preserves facial identity while maintaining full-body coherence

Validated by a new interaction alignment metric

Abstract

We introduce PersonaHOI, a training- and tuning-free framework that fuses a general StableDiffusion model with a personalized face diffusion (PFD) model to generate identity-consistent human-object interaction (HOI) images. While existing PFD models have advanced significantly, they often overemphasize facial features at the expense of full-body coherence, PersonaHOI introduces an additional StableDiffusion (SD) branch guided by HOI-oriented text inputs. By incorporating cross-attention constraints in the PFD branch and spatial merging at both latent and residual levels, PersonaHOI preserves personalized facial details while ensuring interactive non-facial regions. Experiments, validated by a novel interaction alignment metric, demonstrate the superior realism and scalability of PersonaHOI, establishing a new standard for practical personalized face with HOI generation. Our code will be available at https://github.com/JoyHuYY1412/PersonaHOI