HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awareness

TL;DR

HOI-Swap is a diffusion-based framework that enables realistic object swapping in videos with hand-object interaction awareness, addressing limitations of existing models in handling interaction intricacies.

Contribution

The paper introduces a self-supervised, two-stage diffusion-based method for object swapping in videos that preserves hand-object interactions and extends edits across sequences.

Findings

Outperforms existing methods in quality and realism.

Effectively preserves hand-object interaction patterns.

Enables controllable motion alignment in edited videos.

Abstract

We study the problem of precisely swapping objects in videos, with a focus on those interacted with by hands, given one user-provided reference object image. Despite the great advancements that diffusion models have made in video editing recently, these models often fall short in handling the intricacies of hand-object interactions (HOI), failing to produce realistic edits -- especially when object swapping results in object shape or functionality changes. To bridge this gap, we present HOI-Swap, a novel diffusion-based video editing framework trained in a self-supervised manner. Designed in two stages, the first stage focuses on object swapping in a single frame with HOI awareness; the model learns to adjust the interaction patterns, such as the hand grasp, based on changes in the object's properties. The second stage extends the single-frame edit across the entire sequence; we achieve controllable motion alignment with the original video by: (1) warping a new sequence from the stage-I edited frame based on sampled motion points and (2) conditioning video generation on the warped sequence. Comprehensive qualitative and quantitative evaluations demonstrate that HOI-Swap significantly outperforms existing methods, delivering high-quality video edits with realistic HOIs.