Affordance-Aware Object Insertion via Mask-Aware Dual Diffusion

TL;DR

This paper introduces a new framework for object insertion into scenes based on affordance principles, supported by a large dataset and a dual diffusion model, achieving superior results in image composition tasks.

Contribution

It extends affordance concepts to object-scene composition, creates the SAM-FB dataset with over 3 million examples, and proposes the Mask-Aware Dual Diffusion model for improved insertion performance.

Findings

Outperforms state-of-the-art methods in object insertion tasks.

Demonstrates strong generalization on in-the-wild images.

Effectively models insertion masks to enhance composition quality.

Abstract

As a common image editing operation, image composition involves integrating foreground objects into background scenes. In this paper, we expand the application of the concept of Affordance from human-centered image composition tasks to a more general object-scene composition framework, addressing the complex interplay between foreground objects and background scenes. Following the principle of Affordance, we define the affordance-aware object insertion task, which aims to seamlessly insert any object into any scene with various position prompts. To address the limited data issue and incorporate this task, we constructed the SAM-FB dataset, which contains over 3 million examples across more than 3,000 object categories. Furthermore, we propose the Mask-Aware Dual Diffusion (MADD) model, which utilizes a dual-stream architecture to simultaneously denoise the RGB image and the insertion mask. By explicitly modeling the insertion mask in the diffusion process, MADD effectively facilitates the notion of affordance. Extensive experimental results show that our method outperforms the state-of-the-art methods and exhibits strong generalization performance on in-the-wild images. Please refer to our code on https://github.com/KaKituken/affordance-aware-any.