Automated Virtual Product Placement and Assessment in Images using Diffusion Models

TL;DR

This paper presents a fully automated three-stage system for virtual product placement in images, combining language-guided segmentation, diffusion-based inpainting, and quality filtering to improve image integration and quality.

Contribution

It introduces a novel three-stage VPP system utilizing diffusion models and an alignment module for improved placement accuracy and image quality.

Findings

Ensures the presence of the product in every generated image.

Enhances average image quality by 35%.

Demonstrates effectiveness in virtual advertising applications.

Abstract

In Virtual Product Placement (VPP) applications, the discrete integration of specific brand products into images or videos has emerged as a challenging yet important task. This paper introduces a novel three-stage fully automated VPP system. In the first stage, a language-guided image segmentation model identifies optimal regions within images for product inpainting. In the second stage, Stable Diffusion (SD), fine-tuned with a few example product images, is used to inpaint the product into the previously identified candidate regions. The final stage introduces an "Alignment Module", which is designed to effectively sieve out low-quality images. Comprehensive experiments demonstrate that the Alignment Module ensures the presence of the intended product in every generated image and enhances the average quality of images by 35%. The results presented in this paper demonstrate the effectiveness of the proposed VPP system, which holds significant potential for transforming the landscape of virtual advertising and marketing strategies.