ObjectMate: A Recurrence Prior for Object Insertion and Subject-Driven Generation

TL;DR

ObjectMate is a tuning-free diffusion-based method that leverages recurrence priors from large unlabeled datasets to achieve photorealistic object insertion and subject-driven generation, preserving object identity without test-time tuning.

Contribution

It introduces a novel recurrence prior approach that enables high-quality object insertion and subject-driven generation without the need for manual data collection or test-time tuning.

Findings

Outperforms state-of-the-art methods in object identity preservation.

Produces more photorealistic and seamless compositions.

Operates without slow test-time tuning.

Abstract

This paper introduces a tuning-free method for both object insertion and subject-driven generation. The task involves composing an object, given multiple views, into a scene specified by either an image or text. Existing methods struggle to fully meet the task's challenging objectives: (i) seamlessly composing the object into the scene with photorealistic pose and lighting, and (ii) preserving the object's identity. We hypothesize that achieving these goals requires large scale supervision, but manually collecting sufficient data is simply too expensive. The key observation in this paper is that many mass-produced objects recur across multiple images of large unlabeled datasets, in different scenes, poses, and lighting conditions. We use this observation to create massive supervision by retrieving sets of diverse views of the same object. This powerful paired dataset enables us to train a straightforward text-to-image diffusion architecture to map the object and scene descriptions to the composited image. We compare our method, ObjectMate, with state-of-the-art methods for object insertion and subject-driven generation, using a single or multiple references. Empirically, ObjectMate achieves superior identity preservation and more photorealistic composition. Differently from many other multi-reference methods, ObjectMate does not require slow test-time tuning.