Imagining the Unseen: Generative Location Modeling for Object Placement

TL;DR

This paper introduces a generative model that predicts plausible object locations in images, improving scene composition and object insertion tasks by addressing dataset sparsity and the one-to-many nature of placement options.

Contribution

We develop a novel autoregressive transformer-based location model that learns to generate plausible object bounding boxes conditioned on images and object classes, incorporating negative labels for refinement.

Findings

Achieves superior placement accuracy on the OPA dataset.

Enhances object insertion quality with better visual coherence.

Outperforms discriminative baselines and image composition methods.

Abstract

Location modeling, or determining where non-existing objects could feasibly appear in a scene, has the potential to benefit numerous computer vision tasks, from automatic object insertion to scene creation in virtual reality. Yet, this capability remains largely unexplored to date. In this paper, we develop a generative location model that, given an object class and an image, learns to predict plausible bounding boxes for such an object. Our approach first tokenizes the image and target object class, then decodes bounding box coordinates through an autoregressive transformer. This formulation effectively addresses two core challenges in locatio modeling: the inherent one-to-many nature of plausible locations, and the sparsity of existing location modeling datasets, where fewer than 1% of valid placements are labeled. Furthermore, we incorporate Direct Preference Optimization to leverage negative labels, refining the spatial predictions. Empirical evaluations reveal that our generative location model achieves superior placement accuracy on the OPA dataset as compared to discriminative baselines and image composition approaches. We further test our model in the context of object insertion, where it proposes locations for an off-the-shelf inpainting model to render objects. In this respect, our proposal exhibits improved visual coherence relative to state-of-the-art instruction-tuned editing methods, demonstrating a high-performing location model's utility in a downstream application.