MObI: Multimodal Object Inpainting Using Diffusion Models

TL;DR

MObI introduces a diffusion-based framework for realistic, controllable multimodal object inpainting in scenes, enabling precise object insertion across camera and lidar data for autonomous system testing.

Contribution

The paper presents a novel diffusion model approach for multimodal object inpainting with 3D spatial control, enhancing realism and flexibility over traditional methods.

Findings

Enables seamless insertion of objects into multimodal scenes.

Maintains semantic consistency and multimodal coherence.

Provides accurate spatial positioning and realistic scaling.

Abstract

Safety-critical applications, such as autonomous driving, require extensive multimodal data for rigorous testing. Methods based on synthetic data are gaining prominence due to the cost and complexity of gathering real-world data but require a high degree of realism and controllability in order to be useful. This paper introduces MObI, a novel framework for Multimodal Object Inpainting that leverages a diffusion model to create realistic and controllable object inpaintings across perceptual modalities, demonstrated for both camera and lidar simultaneously. Using a single reference RGB image, MObI enables objects to be seamlessly inserted into existing multimodal scenes at a 3D location specified by a bounding box, while maintaining semantic consistency and multimodal coherence. Unlike traditional inpainting methods that rely solely on edit masks, our 3D bounding box conditioning gives objects accurate spatial positioning and realistic scaling. As a result, our approach can be used to insert novel objects flexibly into multimodal scenes, providing significant advantages for testing perception models.