Reference-Guided Diffusion Inpainting For Multimodal Counterfactual Generation

TL;DR

This paper introduces two novel diffusion-based inpainting methods, MObI and AnydoorMed, for generating realistic, controllable, and multimodal synthetic data in autonomous driving and medical imaging, enhancing safety-critical testing.

Contribution

The work presents the first multimodal object inpainting framework using diffusion models and extends it to medical imaging, enabling high-quality, reference-guided inpainting across diverse modalities.

Findings

MObI achieves realistic object insertion in multimodal scenes with spatial accuracy.

AnydoorMed effectively inpaints anomalies in mammography scans with structural and semantic consistency.

Both methods demonstrate adaptability of foundation models to different perceptual modalities.

Abstract

Safety-critical applications, such as autonomous driving and medical image analysis, require extensive multimodal data for rigorous testing. Synthetic data methods are gaining prominence due to the cost and complexity of gathering real-world data, but they demand a high degree of realism and controllability to be useful. This work introduces two novel methods for synthetic data generation in autonomous driving and medical image analysis, namely MObI and AnydoorMed, respectively. MObI is a first-of-its-kind framework for Multimodal Object Inpainting that leverages a diffusion model to produce realistic and controllable object inpaintings across perceptual modalities, demonstrated simultaneously for camera and lidar. Given a single reference RGB image, MObI enables seamless object insertion into existing multimodal scenes at a specified 3D location, guided by a bounding box, while maintaining semantic consistency and multimodal coherence. Unlike traditional inpainting methods that rely solely on edit masks, this approach uses 3D bounding box conditioning to ensure accurate spatial positioning and realistic scaling. AnydoorMed extends this paradigm to the medical imaging domain, focusing on reference-guided inpainting for mammography scans. It leverages a diffusion-based model to inpaint anomalies with impressive detail preservation, maintaining the reference anomaly's structural integrity while semantically blending it with the surrounding tissue. Together, these methods demonstrate that foundation models for reference-guided inpainting in natural images can be readily adapted to diverse perceptual modalities, paving the way for the next generation of systems capable of constructing highly realistic, controllable and multimodal counterfactual scenarios.