From Gallery to Wrist: Realistic 3D Bracelet Insertion in Videos

TL;DR

This paper presents a hybrid approach combining 3D Gaussian Splatting and 2D diffusion models to insert realistic, temporally consistent bracelets into videos, addressing challenges in lighting, motion, and perspective changes.

Contribution

It introduces a novel pipeline that synergizes 3D rendering and 2D diffusion for photorealistic, temporally coherent video object insertion, specifically for bracelets.

Findings

Enhanced temporal consistency in video insertion

Improved photorealistic lighting effects

First to combine 3D Gaussian Splatting with diffusion models for this task

Abstract

Inserting 3D objects into videos is a longstanding challenge in computer graphics with applications in augmented reality, virtual try-on, and video composition. Achieving both temporal consistency, or realistic lighting remains difficult, particularly in dynamic scenarios with complex object motion, perspective changes, and varying illumination. While 2D diffusion models have shown promise for producing photorealistic edits, they often struggle with maintaining temporal coherence across frames. Conversely, traditional 3D rendering methods excel in spatial and temporal consistency but fall short in achieving photorealistic lighting. In this work, we propose a hybrid object insertion pipeline that combines the strengths of both paradigms. Specifically, we focus on inserting bracelets into dynamic wrist scenes, leveraging the high temporal consistency of 3D Gaussian Splatting (3DGS) for initial rendering and refining the results using a 2D diffusion-based enhancement model to ensure realistic lighting interactions. Our method introduces a shading-driven pipeline that separates intrinsic object properties (albedo, shading, reflectance) and refines both shading and sRGB images for photorealism. To maintain temporal coherence, we optimize the 3DGS model with multi-frame weighted adjustments. This is the first approach to synergize 3D rendering and 2D diffusion for video object insertion, offering a robust solution for realistic and consistent video editing. Project Page: https://cjeen.github.io/BraceletPaper/