SG2VID: Scene Graphs Enable Fine-Grained Control for Video Synthesis
Ssharvien Kumar Sivakumar, Yannik Frisch, Ghazal Ghazaei, Anirban Mukhopadhyay

TL;DR
SG2VID is a novel diffusion-based video model that uses Scene Graphs to enable precise, fine-grained control over surgical video synthesis, improving realism and variability for training and analysis.
Contribution
It introduces SG2VID, the first model to leverage Scene Graphs for detailed control in video synthesis, specifically applied to surgical videos.
Findings
Outperforms previous methods qualitatively and quantitatively
Enables precise control over tools and anatomy in generated videos
Improves downstream phase detection tasks with synthetic data
Abstract
Surgical simulation plays a pivotal role in training novice surgeons, accelerating their learning curve and reducing intra-operative errors. However, conventional simulation tools fall short in providing the necessary photorealism and the variability of human anatomy. In response, current methods are shifting towards generative model-based simulators. Yet, these approaches primarily focus on using increasingly complex conditioning for precise synthesis while neglecting the fine-grained human control aspect. To address this gap, we introduce SG2VID, the first diffusion-based video model that leverages Scene Graphs for both precise video synthesis and fine-grained human control. We demonstrate SG2VID's capabilities across three public datasets featuring cataract and cholecystectomy surgery. While SG2VID outperforms previous methods both qualitatively and quantitatively, it also enables…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Vision and Imaging · Computer Graphics and Visualization Techniques · Generative Adversarial Networks and Image Synthesis
