ColoDiff: Integrating Dynamic Consistency With Content Awareness for Colonoscopy Video Generation

TL;DR

ColoDiff is a diffusion-based framework that generates dynamic, content-aware colonoscopy videos with high temporal consistency and precise clinical attribute control, addressing data scarcity and aiding clinical diagnosis.

Contribution

It introduces novel inter-frame and intra-frame modules for dynamic modeling and content control, along with a non-Markovian sampling strategy for real-time video generation.

Findings

Generates smooth, dynamic colonoscopy videos with high quality.

Improves downstream clinical tasks like disease diagnosis and lesion segmentation.

Reduces sampling steps by over 90% for real-time synthesis.

Abstract

Colonoscopy video generation delivers dynamic, information-rich data critical for diagnosing intestinal diseases, particularly in data-scarce scenarios. High-quality video generation demands temporal consistency and precise control over clinical attributes, but faces challenges from irregular intestinal structures, diverse disease representations, and various imaging modalities. To this end, we propose ColoDiff, a diffusion-based framework that generates dynamic-consistent and content-aware colonoscopy videos, aiming to alleviate data shortage and assist clinical analysis. At the inter-frame level, our TimeStream module decouples temporal dependency from video sequences through a cross-frame tokenization mechanism, enabling intricate dynamic modeling despite irregular intestinal structures. At the intra-frame level, our Content-Aware module incorporates noise-injected embeddings and learnable prototypes to realize precise control over clinical attributes, breaking through the coarse guidance of diffusion models. Additionally, ColoDiff employs a non-Markovian sampling strategy that cuts steps by over 90% for real-time generation. ColoDiff is evaluated across three public datasets and one hospital database, based on both generation metrics and downstream tasks including disease diagnosis, modality discrimination, bowel preparation scoring, and lesion segmentation. Extensive experiments show ColoDiff generates videos with smooth transitions and rich dynamics. ColoDiff presents an effort in controllable colonoscopy video generation, revealing the potential of synthetic videos in complementing authentic representation and mitigating data scarcity in clinical settings.