Seeing Through Smoke: Surgical Desmoking for Improved Visual Perception

TL;DR

This paper introduces a transformer-based model for surgical desmoking that enhances endoscopic image clarity, utilizing a large synthetic dataset and real surgical images to improve visualization and downstream tasks.

Contribution

The authors develop a novel physics-inspired desmoking model, create the largest paired surgical smoke dataset, and demonstrate state-of-the-art performance in image reconstruction and downstream surgical tasks.

Findings

State-of-the-art image reconstruction performance.

Synthetic data pipeline yields over 80,000 training samples.

Desmoking improves stereo depth estimation and instrument segmentation.

Abstract

Minimally invasive and robot-assisted surgery relies heavily on endoscopic imaging, yet surgical smoke produced by electrocautery and vessel-sealing instruments can severely degrade visual perception and hinder vision-based functionalities. We present a transformer-based surgical desmoking model with a physics-inspired desmoking head that jointly predicts smoke-free image and corresponding smoke map. To address the scarcity of paired smoky-to-smoke-free training data, we develop a synthetic data generation pipeline that blends artificial smoke patterns with real endoscopic images, yielding over 80,000 paired samples for supervised training. We further curate, to our knowledge, the largest paired surgical smoke dataset to date, comprising 5,817 image pairs captured with the da Vinci robotic surgical system, enabling benchmarking on high-resolution endoscopic images. Extensive experiments on both a public benchmark and our dataset demonstrate state-of-the-art performance in image reconstruction compared to existing dehazing and desmoking approaches. We also assess the impact of desmoking on downstream stereo depth estimation and instrument segmentation, highlighting both the potential benefits and current limitations of digital smoke removal methods.