Stomach 3D Reconstruction Based on Virtual Chromoendoscopic Image Generation

TL;DR

This paper introduces a method to reconstruct the 3D shape of the stomach from endoscopic images without using dye spraying, by generating virtual dye images through style transfer, thus improving clinical efficiency.

Contribution

It proposes a novel style transfer approach to generate virtual chromoendoscopic images, eliminating the need for dye spraying in 3D stomach reconstruction from endoscopic images.

Findings

Green-channel input yields best virtual dye images for reconstruction

Virtual dye images improve structure-from-motion results

Method reduces time and cost of gastric 3D reconstruction

Abstract

Gastric endoscopy is a standard clinical process that enables medical practitioners to diagnose various lesions inside a patient's stomach. If any lesion is found, it is very important to perceive the location of the lesion relative to the global view of the stomach. Our previous research showed that this could be addressed by reconstructing the whole stomach shape from chromoendoscopic images using a structure-from-motion (SfM) pipeline, in which indigo carmine (IC) blue dye sprayed images were used to increase feature matches for SfM by enhancing stomach surface's textures. However, spraying the IC dye to the whole stomach requires additional time, labor, and cost, which is not desirable for patients and practitioners. In this paper, we propose an alternative way to achieve whole stomach 3D reconstruction without the need of the IC dye by generating virtual IC-sprayed (VIC) images based on image-to-image style translation trained on unpaired real no-IC and IC-sprayed images. We have specifically investigated the effect of input and output color channel selection for generating the VIC images and found that translating no-IC green-channel images to IC-sprayed red-channel images gives the best SfM reconstruction result.