Realistic Endoscopic Image Generation Method Using Virtual-to-real Image-domain Translation

TL;DR

This paper introduces a method to generate realistic endoscopic images by translating virtual images from CT data into real image domains using a deep convolutional network, enhancing simulation realism.

Contribution

It presents a novel virtual-to-real image translation approach using a cycle-consistent FCN trained on unpaired data for endoscopic image enhancement.

Findings

Deep U-Net and residual U-Net produce highly realistic images

Image cleansing improves translation quality

Cycle consistency loss effectively trains the FCN

Abstract

This paper proposes a realistic image generation method for visualization in endoscopic simulation systems. Endoscopic diagnosis and treatment are performed in many hospitals. To reduce complications related to endoscope insertions, endoscopic simulation systems are used for training or rehearsal of endoscope insertions. However, current simulation systems generate non-realistic virtual endoscopic images. To improve the value of the simulation systems, improvement of reality of their generated images is necessary. We propose a realistic image generation method for endoscopic simulation systems. Virtual endoscopic images are generated by using a volume rendering method from a CT volume of a patient. We improve the reality of the virtual endoscopic images using a virtual-to-real image-domain translation technique. The image-domain translator is implemented as a fully convolutional network (FCN). We train the FCN by minimizing a cycle consistency loss function. The FCN is trained using unpaired virtual and real endoscopic images. To obtain high quality image-domain translation results, we perform an image cleansing to the real endoscopic image set. We tested to use the shallow U-Net, U-Net, deep U-Net, and U-Net having residual units as the image-domain translator. The deep U-Net and U-Net having residual units generated quite realistic images.