Multi-layer Visualization for Medical Mixed Reality

TL;DR

This paper introduces a multi-layer visualization technique in Medical Mixed Reality that makes occluding surgical instruments transparent, enhancing visualization of the surgical scene using volumetric image synthesis and stereo-RGBD cameras.

Contribution

It presents the first application of volumetric fields for Diminished Reality in Medical Mixed Reality, improving visualization by recovering occluded background scenes.

Findings

Background recovery accuracy between 45%-99%.

Enhanced visualization of surgical scenes with transparent occluding objects.

First use of volumetric fields in medical Diminished Reality.

Abstract

Medical Mixed Reality helps surgeons to contextualize intraoperative data with video of the surgical scene. Nonetheless, the surgical scene and anatomical target are often occluded by surgical instruments and surgeon hands. In this paper and to our knowledge, we propose a multi-layer visualization in Medical Mixed Reality solution which subtly improves a surgeon's visualization by making transparent the occluding objects. As an example scenario, we use an augmented reality C-arm fluoroscope device. A video image is created using a volumetric-based image synthesization technique and stereo-RGBD cameras mounted on the C-arm. From this synthesized view, the background which is occluded by the surgical instruments and surgeon hands is recovered by modifying the volumetric-based image synthesization technique. The occluding objects can, therefore, become transparent over the surgical scene. Experimentation with different augmented reality scenarios yield results demonstrating that the background of the surgical scenes can be recovered with accuracy between 45%-99%. In conclusion, we presented a solution that a Mixed Reality solution for medicine, providing transparency to objects occluding the surgical scene. This work is also the first application of volumetric field for Diminished Reality/ Mixed Reality.