Post-mastoidectomy Surface Multi-View Synthesis from a Single Microscopy Image

TL;DR

This paper presents a pipeline that generates synthetic multi-view videos from a single microscope image using pre-operative CT scans, aiding AR and tool tracking in cochlear implant surgeries.

Contribution

It introduces a novel method for creating a large dataset of synthetic views with known poses from a single microscopy image and pre-operative CT data.

Findings

High-quality synthetic views with SSIM ~0.86 using Pytorch3D and PyVista

Generated dataset supports training for automatic microscope pose estimation

Enables downstream AR and tool tracking in cochlear implant surgeries

Abstract

Cochlear Implant (CI) procedures involve performing an invasive mastoidectomy to insert an electrode array into the cochlea. In this paper, we introduce a novel pipeline that is capable of generating synthetic multi-view videos from a single CI microscope image. In our approach, we use a patient's pre-operative CT scan to predict the post-mastoidectomy surface using a method designed for this purpose. We manually align the surface with a selected microscope frame to obtain an accurate initial pose of the reconstructed CT mesh relative to the microscope. We then perform UV projection to transfer the colors from the frame to surface textures. Novel views of the textured surface can be used to generate a large dataset of synthetic frames with ground truth poses. We evaluated the quality of synthetic views rendered using Pytorch3D and PyVista. We found both rendering engines lead to similarly high-quality synthetic novel-view frames compared to ground truth with a structural similarity index for both methods averaging about 0.86. A large dataset of novel views with known poses is critical for ongoing training of a method to automatically estimate microscope pose for 2D to 3D registration with the pre-operative CT to facilitate augmented reality surgery. This dataset will empower various downstream tasks, such as integrating Augmented Reality (AR) in the OR, tracking surgical tools, and supporting other video analysis studies.