Holographic Visualisation of Radiology Data and Automated Machine Learning-based Medical Image Segmentation

TL;DR

This paper presents an AR platform integrated with automated CNN-based segmentation for medical imaging, enabling intuitive 3D visualization of CT and MRI data for medical staff, aiming to improve clinical workflows.

Contribution

It introduces a scalable AR visualization system combined with an automated CNN segmentation pipeline specifically designed for medical imaging applications.

Findings

Demonstrated good segmentation results on LiTS dataset

Developed a real-time, scalable AR visualization architecture

Proposed a fully automated liver segmentation method

Abstract

Within this thesis we propose a platform for combining Augmented Reality (AR) hardware with machine learning in a user-oriented pipeline, offering to the medical staff an intuitive 3D visualization of volumetric Computed Tomography (CT) and Magnetic Resonance Imaging (MRI) medical image segmentations inside the AR headset, that does not need human intervention for loading, processing and segmentation of medical images. The AR visualization, based on Microsoft HoloLens, employs a modular and thus scalable frontend-backend architecture for real-time visualizations on multiple AR headsets. As Convolutional Neural Networks (CNNs) have lastly demonstrated superior performance for the machine learning task of image semantic segmentation, the pipeline also includes a fully automated CNN algorithm for the segmentation of the liver from CT scans. The model is based on the Deep Retinal Image Understanding (DRIU) model which is a Fully Convolutional Network with side outputs from feature maps with different resolution, extracted at different stages of the network. The algorithm is 2.5D which means that the input is a set of consecutive scan slices. The experiments have been performed on the Liver Tumor Segmentation Challenge (LiTS) dataset for liver segmentation and demonstrated good results and flexibility. While multiple approaches exist in the domain, only few of them have focused on overcoming the practical aspects which still largely hold this technology away from the operating rooms. In line with this, we also are next planning an evaluation from medical doctors and radiologists in a real-world environment.