Action Learning for 3D Point Cloud Based Organ Segmentation

TL;DR

This paper introduces a deep Q-learning based pipeline for 3D organ segmentation from point clouds, leveraging shape models and novel aperture features for robust, multi-resolution, and modality-independent performance.

Contribution

It presents a new reinforcement learning approach for 3D organ segmentation that incorporates shape preservation, hierarchical deformation estimation, and applicability across various organs and imaging modalities.

Findings

Robust segmentation performance on diverse challenge and clinical data.

Fast processing time of 0.3 to 2.7 seconds per organ.

Effective application across different organs and imaging protocols without transfer learning.

Abstract

We propose a novel point cloud based 3D organ segmentation pipeline utilizing deep Q-learning. In order to preserve shape properties, the learning process is guided using a statistical shape model. The trained agent directly predicts piece-wise linear transformations for all vertices in each iteration. This mapping between the ideal transformation for an object outline estimation is learned based on image features. To this end, we introduce aperture features that extract gray values by sampling the 3D volume within the cone centered around the associated vertex and its normal vector. Our approach is also capable of estimating a hierarchical pyramid of non rigid deformations for multi-resolution meshes. In the application phase, we use a marginal approach to gradually estimate affine as well as non-rigid transformations. We performed extensive evaluations to highlight the robust performance of our approach on a variety of challenge data as well as clinical data. Additionally, our method has a run time ranging from 0.3 to 2.7 seconds to segment each organ. In addition, we show that the proposed method can be applied to different organs, X-ray based modalities, and scanning protocols without the need of transfer learning. As we learn actions, even unseen reference meshes can be processed as demonstrated in an example with the Visible Human. From this we conclude that our method is robust, and we believe that our method can be successfully applied to many more applications, in particular, in the interventional imaging space.