Registration of pre-surgical MRI and whole-mount histopathology images in prostate cancer patients with radical prostatectomy via RAPSODI

TL;DR

The paper introduces RAPSODI, a robust framework for accurately registering pre-surgical MRI with whole-mount histopathology images in prostate cancer patients, enabling precise mapping of cancer extent onto MRI.

Contribution

RAPSODI is a novel three-step registration framework that effectively aligns histopathology and MRI images, accounting for tissue deformation and specimen handling variations.

Findings

RAPSODI reliably corrects rotations within ±15° and shrinkage up to 10%.

Achieved high registration accuracy with Dice coefficient of 0.98 and low boundary deviations.

Validated on 89 patients with consistent results across institutions.

Abstract

Magnetic resonance imaging (MRI) has great potential to improve prostate cancer diagnosis. It can spare men with a normal exam from undergoing invasive biopsy while making biopsies more accurate in men with lesions suspicious for cancer. Yet, the subtle differences between cancer and confounding conditions, render the interpretation of MRI challenging. The tissue collected from patients that undergo pre-surgical MRI and radical prostatectomy provides a unique opportunity to correlate histopathology images of the entire prostate with MRI in order to accurately map the extent of prostate cancer onto MRI. Here, we introduce the RAPSODI (framework for the registration of radiology and pathology images. RAPSODI relies on a three-step procedure that first reconstructs in 3D the resected tissue using the serial whole-mount histopathology slices, then registers corresponding histopathology and MRI slices, and finally maps the cancer outlines from the histopathology slices onto MRI. We tested RAPSODI in a phantom study where we simulated various conditions, e.g., tissue specimen rotation upon mounting on glass slides, tissue shrinkage during fixation, or imperfect slice-to-slice correspondences between histology and MRI. Our experiments showed that RAPSODI can reliably correct for rotations within $\pm15^{\circ}$ and shrinkage up to 10%. We also evaluated RAPSODI in 89 patients from two institutions that underwent radical prostatectomy, yielding 543 histopathology slices that were registered to corresponding T2 weighted MRI slices. We found a Dice coefficient of 0.98$ \pm $0.01 for the prostate, prostate boundary Hausdorff distance of 1.71$ \pm $0.48 mm, a urethra deviation of 2.91$ \pm $1.25 mm, and a landmark deviation of 2.88$ \pm $0.70 mm between registered histopathology images and MRI. Our robust framework successfully mapped the extent of disease from histopathology slices onto MRI.

Tables

Table 1. Table 1 : Summary of previous approaches (not exhaustive). We excluded publications with <2 subjects [ 18 ] , only synthetic data [ 19 ] , or manually intensive approaches [ 20 ] . All summarized methods require as input the in vivo pre-surgical T2 weighted MRI, digitized serial histopathology images, and the segmentation of the prostate on MRI and histopathology images; Additional input requirements are listed here; Abbreviations: TPS - Thin Plate Spline; NA - Not available

Publication	Subject #	Approach	Additional Input	Dice Coef.	Landmark Error (mm)
Park 2008 [9]	2	3D reconstruction + affine and TPS registration	block face picture, ex vivo MRI	NA	3-3.74
Chappelow 2011[10]	25	Feature Based Mutual Information + BSpline	-	NA	NA
Ward 2012 [11]	13	2D Affine + TPS Registration	Strand-shaped fiducials, Ex vivo MRI	NA	1.1
Kalavagunta 2014 [12]	35	Local affine registration	Internal landmarks, 3D Printed Molds	0.99	1.54$\pm$0.64
Reynolds 2015 [13]	6	2D TPS registration + deformable registration	Control Points, ex vivo MRI, sectioning box	0.93	3.3
Li 2017 [14]	19	Multi-Scale Representation + deformable registration	-	0.96$\pm$0.01	2.96$\pm$0.76
Losnegard 2018 [15]	12	3D histopathology reconstruction, 3D affine and deformable registration	-	0.94	5.4
Wu 2019 [16]	17	2D Rigid, TPS Registration (automatic landmarks)	ex vivo MRI, 3D printed molds	0.87$\pm$0.04	2.0$\pm$0.5
Rusu 2019 [17]	15	3D histopathology reconstruction, 2D Affine+Deformable	3D printed Molds	0.94$\pm 0.02$	1.11$\pm$0.34

Table 2. Table 2 : Data Summary: Abbreviations: T2-weighted MRI (T2w), Hematoxylin & Eosin (H&E), Relaxation Time (TR) in seconds, Echo Time (TE) in milliseconds; MRI Matrix Size: K × L × M 𝐾 𝐿 𝑀 K\times L\times M , Histology Matrix Size: W × \times H, * estimated, pseudo-whole mount: quadrants were stitched together

	Cohort C1: Internal		Cohort C2: Public [28]
Variable	MRI	Pathology	MRI	Pathology
Manufacturer: Coil type	GE: Surface	-	Siemens: Endorectal	-
Sequence/Data Type	T2w	whole-mount	T2w	pseudo-whole mount
Acquisition	TR: [3.9, 6.3];	H&E	TR: [3.7, 7.0];	H&E
Characteristics	TE: [122, 130]		TE: [107]
Number of Patients/Slices	74/1994	74/478	16/430	16/65
Matrix Size	$K,L\in [256,512]$ $M\in [24,43]$	W,H $\in$ [1663,7556]	$K,L=320$, $M\in [21,31]$	W,H $\in$ [2368,6324]
Pixel Spacing (mm)	$\in$ [0.27,0.94]	$\in$ {0.0081,0.0162}	$\in$ [0.41,0.43]	0.0072*
Distance Between Slices (mm)	$\in$ [3,5.2]	Same as MRI via 3D printed mold	4	Free hand
Annotations	Prostate, Anatomic Landmarks	Prostate, Anatomic Landmarks, Cancer	Prostate, Cancer, Urethra	Prostate, Urethra, Cancer

Table 3. Table 3 : Quantitative results for the two cohorts and aggregated or all subjects in our study.

Cohort	Dice Prostate	Haussdorff Distance (mm)	Urethra Deviation (mm)	Landmarks Deviation (mm)	Dice Cancer
C1	0.98$\pm$0.01	1.79$\pm$0.45	2.74$\pm$0.82	2.88$\pm$0.70	-
C2	0.98$\pm$0.01	1.37$\pm$0.47	3.14$\pm$1.71	-	0.53$\pm$0.18
All	0.98$\pm$0.01	1.71$\pm$0.48	2.91$\pm$1.25	2.88$\pm$0.70	0.53$\pm$0.18

Equations

$$Dice(\mathcal{H},\mathcal{M})=\frac{1}{D}\sum _{i=1}^D\frac{2\times |{\mathcal{H}}_i^{Pr}\cap {\mathcal{M}}_i^{Pr}|}{|{\mathcal{H}}_i^{Pr}|+|{\mathcal{M}}_i^{Pr}|}$$

$$\begin{array}{r}Hausdorf{f}^{Pr}(\mathcal{H},\mathcal{M})=max\{su{p}_{h\in {\mathcal{H}}^{Pr}}in{f}_{m\in {\mathcal{M}}^{Pr}}d(\mathcal{H},\mathcal{M}),\\ su{p}_{m\in {\mathcal{M}}^{Pr}}in{f}_{h\in {\mathcal{H}}^{Pr}}d(\mathcal{H},\mathcal{M})\}\end{array}$$

$$Dist({\mathcal{L}}^H,{\mathcal{L}}^M)=\frac{1}{X}\sum _{j=1}^X|{\mathcal{L}}_j^H,{\mathcal{L}}_j^M{|}_2$$

Full text

Keywords: radiology-pathology registration $|$ prostate cancer $|$ whole-mount histopathology $|$ radical prostatectomy $|$ magnetic resonance imaging

1 Introduction

Despite advances in diagnosis and treatment, prostate cancer remains the second leading cause of cancer death in American men [1]. Overdiagnosis of low-grade cancers that do not require treatment and the underdiagnosis of aggressive cancers are still a concern [2], even after the changes in the recommendation of prostate biopsy for elevated Prostate Specific Antigen (PSA). Magnetic Resonance Imaging (MRI) can help address all of these problems [3]. When MRI is normal, up to 50% of men can safely avoid prostate biopsy, thereby reducing overdiagnosis of low-grade cancer and infectious complications of biopsy. However, this is only true when MRI is interpreted by world-leading experts [4]. In practice, lack of widespread expertise and alarming levels of inter-reader variation greatly reduce the potential of MRI to revolutionize prostate cancer diagnosis [5]. Both false negatives and false positives, even when using the recommended PIRADS reporting system [6], are very common and the vast majority of men who undergo MRI still undergo biopsy. Finally, MRI has yet to supplant biopsy which is still required to confirm the presence and aggressiveness of prostate cancer [7].

In men diagnosed with prostate cancer on biopsy, radical prostatectomy remains the most common treatment [8]. The resected prostate provides a unique opportunity to correlate pre-surgical MRI with digitized histopathology images and map the exact extent of cancer from histopathology images onto MRI. Developing a large dataset of prostatectomy cases where cancer and Gleason grade is accurately mapped on MRI has two potentially transformative applications. First is helping to improve existing MRI interpretation schemes that are still affected by many false positive and false negative findings. Second, it may facilitate the development of machine learning methods to identify prostate cancer on MRI by accurately labeling of cancer for model training and validation.

Although numerous approaches for the radiology-pathology registration in the prostate have been introduced (see section “Prior Work“), these approaches have not been widely adopted and have not been carefully tested by scientists outside the developer teams. Recent publications using histopathology images as reference to improve MRI and automatically detect cancer [21, 22, 23, 24, 25] still use manual approaches to align the histopathology to MRI images, which are known to be labor-intensive and subjective. The reduced adoption of previous methods is due to the challenges associated with managing and registering the histopathology and Magnetic Resonance (MR) images, the lack of open source release of existing methods, and the time constraints associated with running these methods.

Specifically, the registration of histopathology images and prostate MRI has the following challenges. Histologic processing of the resected tissue causes artifacts, e.g., deformations, shrinkage, and tissue ripping. Some of these artifacts (e.g., deformation and shrinking) can be corrected through registration, while others (e.g. tissue ripping) are challenging to correct and may result in discarding slices when such artifacts are major. Furthermore, our method and many others [12, 13, 16] assume slice-to-slice correspondence between histopathology and MRI images, which can be improved through the use of customized 3D printed molds based on pre-operative MRI [26]. However, this approach requires a change in clinical protocol that is not present in the vast majority of institutions performing radical prostatectomy. Finally, the acquired data is different between the histopathology images and MRI. Histopathology images provide a discontinuous serial stack of 4$\mu m$ high-resolution colored images with a pixel size of 0.0005 mm separated by roughly 4 mm spaces, while MRI has a typical resolution of 0.4$\times$0.4$\times$4 mm3.

Here, we introduce the RAPSODI (RAdiology Pathology Spatial Open-Source multi-Dimensional Integration) framework for the registration of histopathology slices and pre-operative MRI. RAPSODI includes a dictionary-based data management system, a memory-efficient registration methodology and a Graphical User Interface Plugin to 3D Slicer [27]. Our registration approach relies on the 3D reconstruction of the histopathology specimen to create a digital representation of the tissue before gross sectioning. Next, RAPSODI registers corresponding histopathology and MRI slices. Finally, the optimized transforms are applied to the cancer regions outlined on the histopathology images to project those labels onto the pre-operative MRI.

We evaluated our methodology using a digital phantom study where we simulated various conditions resulting from the histologic preparation of the excised tissue, e.g., rotation of the tissue when mounting on the glass slide or shrinking of the tissue. Moreover, we tested RAPSODI in 89 prostate cancer patients that underwent radical prostatectomy from two institutions, ours and a public cohort [28]. RAPSODI is open-source and can be downloaded from https://github.com/pimed/Slicer-RadPathFusion, while the phantom data is available at https://github.com/pimed/rad-path-phantom.

1.1 Prior Work and Our Contribution

Although numerous automated approaches for the registration of radiology and histopathology images have been developed, manual approaches are still employed, even in recent publications [21, 22, 23, 24, 25]. Some manual or semi-automatic approaches utilize landmark-based registration approaches, either alone [21, 22, 25] or in combination with automated registration steps [13, 22]. These approaches are labor-intensive and require the human operator to possess expertise in both MRI and histopathology, and necessitate identification of corresponding landmarks on both modalities. Other such approaches [26, 23] employ cognitive alignment in which a radiologist with the help of a pathologist directly outlines the cancer region on MRI considering the histopathology images as reference. Such methods are tedious to apply and may be prone to underestimating the dimensions of the lesion [29] while MRI invisible lesions are hard if not impossible to outline and thereby they are often omitted from follow-up analysis. A few approaches use interactive image transformations [20], in which a user indicate scaling, rotations and translations to be applied to the images. Such approaches are also tedious to utilize and require extensive knowledge in both radiology and pathology of the prostate.

The automated registration of histopathology images with pre-surgical prostate MRI has been performed in proof-of-concept studies, which usually only include a small number of subjects, often < 20 (Table 1). Most approaches assume a slice-to-slice correspondence between the histopathology images and T2 weighted (T2w) MRI slices. Some partial correspondence commonly results from the gross sectioning of the prostate in histologic preparation which is done perpendicular to the urethra. Yet, more advanced methods have been introduced to enforce such correspondences. For example, three dimensional (3D) printed patient-specific molds [26] have been used [12, 13, 16] to help preserve the correspondences during tissue sectioning. Some studies additionally included blockface picture [9], ex vivo MRI [13, 16, 9, 11] or external fiducials [11] to help improve the accuracy of the registration. Yet, these approaches required modifications of the clinical protocols usually resulting in only a small number of subjects to be recruited for such research studies.

Once correspondences between the histopathology images and T2w MRI are identified, their registration can still be challenging, partially due to the artifacts induced by the tissue preparation. Textural features [10, 14] have been proposed, yet they may be cumbersome to use due to the high-dimensional scoring function optimization and the choice of textural features. Other approaches rely solely on image intensity to drive the deformable alignment [15, 17], but require accurate affine alignment prior to the deformable registration.

Previous work in the lung [30, 31], breast [32] or prostate [15, 17], has relied on approaches that reconstruct the sequential histopathology slices and created a 3D volume representing the histopathology specimen prior to sectioning, which facilitates the spatial registration with the 3D volumetric MRI and alleviates the need for slice correspondences. However, these methods are prone to overfitting the histopathology reconstruction due to the large number of degrees of freedom and may suffer from partial volume effects due to the missing data associated with thick MRI slices and the histopathology slice spacing.

Our approach makes the following contributions: 1) Our registration methodology combines a 3D reconstruction of the histopathology specimen with 2D affine and deformable registration of corresponding histopathology and MRI slices and was optimized for an accurate alignment, 2) Our approach was tested in a digital phantom where the ground truth is known as well as in the largest cohort considered to date in a radiology-pathology registration study, and 3) To the best of our knowledge, we are the first to release the source code for the registration of histopathology and radiology images in the prostate, which is essential to test the reproducibility and robustness of the approach while allowing the wide adaption.

Methods

Notations

Let $\mathcal{M}$ be the T2 weighted (T2w) MRI image, $\mathcal{M}:\mathcal{R}^{3}\rightarrow\mathcal{R}$ and has a matrix size $K\times L\times M$, where $K$, $L$ and $M$ represent the width, height and number of slices of the axial T2w MRI. Let $\mathcal{H}$ be the stack of histopathology slices, $\mathcal{H}:\mathcal{R}^{3}\rightarrow\mathcal{R}^{3}$, obtained by stacking 2D histopathology slices, $\mathcal{H}_{i}:\mathcal{R}^{2}\rightarrow\mathcal{R}^{3}$. The histopathology images are colored images, with Red, Green and Blue channels. The volume $\mathcal{H}$ has $W\times H\times D$ voxels with three components corresponding to the Red, Green and Blue channels, where $D\in[3,9]$ represents the number of slices, while $W$ and $H$ are the width and height of the histopathology images. The index $i$, is used to indicate either an axial slice within the MRI volume or an image in the histopathology stack. $\mathcal{M}^{Pr}$ and $\mathcal{H}^{Pr}$ represent the prostate segmentation on MRI and histopathology images, respectively.

Data Description

Our IRB approved study includes $N_{1}=73$ subjects from Stanford Hospital (Cohort C1) and $N_{2}=16$ patients from the "Prostate Fused MRI Pathology" collection, The Cancer Imaging Archive [28] (Cohort C2) (Table 2). A subset of 15 subjects from cohort C1 was previously utilized in [17]. The subjects in C1 had an MRI acquired between 2016-2019 prior to the radical prostatectomy, and the excised prostate was submitted for histologic preparation to generate whole-mount sections. The subjects in C2 (description available at https://wiki.cancerimagingarchive.net/display/Public/Prostate+Fused-MRI-Pathology), underwent radical prostatectomy and had the prostate sectioned in quadrants before being submitted for histology processing.

MRI: The MRI exams for the patients in cohort C1 were acquired using 3 Tesla scanners (MR750, GE Healthcare, Waukesha, WI) with an external 32-channel body array coil without an endorectal coil. The imaging protocol included T2 weighted MRI (T2w), diffusion weighted imaging (DWI) and derived Apparent Diffusion Coefficient (ADC), and dynamic contrast-enhanced imaging sequences. For the patients in Cohort C2 were acquired on a 3T scanner (Siemens) using an endorectal coil. The public repository provides T2w MRI and dynamic contrast-enhanced MRI for these patients. In this study, we utilized the Axial T2w MRI, which is acquired using a 2D Spin Echo protocol (Table 2)

Histopathology: The cohort C1 patients, following resection, the prostate was fixed in formalin, sectioned using a patient-specific 3D printed mold built based on the pre-surgical MRI to maintain the correspondence of histopathology slices and T2w images, and embedded in paraffin. This process is now part of our clinical standard of care. The histopathology for the patients in cohort C2, was cut without the use of 3D printed molds, but seeking gross sectioning perpendicular to the urethra. Mounting of the 5$\mu m$ thick tissue on the glass slide can result in a rotation of the histopathology slice as well as mounting with either aligned or misaligned left-right orientation. To account for this variability, an expert indicated the gross rotation angle and whether the slice requires left-right flipping. The whole-mount slices in C1 and quadrants sections in C2 were stained using Hematoxylin & Eosin (H&E) and were digitized at 20x magnification (pixel size 0.5 $\mu m$). Pseudo-whole mounts were generated for the images in C2, by stitching adjacent quadrants as described in [33].

Labels: Our expert radiologist (PG) outlined the prostate on MRI, $\mathcal{M}^{Pr}$, while our expert pathologist (CK) outlined the prostate, $\mathcal{H}^{Pr}$, and the cancer on the high-resolution scanned images of the histopathology specimen. Two hundred fifty-seven matching anatomic landmarks were picked on both histopathology and radiology images for a subset of 12 subjects, targeting 3 landmarks for each corresponding pair of histopathology and MRI slices. Examples of anatomic landmarks include benign prostate hyperplasia nodules, ejaculatory ducts, predominant glands, etc. The urethra was outlined on 22 studies in cohort C1 and the 16 studies in cohort C2. Outlining the urethra on MRI is relatively straightforward at the apex of the prostate, yet it becomes challenging towards the base. Thereby, often urethra annotations are available on the MRI from the mid-gland to apex, but lacking between the mid-gland and the prostate base. Slice correspondences between the MRI and histopathology were identified by an expert urologist and a radiology-pathology registration expert and validated by a multi-disciplinary team of radiologists, pathologists and urologists.

Radiology - Pathology Registration

Our approach is summarized in Figure 1 and described below:

1. Step 0)

Pre-Processing: We applied the prostate masks, $\mathcal{M}^{Pr}$ and $\mathcal{H}^{Pr}$ onto $\mathcal{M}$ and $\mathcal{H}$, respectively, to exclude the structures outside the prostate from image registration. The gross rotation angles or left-right flipping was applied as well. 2. Step 1)

3D Histopathology Reconstruction: We registered $\mathcal{H}_{i}$ relative each other, to ensure their 3D consistency within the 3D reconstruction, $\mathcal{H}$. We selected the middle slice $\mathcal{H}_{i},i=\frac{M}{2}$ as the first fixed image and registered $\mathcal{H}_{i-1}$ to $\mathcal{H}_{i}$, $\mathcal{H}_{i-2}$ with $\mathcal{H}_{i-1}$, etc, as well as $\mathcal{H}_{i+1}$ to $\mathcal{H}_{i}$, $\mathcal{H}_{i+2}$ with $\mathcal{H}_{i+1}$, etc. With the exception of the middle slice, all histopathology images will have a corresponding rigid transform following the registration with the adjacent slice $\mathcal{T}^{H}_{Rig}$. 3. Step 2)

2D Registration: We registered $\mathcal{M}_{i}$ with $\mathcal{H}_{i}$, for $\forall i\in[1,D]$, by optimizing rigid $\mathcal{T}^{M}_{Rig}$, affine $\mathcal{T}^{M}_{Aff}$ and deformable $\mathcal{T}^{M}_{Def}$ transforms using gradient-based approaches. The rigid and affine registrations only use the prostate masks during the optimization and applied Sum of Square Differences as scoring function. The deformable registration used Free-Form Deformations [34] to optimize the Mattes Mutual Information computed based on the image intensities. We used a multi-resolution pyramid with three layers (with shrinking factors 16, 8, and 4 respectively, and a smoothing sigma of 4, 2, and 1 respectively). The affine transform optimization was done using a gradient descent optimizer with a learning rate of 0.01 and 250-500 iterations per resolution layer, while, the deformable registration employed a LBFGSB optimizer with 10-50 iterations per resolution layer. 4. Step 3)

Mapping Cancer onto MRI: A composite transform of $\mathcal{T}^{H}_{Rig}$, $\mathcal{T}^{M}_{Rig}$, $\mathcal{T}^{M}_{Aff}$ and $\mathcal{T}^{M}_{Def}$ is applied to deform the histopathology image as well as the cancer label and anatomic landmarks into the coordinates of the T2w MRI.

Our approach was developed using the Insight Toolkit (ITK) [35]and its Simple ITK API in python. The approach is available as a 3D Slicer python plugin [27] (Figure 2) or as a stand-alone application to be run in batch mode. RAPSODI can be downloaded from https://github.com/pimed/Slicer-RadPathFusion. We measured the performance of the approach on an Intel i7-8700 CPU, 3.70GHz, 64GB Memory Computer.

Digital Phantom for Radiology-Pathology Registration

We created a digital phantom to assess the quality of the alignment during the development of RAPSODI and to evaluate its performance when ground truth exists. The phantom is used to simulate artifacts known to affect the histopathology sample. We constructed the phantom by first outlining different prostatic regions, peripheral zone, cancer, and urethra in a 3D T2w MRI (Figures 3a-c). Then, we synthesized the phantom T2 MRI by filling the segmented regions with the average intensities from the input T2 image (Figure 3d). Moreover, we created the pathology phantom based on the histopathology images already registered to the T2w MRI (data not shown), by averaging their color intensities within the segmented regions (Figures 3e-f). Our simulations included Gaussian noise on both the MRI and histopathology phantom slices.

Using the T2w and pathology phantom, we tested three conditions: 1) the influence of the rotation angle when mounting the tissue slice on the glass slide, 2) the influence of shrinkage caused by fixation of the tissue during histology processing, and 3) the influence of imperfect slice correspondences between the MRI and histopathology slices, e.g., Figures 3d-e have a perfect correspondence, while Figures 3d-f are 2mm apart from each other.

To evaluate RAPSODI, we used one or multiple conditions and evaluated different quantitative metrics. Ten experiments were run for each condition to assess the mean and variance in performance of RAPSODI. When a random rotation of $r$ was assigned to the histopathology phantom, it resulted in applying a random angle ranging between $-r$ and $r$ to each slide and running 10 experiments with different noise and random angle conditions. When rotations were applied alone, no translation or scaling were applied. When a shrinkage factor $s$ is applied, all histopathology slices are shrunk by $s$ relative to their original appearance. Moreover, along with shrinking the images, we also apply a random translation of as much as 5% relative to the entire image in either x or y directions. Thereby, the experiments that include rotation and shrinkage also include random translation, and when combined with the imperfect slice correspondences represent the closest condition to the real data.

Quantitative Evaluation

The accuracy of the radiology-histopathology registration was evaluated using the Dice similarity coefficient, which assesses the overlap of the prostate outlined on T2w MRI and the outline of the prostate from the histopathology reconstruction:

[TABLE]

where $D$ is the number of slices in the histopathology specimen, $\mathcal{H}^{Pr}_{i}$ represents the slice $i$ in the prostate segmentation on histopathology, while $\mathcal{M}^{Pr}_{i}$ represents the slice $i$ in the prostate segmentation on MRI.

Additionally, we evaluated the Hausdorff distance between the prostate boundary, to asses how far the boundary is after performing the registration:

[TABLE]

where $sup$ represents the supremum operator and $inf$ represents the infimum operators.

Moreover, we evaluated the landmark distance:

[TABLE]

where $|.|_{2}$ represents the Euclidean distance of the center of mass of the j landmark $\mathcal{L}^{H}_{j}$ on histopathology and center of mass of the j landmark $\mathcal{L}^{M}_{j}$ on MRI, while X represents the number of landmarks. Similarly, we computed the urethra distances, using per-slice correspondences, and slices where the urethra was visible on both MRI and histopathology slices.

Results

Phantom Study

The phantom study is used to assess the average performance and variability of RAPSODI under conditions known to affect the tissue during the histopathology preparation. We ran our registration approach for 480 different conditions, to estimate the trends of the evaluation metrics as well as to their variations. Figure 4 summarizes our results in which we tested the effect of the rotation of histopathology slices while mounting on glass slide (range: 0-40∘), and the effect of shrinkage (range: 0-30%) when perfect slice correspondences exist between the histopathology and the MRI images in the phantom. Our approach is able to perfectly recover rotation angles ranging between 0-20∘ or shrinkage of 0-10% when applied alone (Figure 4), indicated by the perfect $\tilde{}$ 1 dice coefficient and the sub-pixel error. When combined, either using 20% shrinkage and random rotation (Figures 4b,f,j) or using 20∘ rotation and shrinkage (Figures 4d,h,l), sub-pixel accuracy was observed for angles ranging between 0-15∘ or shrinkage of 0-5%. Beyond these conditions, RAPSODI is still able to recover induced rotation and shrinkage, yet with some misalignment as the initial starting conditions are far from the correct solution.

Moreover, the limitations of the registration may be observed when perfect correspondences are lacking between the histopathology and MRI slices (Figure 5). Not surprising, the landmark and prostate border deviation are as large as $\sim$ 4 pixels (1.6 mm), as these features are not perfectly matching. Yet we can observe the relative stability of the approach for rotations ranging 0-30∘ and all tested shrinkage factors ranging between 0-30%, as the induced rotation and shrinkage are properly recovered.

Qualitative Results

We applied RAPSODI to register the histopathology slices and T2w MRI in our radical prostatectomy cohorts of 89 patients. Figure 6 shows the qualitative results for a subject in cohort C1 that had a Dice Coefficient of 0.98 and a Hausdorff distance of 1.75 mm of the prostate border ($\sim$ 4 pixels). Figure 7 shows the same slice as Figures 6 Raw 2, with the histopathology slice shown with progressive transparency from right-left (Figure 7a) and left to right (Figure 7b) to emphasize the alignment of the two modalities. The qualitative and quantitative evaluation suggest that a good alignment was obtained for this subject.

The accurate registration allowed us to map the extent of two cancer foci with different Gleason groups (Figure 7, blue - Gleason group 2; red - Gleason Group 3). Although the higher grade cancer is visible on MRI, its MRI visible borders are smaller than the histopathology projected lesion, confirming previous work showing that MRI underestimates actual tumor size [29]. The fusion enabled the mapping of the Gleason Group 2 cancer, which is not clearly visible on MRI, and would have been otherwise difficult to outline on MRI.

Figure 8 shows a subject in cohort C2 for which the alignment of the prostate achieved a Dice coefficient of 0.98 and a Hausdorff distance of 2.50 mm on the prostate boundary. As for the results in Figures 6-7, the results for this subject are average and not outline. The five histopathology images (Figure 8 Column 1) were registered with the MRI (Figure 8 Column 2), and the cancer outline (red) was mapped onto MRI (Figure 8 Columns 3-4). The public dataset includes the cancer annotation (blue) for this subject, which was obtained using a landmark-based registration [33]. The cancer annotations obtained via RAPSODI overlaps well with the labels provided by the dataset authors, with a dice overlap of 0.58 and a Hausdorff Distance of 2.71 mm. The relatively low overlap indicated by the dice coefficient may be accounted by the relatively small size of the tumor, and the misalignment of the regions in the apex slice (Figure 8 Row 1).

Quantitative Results

An improvement in the alignment of the histopathology images and the T2w MRI can be observed across the different steps of our framework (Figure 9). Statistically significant differences in Dice coefficients and Hausdorff distances were found between the input and the results of the registration performed using RAPSODI(Mann-Whitney test is statistically significant for $\alpha<0.05$). These statistically significant differences were observed both between the input and the affine registration in step 2 as well as between the affine registration and the deformable registration in step 2, suggesting that both affine and deformable registrations are required to facilitate an accurate alignment. The urethra (Figures 9c,f) and the landmark deviations failed to show a clear trend, but we observe a moderate decrease of landmark deviation between the input, 2.94$\pm$0.54 mm, and after applying RAPSODI, 2.88$\pm$0.7 mm, however, these differences were not statistically significant..

The comparison of results from cohorts C1 and C2 indicated that RAPSODI produces consistent results, with Dice coefficients on the prostate border of 0.98, and Hausdorff distances averaging 1.36-1.78 mm (Table 3). The subjects in cohort C2 have MRIs acquired using an endorectal coil which causes larger deformations of the prostate. Thereby, the input data and affine registration results show worse alignment in cohort C2 compared to cohort C1. However, similar metrics are evaluated after the deformable registration in RAPSODI, suggesting that our approach generalizes even for larger deformations, as those induced by an endorectal coil.

Additional evaluation was possible in cohort C2, since the authors of the dataset [28] have provided the mapped cancer obtained via landmark-based registration [33]. Thereby, we compared the mapped cancer from RAPSODI with those provided by the dataset authors, and we observed a dice coefficient of $0.53\pm 0.18$ and deviation computed on the center of mass of $2.71\pm 1.31$ mm. The relatively reduced alignment of the cancer labels may be attributed to the general misalignment error, which is within 3 mm inside the prostate and 2 mm on the prostate border. This misalignment can have a significant effect on the value of the overlap evaluated via Dice coefficient for regions of small size, such as the cancer.

Due to the use of stitched histopathology images, and of endorectal coil MRI, larger deformations needed to be recovered when aligning the histopathology images to MRI in the patients in cohort C2. The pseudo-whole mounts can have stitching artifacts that are absent in the whole-mount histopathology images. For example, the stitched pseudo-whole mount images are elongated in the anterior-posterior direction, e.g. slice C1234 of patient aaa0054. The affine parameters, i.e. scales, of the registration were relaxed, in order to enable the recovery of such large anisotropic stretching. Such modifications were only required for processing two patients, aaa0054 and aaa0072.

In order to identify the optimal set of steps in our registration, we tested multiple combinations of processing steps (data not shown) for the subjects in cohort C1, e.g., skipping step 1 (histopathology reconstruction), or applying step 2 without performing the deformable registration. We found that the RAPSODI approach (Histopathology reconstruction followed by 2D affine and deformable transforms) achieves the highest accuracy, showing the highest Dice coefficients and lowest Hausdorff distances and landmark deviations compared to approaches where we skipped step 1 or the deformable registration in step 2.

Discussion

Here, we introduced the RAPSODI platform that enables the registration of histopathology and MR images in the prostate. RAPSODI first reconstructs the histopathology volume using pair-wise registration starting from the mid-gland slices towards the apex and base slices, respectively, followed by a slice-to-slice alignment between the corresponding histopathology and T2w images. The reconstruction ensures the consistent stacking of the histopathology slices relative to each other, independent of the MRI, which results in a better initialization of the histopathology slices in the registration with the MRI images.

We first evaluated RAPSODI in a digital phantom and showed that our framework can recover the rotation angles of the histopathology slices resulting from the slide mounting on glass slides when these angles are within 0-15∘ from the correct solution and with tissue shrinkage up to 10%. Correcting for large rotation angles can be achieved prior to applying RAPSODI either by using automated approaches, e.g., by aligning the major axis of the data [17], or via manual approaches where the user indicates an angle, as was done in our study. The tissue shrinks during fixation with a factor that is outside our control. The affine transform helps identify the shrinkage factor, yet the accuracy of the registration declines as the initial conditions are further away from the optimal solution. Registration errors are most apparent at the prostate apex, where the prostate size, shape and textures are reduced.

RAPSODI successfully registered histopathology images with corresponding T2w MR images in the 89 subjects (543 slices) achieving a prostate boundary error within 2 mm and an interior error within 3 mm. Through the use of the prostate segmentation during the registration, we emphasize the importance of the prostate border resulting in a better alignment compared to the interior landmarks. Moreover, picking the landmarks used for evaluation can be challenging as we sought to capture 3+ landmarks/slice, and the resolution of the MRI is relatively reduced due to the surface coil acquisition.

We acknowledge the following limitations for our approach. Although the registration approach is fully automated and does not require patient-specific parameterization for general cases without unusual artifacts, similar to existing approaches, some manual interventions are needed to either segment the prostate on both MRI and histopathology images, to identify slice correspondences between the histopathology and T2w MRI or to correct the gross rotation of the histopathology slices. Unlike other approaches, RAPSODI does not require landmark selection, but only uses them to evaluate the accuracy of registration.

The registration assumes that a slice-to-slice correspondence exists between the histopathology and MR images. While this is improved by using 3D printed molds, slice misalignment is possible due to the shrinking of the prostate during fixation and shifting in the mold during slicing. Such misalignment is occasionally observed at the base and apex of the prostate. The digital phantom allowed us to study the effect of such misalignment and showed that a perfect alignment cannot be obtained in this situation (we observed a $\sim$4 pixels error) yet the induced shrinkage and rotations are well recovered.

The registration runtime for our approach is 6-8 minutes which is limiting for a Graphical User Interface execution, yet it is acceptable when running the approach in batch mode. We investigated a computationally efficient method that skips Step 1 - reconstructing the histopathology, and runs the registration at lower resolution. The fast approach run in 3.1 minutes and achieved a Dice coefficient of 0.97$\pm$0.01, a Hausdorff distance of 2.23$\pm$0.66 mm, urethra deviation 2.81$\pm$0.73mm and landmark deviation of 2.92$\pm$0.7 mm in a 66 patient subset from Cohort C1. Although the results are slightly less accurate, the fast approach is 2-3 times faster, and is more suited to running in a Graphical User Interface.

Although our study only includes 89 patients from two institutions, to date it represents the largest study of this magnitude with data from multiple institutions, with MRI acquired either using surface or endorectal coils, and the only study to evaluate the approach in a digital histopathology-MRI prostate phantom. Compared to previous approaches outlined in Table 1, our quantitative results place us close to the method by Kalavagunta et. al. [12] in terms of Dice similarity coefficient. The latter approach relies on heavily annotated datasets that include the border of the transitional zone and the peripheral zone as well as other landmarks. Such landmarks are used to drive the registration at the interior of the prostate resulting in better landmark alignment, yet the approach is labor-intensive and requires careful examination of the data to identify matching landmarks in the pathology images and MRI, which is not trivial.

RAPSODI aims at registering the histopathology and MRI images with the sole goal of mapping the extent and grade of cancer from histopathology images onto T2 weighted MRI, thus creating careful and objective spatial labels on pre-operative MRI. Such mapping may help develop advanced image analysis tools to reliably predict prostate cancer and its aggressiveness on MRI, help improve current MRI interpretation schemes as well as help validate novel MRI protocols or other imaging techniques. Better imaging accompanied by better interpretation schemes can have great impact in reducing overdiagnosis of low-grade cancers, the underdiagnosis of aggressive cancers, and infectious complications of biopsy.

Conclusion

Our radiology-pathology registration framework, RAPSODI, allowed the alignment of histopathology slices and pre-surgical MRI, enabling the accurate mapping of the labels from histopathology onto MRI. The reconstruction of the 3D histopathology specimen followed by 2D registration of corresponding histopathology and T2w MRI slices ensured a robust alignment that provides accurate prostate cancer labels for MRI.

Acknowledgments

We thank the Department of Radiology at Stanford University, for their support for this work.