Synthesis and Inpainting-Based MR-CT Registration for Image-Guided Thermal Ablation of Liver Tumors

TL;DR

This paper introduces a rapid, multi-step image registration method combining synthesis, inpainting, and deep learning to accurately align MR and CT images for guiding liver tumor ablation procedures.

Contribution

It presents a novel fast registration framework that integrates Cycle-GAN synthesis, inpainting, and unsupervised registration networks for MR-CT image alignment.

Findings

Achieves high registration accuracy.

Operates with very fast computational speed.

Effective in guiding thermal ablation procedures.

Abstract

Thermal ablation is a minimally invasive procedure for treat-ing small or unresectable tumors. Although CT is widely used for guiding ablation procedures, the contrast of tumors against surrounding normal tissues in CT images is often poor, aggravating the difficulty in accurate thermal ablation. In this paper, we propose a fast MR-CT image registration method to overlay a pre-procedural MR (pMR) image onto an intra-procedural CT (iCT) image for guiding the thermal ablation of liver tumors. By first using a Cycle-GAN model with mutual information constraint to generate synthesized CT (sCT) image from the cor-responding pMR, pre-procedural MR-CT image registration is carried out through traditional mono-modality CT-CT image registration. At the intra-procedural stage, a partial-convolution-based network is first used to inpaint the probe and its artifacts in the iCT image. Then, an unsupervised registration network is used to efficiently align the pre-procedural CT (pCT) with the inpainted iCT (inpCT) image. The final transformation from pMR to iCT is obtained by combining the two estimated transformations,i.e., (1) from the pMR image space to the pCT image space (through sCT) and (2) from the pCT image space to the iCT image space (through inpCT). Experimental results confirm that the proposed method achieves high registration accuracy with a very fast computational speed.

Tables

Table 1. Table 1: The in-plane FOV, resolution and scanner used in acquisition of the pMR, pCT and iCT images.

	In-plane FOV	Resolution	Scanner
pMR image	$320\times 260$	$1.188\times 1.188\times 3{\text{mm}}^3$	Siemens 3.0T Skyra
pCT image	$512\times 512$	$0.7559\times 0.7559\times 3{\text{mm}}^3$	Philips Brilliance 64 CT
iCT image	$512\times 512$	$0.7559\times 0.7559\times 3{\text{mm}}^3$	Philips Brilliance 64 CT

Table 2. Table 2: Results of pMR-iCT (rigid and deformable), pMR-pCT (rigid and deformable), and pCT-iCT (deformable only) registration.

		Rigid		Deformable
		FLIRT	ANTs	ANTs	Proposed
pMR-iCT
Liver	Dice (%)	48.44$\pm$40.40	85.52$\pm$2.92	86.59$\pm$3.30	86.96$\pm$3.00
	TRE (mm)	-	52.56$\pm$9.85	5.18$\pm$2.43	4.93$\pm$2.72
pMR-pCT
Liver	Dice (%)	48.07$\pm$40.40	87.03$\pm$2.60	89.55$\pm$2.04	90.59$\pm$1.73
	TRE (mm)	-	6.63$\pm$2.73	5.59$\pm$2.01	4.67$\pm$2.00
Tumor	Dice (%)	12.88$\pm$25.76	51.10$\pm$17.13	55.34$\pm$5.70	62.45$\pm$4.66
	TRE (mm)	-	6.71$\pm$2.27	6.08$\pm$1.40	3.89$\pm$0.99
pCT-iCT
Liver	Dice (%)	-	-	87.90$\pm$5.25	88.63$\pm$5.53
	TRE (mm)	-	-	5.06$\pm$3.29	4.37$\pm$3.30

Equations

$${\mathcal{L}}_{\text{MI}}=\sum \sum p(x,y)\text{log}\frac{p(x,y)}{p(x)p(y)},$$

$$\mathcal{L}=-\mathrm{S}(I_{\text{inpCT}},I_{\text{pCT}}(\varphi ))+\mathrm{Reg}(\varphi ),$$

Full text

1 Introduction

Thermal ablation [1] elevates the temperature (55*∘–65∘* Celsius) of a focal zone in the tumor and induces irreversible cell injury and eventually tumor apoptosis and coagulative necrosis. Therefore, accurate targeting of the tumor area is critical for ablating tumor tissues only and leaving the surrounding healthy tissues intact.

CT imaging is typically used to guide the interventional procedure in thermal ablation, where pre-procedural CT (pCT) is used for planning, and intra-procedural CT (iCT) is captured during the treatment to facilitate safe placement of the ablation probe and accurate targeting of the tumor [2]. However, CT is relatively poor in tissue contrast (e.g., arteries) and is susceptible to artifacts introduced by the probe during the procedure. Therefore, high-resolution pCT and pre-procedural MR (pMR) images are typically aligned during planning and then registered onto the iCT image for more precise guidance in positioning the probe to the desired region of interest (ROI) [3, 4]. In liver tumor ablation, accuracy and speed of such an alignment are both important as it can compensate for deformations caused by patient positioning and respiratory motion without delay.

Most volumetric registration algorithms [5, 6, 7] are only feasible in the pre-procedural stage as they involve iterative yet time-consuming optimization. Moreover, they do not deal with the probe-induced artifacts. In order to overcome these challenges, we propose a fast image registration framework to align pMR images onto iCT images for guiding thermal ablation of the liver tumor. Meanwhile, our method also eliminates the probe artifacts so that they do not interfere during registration. The proposed registration framework consists of two stages:

Pre-Procedure: Rigid and deformable registrations between pMR and pCT images. We use mutual-information (MI)-based Cycle-GAN to generate sCT from pMR images to convert the cross-modality registration into a mono-modality problem. 2. 2.

Intra-Procedure: Fast deformable registration of the inpainted iCT (inpCT) image with the pCT image, using an unsupervised registration network (UR-Net).

Finally, the pMR image is aligned to the iCT image by composing the two transformations estimated in the above two stages.

2 Methods

In order to accurately and efficiently register pMR images onto iCT images for guiding thermal ablation of liver tumors, we propose a two-stage registration framework. First, we convert the cross-modality MR-CT image registration into

mono-modality registration (CT-CT) by synthesizing sCT images from pMR images through MI-based Cycle-GAN. This mono-modality rigid and deformable registration is performed by using ANTs [8]. Then, the trained UR-Net performs deformable registration between pCT and inpCT, which is inpainted by a partial-convolution (PConv)-based network from the iCT image. Thus, the pMR image can be warped onto the iCT space by composing the output transformations of the two registration stages. The pipeline of our method is shown in Fig. 1.

2.1 Pre-Procedural MR-CT Registration

MR-CT registration is challenging due to large appearance differences between the two modalities. Previous works [9] have shown that the cross-modality registration can be converted into a mono-modality registration to achieve better performance. To this end, we synthesize the sCT image from an input pMR image, which then facilitates the subsequent registration between pMR and pCT images.

The MR-to-CT synthesis is completed by using Cycle-GAN with mutual information constraint. Cycle-GAN [10], as one of the state-of-the-art image synthesis algorithms, adopts the adversarial loss given by two discriminators ($D_{\text{MR}}$ and $D_{\text{CT}}$), such that the distribution of the output images of the two generators ($G$ and $F$) is indistinguishable from that of the input images. It also uses the cycle-consistency (shown in Fig. 2(a)) to enforce the forward (MR-to-CT) and backward (CT-to-MR) syntheses to be bijective. However, Cycle-GAN fails to enforce structural similarity between the pMR and sCT images, which may lead to uncontrollable drifting of tissue/organ boundaries in the synthesized images (see red dashed contours in Fig. 2(b)).

Therefore, in addition to the cycle-consistency loss $\mathcal{L}_{\text{cyc}}$, we propose to introduce the MI loss $\mathcal{L}_{\text{MI}}$ to the generators ($G$ and $F$) to directly enforce the structural similarity between the input and synthesized images (as shown in Fig. 2(a)). The MI loss is defined as:

[TABLE]

where $p(x)$ and $p(y)$ denote the histograms of $I_{\text{MR}}$ and $G(I_{\text{MR}})$, respectively, and $p(x,y)$ refers to the joint histogram of $I_{\text{MR}}$ and $G(I_{\text{MR}})$. After synthesizing the cross-modality registration (pMR-pCT) is converted into a mono-modality registration problem (sCT-pCT). We then perform a conventional mono-modality registration using ANTs [8] to estimate the field that accounts for both rigid and deformable transformations.

2.2 Intra-procedural Registration

Due to the high computational efficiency requirement during intra-procedural stage, we propose to train a UR-Net (as shown in Fig. 3) in an unsupervised manner to perform expeditious deformable registration between pCT and inpCT images. The advantage is its fast speed by using parallel convolution on GPU during network inference. Dalca et al. have applied a similar unsupervised registration network to perform brain MR image registration [11], which shows comparable performance with the state-of-the-art optimization-based methods. In our work, the UR-Net is trained by pCT-inpCT image pairs with the loss function defined as:

[TABLE]

where $I_{\text{inpCT}}$ and $I_{\text{pCT}}$ represent the fixed and moving images, respectively. The loss function comprises of (1) similarity (S) in terms of cross-correlation ($\mathcal{L}_{\text{cc}}$) between the warped moving image ($I_{\text{pCT}}(\phi)$) and the fixed image ($I_{\text{inpCT}}$), and (2) the regularization (Reg) penalty defined in terms of L2-norm of the gradients of the deformation fields ($\text{L}2(\phi^{{}^{\prime}})$). The detailed network architecture is shown in Supplementary file.

The probe and its artifacts, which hinder the interpretation of the underlying anatomy, may cause inaccurate registration [6]. In order to remove the probe and its streak artifacts in the iCT image, we train a PConv-based [12] 3D U-Net to obtain inpCT image ($I_{\text{inpCT}}$). Specifically, given a rough mask (e.g., a bounding box or a polygon) covering the probe and its artifacts, the inpainting network can reconstruct the underlying tissues and update the mask layer-by-layer until the mask shrinks away. The inpainting is based on 3D convolution (Conv)-based U-Net architecture, while each convolution layer is replaced with partial convolution to ensure that the inpainted contents will not be affected by the probe and its streak artifacts in the mask. The probe and its streak artifacts exist in iCT images but not in pCT images. The mask can be manually drawn in the iCT images by clinicians during the procedure, or prepared in advance as part of the procedure planning.

3 Experiments and Results

3.0.1 Dataset and Pre-processing –

Thirty-nine subjects undergoing liver tumor ablation were included in our experiment. Each subject was scanned with his/her own pMR, pCT and iCT images (see Table 1 for a summary of the parameters). Livers were delineated from pMR, pCT and iCT images, respectively, while tumors were delineated from pMR and pCT images but not iCT due to probe artifacts and limited contrast. We used 11 labeled subjects for testing, and the remaining 28 subjects were used for training the three networks. Before normalizing intensities into the range [0,1], the intensities of pCT and iCT images were thresholded in the range [-800,800]. pMR images were also rigidly aligned onto pCT images. Then, all the image sizes were resampled to $256\times 256\times 128$ with isotropic voxel distances. If not stated otherwise, the same training/testing datasets were used.

3.0.2 Implementation –

Three networks were trained: (1) pre-procedural stage: MI-based Cycle-GAN network for CT synthesis, and PConv-based U-Net for probe inpainting; (2) intra-procedural stage: UR-Net for deformable registration. All these networks were implemented in Keras and trained on a single NVIDIA Titan X GPU.

3.0.3 Evaluation Metrics –

We computed the Dice ratio over ROIs and the target registration error (TRE) over several landmarks of livers and tumors, to evaluate the registration accuracy. These two metrics are widely used for evaluation of registration performance, with higher Dice ratio (or lower TRE) characterizing better registration quality.

3.1 Registration Results for pMR and iCT

We conducted pMR-iCT image registration over the test dataset by using FSL FLIRT (rigid), ANTs (rigid and deformable) and the proposed method, respectively. As shown in Fig. 4, our proposed method achieves better accuracy against ANTs in tumor region and body alignment. Quantitative results using Dice ratio and TRE are presented in Table 2. It can be observed that our method yields better performance than ANTs. The last line of Table 2 reports the computational time in intra-procedural stage of ANTs and our proposed method, and our proposed method performed the registration in several seconds. Notice that our proposed method in the intra-procedural stage includes the iCT inpainting and the pCT-iCT image registration steps, which were computed on GPU.

3.2 Pre-procedural Stage

3.2.1 Pre-procedural Stage Registration –

The Dice and TRE metrics were evaluated over pMR-pCT pairs and their corresponding sCT-pCT pairs of the testing dataset. As shown in Table 2, the registration performance of our algorithm is improved over sCT-pCT pair, especially on the target tumor region (more than 7% Dice and 1.1mm TRE improvement), which proves that our proposed synthesis algorithm can facilitate the cross-modality registration.

3.2.2 MR-to-CT Synthesis –

We extracted 2240 slices from the transverse planes of 28 unpaired pMR and pCT images and used them as the training dataset. pMR and pCT images of each subject were linearly registered. For testing, we applied the MR-to-CT generator to synthesize CT images from pMR images slice-by-slice and then concatenated the synthesized slices into 3D volumes. As shown in Fig. 2(b), our method predicted better CT-like images from pMR images.

3.3 Intra-procedural Stage

3.3.1 Intra-procedural Stage Registration –

UR-Net was trained by $28$ rigidly registered pCT-inpCT image pairs in an unsupervised manner, where inpCT images were generated by the trained PConv-base inpainting network. We compared the UR-Net registration performance with ANTs over pCT-inpCT image pairs. The Dice and TRE metrics of liver and the intra-procedural computation time were evaluated (see Table 2). It can be seen that the UR-Net yielded better performance and was more efficient. The UR-Net method predicted a deformation field in around 3 secs for a $256\times 256\times 128$ image pair.

3.3.2 Inpainting –

pCT images were used for training the inpainting network. Each image was augmented 500 times using 2 or 3 random 3D shapes (i.e., 3D balls and bars) with random locations and sizes to imitate manual mask (cf. Fig. 5(b)). In the testing stage, we compared the inpainting results quantitatively and qualitatively on the iCT images. The results in Fig. 5(a) shows that PConv-based network obtained around 7 dB PSNR improvement in the masked region than Conv-based network. Particularly, each iCT image was combined with a cubic mask ($40\times 40\times 20$) in healthy tissue region and input into PConv and Conv-based networks, respectively. The PSNR is computed over the reconstructed region with the ground-truth. For clinical cases, the results are shown in Fig. 5(c). The mask was drawn in the pre-procedural images by physicians to cover the tumor and the planned puncture pathway in the tumor-centered transverse plane. Then, an appropriate height was chosen in the intra-procedural stage to ensure that the closed 3D contour can cover the probe and its artifacts. The inference time was around 2 secs for a $256\times 256\times 128$ subject. Note that we will further quantitatively evaluate the improvement of tumor registration, and investigate the possibility of reconstructing the tumor directly by the PConv-based inpainting network.

4 Conclusion

A learning-based registration framework is proposed to align pre-procedural MR and intra-procedural CT images for image-guided thermal ablation of liver tumor. Experimental results showed that our method can efficiently and effectively overlay pMR onto iCT during ablation with high registration accuracy, compared to the state-of-the-art ANTs algorithm. We also showed that MI-based Cycle-GAN synthesis and unsupervised registration improves the overall performance.

Acknowledgement

This work was partially supported by the National Key Research and Development Program of China (2018YFC0116400) and STCSM (19QC1400600).