Real-Time Quality Assessment of Pediatric MRI via Semi-Supervised Deep Nonlocal Residual Neural Networks

TL;DR

This paper presents a semi-supervised deep neural network approach for real-time quality assessment of pediatric MRI images, achieving high accuracy with minimal annotated data and robustness to noise.

Contribution

Introduces a novel semi-supervised nonlocal residual neural network for efficient, real-time MRI quality assessment requiring limited annotated data.

Findings

High accuracy in MRI quality classification

Real-time processing speed (milliseconds per volume)

Robustness to annotation noise

Abstract

In this paper, we introduce an image quality assessment (IQA) method for pediatric T1- and T2-weighted MR images. IQA is first performed slice-wise using a nonlocal residual neural network (NR-Net) and then volume-wise by agglomerating the slice QA results using random forest. Our method requires only a small amount of quality-annotated images for training and is designed to be robust to annotation noise that might occur due to rater errors and the inevitable mix of good and bad slices in an image volume. Using a small set of quality-assessed images, we pre-train NR-Net to annotate each image slice with an initial quality rating (i.e., pass, questionable, fail), which we then refine by semi-supervised learning and iterative self-training. Experimental results demonstrate that our method, trained using only samples of modest size, exhibit great generalizability, capable of real-time (milliseconds per volume) large-scale IQA with near-perfect accuracy.

Figures

Tables

Table 1. TABLE I: Datasets for Training and Testing

Data Type	Labeled Dataset						Unlabeled Dataset
	Training Dataset			Testing Dataset
	Pass	Ques	Fail	Pass	Ques	Fail
T1w	20	20	20	25	9	6	434
T2w	20	20	20	21	6	13	380

Table 2. TABLE II: Comparison of different thresholds

$p_{\text{slice}}$	$p_{\text{volume}}$
	0.6	0.7	0.8	0.9
0.6	0.8750	0.9000	0.9000	0.8750
0.7	0.9750	1.0000	1.0000	1.0000
0.8	0.9750	1.0000	1.0000	1.0000
0.9	0.9750	1.0000	1.0000	1.0000

Table 3. (a) Threshold Comparison for T1-Weighted Images

$p_{\text{slice}}$	$p_{\text{volume}}$
	0.6	0.7	0.8	0.9
0.6	0.8750	0.9000	0.9000	0.8750
0.7	0.9750	1.0000	1.0000	1.0000
0.8	0.9750	1.0000	1.0000	1.0000
0.9	0.9750	1.0000	1.0000	1.0000

Table 4. (b) Threshold Comparison for T2-Weighted Images

$p_{\text{slice}}$	$p_{\text{volume}}$
	0.6	0.7	0.8	0.9
0.6	0.8250	0.8250	0.8750	0.8750
0.7	0.9250	1.0000	1.0000	1.0000
0.8	0.9250	1.0000	1.0000	1.0000
0.9	0.9250	1.0000	1.0000	1.0000

Table 5. TABLE III: Number of parameters (NoP), maximal dimension (MD), and time cost (TC) on GPU and CPU.

Network	NoP	MD	TC (Slice)		TC (Volume)
			GPU	CPU	GPU	CPU
CRes	4.86M	512	10.71ms	0.228s	355ms	11.635s
CRes+NRes	1.31M	256	10.35ms	0.197s	333ms	10.251s
DSRes	0.75M	512	10.58ms	0.175s	347ms	7.874s
DSRes+NRes	0.33M	256	10.01ms	0.159s	312ms	6.723s

Table 6. TABLE IV: Confusion matrices of CRes, CRes+NRes, DSRes and DSRes+NRes for T1-weighted images

Image Quality		Predicted
		Pass		Ques		Fail
		Slice	Volume	Slice	Volume	Slice	Volume
Actual	Pass	1421	23	79	2	0	0
	Ques	-	0	-	9	-	0
	Fail	0	0	3	0	357	6

Table 7. (a) CRes

Image Quality		Predicted
		Pass		Ques		Fail
		Slice	Volume	Slice	Volume	Slice	Volume
Actual	Pass	1421	23	79	2	0	0
	Ques	-	0	-	9	-	0
	Fail	0	0	3	0	357	6

Table 8. (b) CRes+NRes

Image Quality		Predicted
		Pass		Ques		Fail
		Slice	Volume	Slice	Volume	Slice	Volume
Actual	Pass	1470	24	30	1	0	0
	Ques	-	0	-	9	-	0
	Fail	0	0	6	0	354	6

Table 9. (c) DSRes

Image Quality		Predicted
		Pass		Ques		Fail
		Slice	Volume	Slice	Volume	Slice	Volume
Actual	Pass	1437	24	62	1	1	0
	Ques	-	0	-	9	-	0
	Fail	0	0	5	0	355	6

Table 10. (d) DSRes+NRes

Image Quality		Predicted
		Pass		Ques		Fail
		Slice	Volume	Slice	Volume	Slice	Volume
Actual	Pass	1500	25	0	0	0	0
	Ques	-	0	-	9	-	0
	Fail	0	0	56	0	304	6

Table 11. TABLE V: Sensitivity and specificity of CRes, CRes+NRes, DSRes and DSRes+NRes for T1-weighted images

Image Quality	Sensitivity		Specificity
	Slice	Volume	Slice	Volume
Pass	0.9473	0.9200	1.0000	1.0000
Ques	-	1.0000	-	0.9355
Fail	0.9917	1.0000	1.0000	1.0000

Table 12. (a) CRes

Image Quality	Sensitivity		Specificity
	Slice	Volume	Slice	Volume
Pass	0.9473	0.9200	1.0000	1.0000
Ques	-	1.0000	-	0.9355
Fail	0.9917	1.0000	1.0000	1.0000

Table 13. (b) CRes+NRes

Image Quality	Sensitivity		Specificity
	Slice	Volume	Slice	Volume
Pass	0.9800	0.9600	1.0000	1.0000
Ques	-	1.0000	-	0.9667
Fail	0.9833	1.0000	1.0000	1.0000

Table 14. (c) DSRes

Image Quality	Sensitivity		Specificity
	Slice	Volume	Slice	Volume
Pass	0.9580	0.9600	1.0000	1.0000
Ques	-	1.0000	-	0.9677
Fail	0.9861	1.0000	0.9993	1.0000

Table 15. (d) DSRes+NRes

Image Quality	Sensitivity		Specificity
	Slice	Volume	Slice	Volume
Pass	1.0000	1.0000	1.0000	1.0000
Ques	-	1.0000	-	1.0000
Fail	0.8444	1.0000	1.0000	1.0000

Table 16. TABLE VI: Confusion matrices of CRes, CRes+NRes, DSRes and DSRes+NRes for T2-weighted images

Image Quality		Predicted
		Pass		Ques		Fail
		Slice	Volume	Slice	Volume	Slice	Volume
Actual	Pass	1243	21	12	0	5	0
	Ques	-	1	-	4	-	1
	Fail	0	0	0	0	780	13

Table 17. (a) CRes

Image Quality		Predicted
		Pass		Ques		Fail
		Slice	Volume	Slice	Volume	Slice	Volume
Actual	Pass	1243	21	12	0	5	0
	Ques	-	1	-	4	-	1
	Fail	0	0	0	0	780	13

Table 18. (b) CRes+NRes

Image Quality		Predicted
		Pass		Ques		Fail
		Slice	Volume	Slice	Volume	Slice	Volume
Actual	Pass	1257	21	3	0	0	0
	Ques	-	2	-	4	-	0
	Fail	0	0	6	0	774	13

Table 19. (c) DSRes

Image Quality		Predicted
		Pass		Ques		Fail
		Slice	Volume	Slice	Volume	Slice	Volume
Actual	Pass	1242	21	15	0	3	0
	Ques	-	1	-	3	-	2
	Fail	0	0	6	0	774	13

Table 20. (d) DSRes+NRes

Image Quality		Predicted
		Pass		Ques		Fail
		Slice	Volume	Slice	Volume	Slice	Volume
Actual	Pass	1248	21	12	0	0	0
	Ques	-	0	-	6	-	0
	Fail	0	0	0	0	780	13

Table 21. TABLE VII: Sensitiviy and specificity of CRes, CRes+NRes, DSRes and DSRes+NRes for T2-weighted images

Image Quality	Sensitivity		Specificity
	Slice	Volume	Slice	Volume
Pass	0.9865	1.0000	1.0000	0.9474
Ques	-	0.6667	-	1.0000
Fail	1.0000	1.0000	0.9960	0.9630

Table 22. (a) CRes

Image Quality	Sensitivity		Specificity
	Slice	Volume	Slice	Volume
Pass	0.9865	1.0000	1.0000	0.9474
Ques	-	0.6667	-	1.0000
Fail	1.0000	1.0000	0.9960	0.9630

Table 23. (b) CRes+NRes

Image Quality	Sensitivity		Specificity
	Slice	Volume	Slice	Volume
Pass	0.9976	1.0000	1.0000	0.8947
Ques	-	0.6667	-	1.0000
Fail	0.9923	1.0000	1.0000	1.0000

Table 24. (c) DSRes

Image Quality	Sensitivity		Specificity
	Slice	Volume	Slice	Volume
Pass	0.9857	1.0000	1.0000	0.9474
Ques	-	0.5000	-	1.0000
Fail	0.9923	1.0000	0.9976	0.9259

Table 25. (d) DSRes+NRes

Image Quality	Sensitivity		Specificity
	Slice	Volume	Slice	Volume
Pass	0.9905	1.0000	1.0000	1.0000
Ques	-	1.0000	-	1.0000
Fail	1.0000	1.0000	1.0000	1.0000

Table 26. TABLE VIII: Effectiveness of semi-supervised learning

Image Quality		Predicted			Sensitivity	Specificity
		Pass	Ques	Fail
Label	Pass	319	1	0	0.9969	0.9561
	Ques	5	79	0	0.9405	0.9971
	Fail	0	0	30	1.0000	1.0000

Table 27. (a) T1-weighted images

Image Quality		Predicted			Sensitivity	Specificity
		Pass	Ques	Fail
Label	Pass	319	1	0	0.9969	0.9561
	Ques	5	79	0	0.9405	0.9971
	Fail	0	0	30	1.0000	1.0000

Table 28. (b) T2-weighted images

Image Quality		Predicted			Sensitivity	Specificity
		Pass	Ques	Fail
Label	Pass	169	5	0	0.9713	0.9903
	Ques	2	145	0	0.9864	0.9831
	Fail	0	0	59	1.0000	1.0000

Equations

$$r_i=\frac{1}{C(x_i)}\sum _{\forall j}f(x_i,x_j)g(x_j),x_i,x_j\in {\mathbb{R}}^c$$

$$f(x_i,x_j)=\exp \left[{\varphi }^T(x_i)\psi (x_j)\right],$$

$$\mathcal{L}(p_t)=-{\alpha }_t(1-p_t{)}^{\kappa }\log (p_t)+\frac{\lambda }{2n_w}\sum _w\Vert w{\Vert }_2^2,$$

Full text

I Introduction

Structural magnetic resonance imaging (sMRI) is widely used for brain morphological analysis due to its high spatial-resolution details of anatomical structures. However, sMRI is susceptible to image artifacts caused for instance by eye and head motion, hemodynamic changes, and magnetic field inhomogeneities [1]. Among these artifacts, motion artifacts are particularly prevalent when scanning pediatric subjects. As poor-quality images may bias subsequent analysis and result in incorrect conclusions, it is vital to correctly identify problematic images and exclude them from analysis.

Image quality assessment (IQA) is an important step to determine whether the acquired data are usable and whether a re-scan is necessary. IQA can be performed subjectively by a human rater or objectively by a computer algorithm. The most commonly used subjective quality ratings can be grouped into two categories, i.e.,

1. score-based rating, where a visual quality score metric, such as Mean Opinion Score (MOS) [2], is used for quality grading; and

2. class-based rating, where a visual quality class spectrum, such as Excellent/Very Good/Good/Fair/Poor/Unusable [3] or Pass (Excellent to Very Good)/Questionable (Good to Fair)/Fail (Poor to Unusable) [4], is used for quality grading.

Subjective IQA with visual inspection, even when carried out by experienced radiologists, is time-consuming, labor-intensive, costly, and error-prone [5]. Therefore, a reliable, accurate, and fully-automated objective IQA of sMRI is highly desirable.

Based on the availability of a reference image, objective IQA can be grouped into three categories, i.e.,

(i) full-reference IQA (FR-IQA), which requires a pristine image as reference;

(ii) reduced-reference IQA (RR-IQA), which requires partial information from a reference image; and

(iii) no-reference IQA (NR-IQA) or blind IQA, which requires no reference image.

FR-IQA measures the quality of an image by comparing it with a reference using some evaluation metrics. On the other hand, RR-IQA uses only a limited number of features extracted from a reference [6] to provide a near FR-IQA performance. However, FR-IQA and RR-IQA have limited practical application as full or partial information from a pristine reference image is not always available. For quality assessment without any reference image, as in our case, NR-IQA [7] is needed. Currently, most NR-IQA methods [8, 9, 10] are designed for natural 2D images. MR images are typically 3D with intensity distributions and artifacts that are very different from natural images.

Recently, deep neural networks (DNNs), particularly convolutional neural networks (CNNs), have demonstrated great potential for IQA [11]. Instead of hand-crafted features, CNNs automatically learn image features that are pertinent to IQA. However, the performance of these deep learning methods generally depends on a large number of correctly labeled training samples, which are typically lacking for medical images, as labeling involves a huge amount of effort from experts. For example, it is labor-intensive and time-consuming to annotate quality scores for all the image slices in MR images to train a slice-wise IQA network. Thus, annotation is typically performed volume-wise, where each MR volume is associated with a single quality label. This is however inaccurate since each volume might contain a mix of good or bad slices.

In this paper, we address the above issues by introducing a deep learning based slice- and volume-wise IQA method that is robust to annotation errors and requires only a small amount of annotated images for training. To the best of our knowledge, this is the first work on deep learning based slice- and volume-wise NR-IQA of sMRI with explicit consideration of limited training samples and labeling noise. The key features of our method are summarized as follows:

Our method consists of a nonlocal residual neural network (NR-Net) for slice-wise IQA and a random forest to agglomerate the slice IQA results for volume-wise IQA. We train our NR-Net using the slices of annotated image volumes, effectively increasing training sample size. 2. 2.

We employ depthwise separable residual (DSRes) blocks [12] and nonlocal residual (NRes) blocks [13] to construct the NR-Net. Compared to residual networks with standard convolutions, the computation-reduction property of the DSRes block and also the information fusion property of the NRes block make NR-Net much lighter, hence allowing fast real-time IQA. 3. 3.

We utilize semi-supervised learning to deal with the scenario where we have a small amount of labeled data but a large amount of unlabeled data. With a small number of labeled samples, we pre-train the NR-Net to label the slices of unlabeled volumes, which are then used to re-train the NR-Net. 4. 4.

We use an iterative self-training mechanism to prune or relabel unreliable labels to improve training effectiveness. Self-training is iterated until convergence.

The rest of this paper is organized as follows. In Section II, we briefly review related work on NR-IQA. Section III describes the NR-Net architecture, semi-supervised learning, and slice/volume self-training. We present the experimental results in Section IV and conclude this paper in Section V.

II Related Work

II-A Deep Learning NR-IQA

Deep learning NR-IQA methods can be either score-based or class-based [14]. Score-based methods predict continuous-valued quality ratings, typically by treating quality assessment as a regression problem. Deep learning NR-IQA methods in this category (mostly for 2D natural images) [15, 16, 17, 18, 19] are typically trained image-wise and thus a large number of annotated images are needed. Patch-wise methods [20, 21, 22, 23] use image patches as training samples, thus effectively increasing sample size.

Class-based methods [24, 25, 26, 27, 28, 29, 30, 31, 32, 33] predict discrete-valued quality categories, often associated with human perception on image quality. Quality annotation in structural MRI is carried out manually via visual inspection. There is no reliable metric that can be used to provide continuous-valued quality ratings. Therefore, our IQA method is based on classification instead of regression to predict quality categories of slices and volumes.

II-B Label Noise

Expert annotations are not always accurate, thus leading to “label noise”. Methods to cope with label noise can be divided into three categories [34]:

1. designing a robust loss,

2. cleansing noisy data, and

3. modeling the label noise distribution.

First, loss function can be reweighted or rectified to be robust to label noise [35, 36, 37].

Second, label noise can be “cleaned” before training, by either relabeling or removing mislabeled data [38, 39]. Removing mislabeled data, while shown to be effective [38, 39], reduces sample size and hence degrade training effectiveness. Data cleansing is also affected by data imbalance [40], where minority classes may more likely be entirely removed. In comparison, relabeling mislabeled data maintains the sample size, but incorrect relabeling may lead to performance degradation.

Third, if some information about the label noise is available, it is possible to predict the distribution of label noise and use it to improve the classifier. Explicitly modeling or learning the label noise distribution [41, 42, 43, 44] allows noisy labels to be detected and discarded during training. However, this approach depends on the accuracy of the label noise model, increases the complexity of learning algorithms, and may result in overfitting due to additional model parameters.

III Architecture

Fig. 1 shows an overview of our sMRI IQA method, which consists of two stages, i.e., slice assessment stage and volume assessment stage. The slice assessment stage is designed to predict the quality rating of each slice and is trained using semi-supervised learning and slice self-training. The volume assessment stage, trained using volume self-training, evaluates the quality rating of each volume by ensembling the quality ratings of slices belonging to this volume. Details of our method are described next.

III-A Slice Quality Assessment Network

Our slice quality assessment network is designed with both accuracy and speed in mind. Fig. 2 shows the proposed network, NR-Net, which consists of four types of network blocks, i.e., convolution (Conv), depthwise separable residual (DSRes), nonlocal residual (NRes), and classifier blocks. The Conv and DSRes blocks extract low- and high-level features, respectively, whereas the NRes block [13] computes the response function at each position as a weighted summation of features from different spatial locations. The classifier block (realized with a convolutional layer, global average pooling, and a softmax activation function) outputs three probability values indicating whether a slice is “pass”, “questionable”, or “fail”. The slice is finally annotated with the label associated with the highest probability.

III-A1 DSRes Block

We construct the DSRes block by integrating depthwise separable convolution (DSConv) layers in the NR-Net (blue in Fig. 2). Fig. 3(a) shows how a standard convolution layer filters all input channels and combine the results in an output channel. In comparison, a DSConv layer filters using a combination of depthwise convolution and pointwise convolution. Specifically, depthwise convolution performs channel-wise spatial convolution and concatenation, as shown in Fig. 3(b). Pointwise convolution subsequently projects the channels by depthwise $1\times 1$ convolution onto a new channel, as shown in Fig. 3(c).

Based on Fig. 3, we show that DSConv is computationally more efficient than standard convolution. Given a $c$-channel $h\times w$ input feature map, a $c^{\prime}$-channel $h^{\prime}\times w^{\prime}$ output feature map, and a $d\times d$ kernel, the computational cost (CC) of a standard convolution layer is given by $\textrm{CC}_{\textrm{StdConv}}=cc^{\prime}d^{2}h^{\prime}w^{\prime}$. In contrast, the computational cost of a DSConv layer with the same input and output, as shown in Fig. 3(b) and (c), is given by $\textrm{CC}_{\textrm{DSConv}}=cd^{2}h^{\prime}w^{\prime}+cc^{\prime}h^{\prime}w^{\prime}$. The computational reduction factor (CRF) of replacing standard convolution with DSConv is therefore given as $\textrm{CRF}_{\textrm{conv}}=1/c^{\prime}+1/{d^{2}}$. Furthermore, the CRF of replacing a residual block constructed by standard convolutions with a DSRes block is $\textrm{CRF}_{\textrm{DSRes}}\approx 1/c^{\prime}+3/(2d^{2}+1)$ with $c^{\prime}=2c$. We use a $3\times 3$ convolution kernel ($d=3$) in each DSRes block, so that the computational cost of a DSRes block is 6 to 7 times smaller than that of a residual block with standard convolution.

III-A2 NRes Block

The quality of an image slice is determined by the existence of artifacts at different spatial locations. We employ a nonlocal residual (NRes) block [13] in our network (orange in Fig. 2) to capture information from potentially distant locations. Given a $c$-channel $h\times w$ input feature map $x$, the output at the $i$-th location, $r_{i}\in\mathbb{R}^{c}$, of the NRes block is computed as the weighted sum of all features:

[TABLE]

where the weight function $f(\cdot,\cdot)$ encodes the pairwise similarity between feature vectors at locations $i$ and $j$, $g(\cdot)$ computes a representation of a feature vector, and $C(x_{i})=\sum_{\forall j}f(x_{i},x_{j})$ is a normalization factor. In this work, the weight function $f(x_{i},x_{j})$ is defined as

[TABLE]

where $\phi(\cdot)$ and $\psi(\cdot)$ are unary kernel functions that are implemented with $1\times 1$ convolution kernels, thus making $\frac{1}{C(x_{i})}f(x_{i},x_{j})$ a softmax function. The NRes block is incorporated in the NR-Net using a residual form.

NR-Net is lighter with less parameters by employing DSRes blocks and captures long-range dependencies between features regardless of their positional distances. Batch normalization and global average pooling are used for regularization without dropout to speed up training.

III-B Semi-Supervised Learning

We employ semi-supervised learning to make full use of a small amount of labeled data and a large amount of unlabeled data. This is done by progressive annotation of unlabeled slices to retrain the network (orange box in Fig. 1). We begin by utilizing the NR-Net pre-trained with the labeled dataset to predict the “pass”, “questionable”, or “fail” probabilities of the slices of the unlabeled volumes. Each slice is annotated with the quality rating associated with the maximal probability. The labeled slices are then merged into the original labeled dataset to be used for retraining of the NR-Net.

III-C Slice Self-Training

To deal with noisy labels, we propose a slice self-training method to sample “clean” data for training. This involves iterative slice relabeling/pruning and NR-Net retraining (green box in Fig. 1). Slices are quality-predicted using the pre-trained NR-Net and then selected based on the following conditions:

1. Predicted labels that are identical to those predicted in the previous iteration;

2. Predicted labels with high-confidence, i.e., maximal probabilities beyond a threshold.

The labels of the selected slices are replaced with the predicted labels. Slices that do not meet these criteria are pruned from the training dataset. NR-Net is then retrained for the next iteration until accuracy improvement is minimal.

III-D Volume Self-Training

Random forest, effective even with small training datasets [45], is employed to predict the volumetric quality based on the slice quality ratings. Both labeled and unlabeled volumes are utilized to train the random forest. The initial quality ratings of unlabeled volumes are determined based on the following rules:

1. “Pass” if more than 80 percent of the slices in the volume are labeled as “pass”;

2. “Fail” if more slices are labeled as “fail” than “pass” or “questionable”;

3. “Questionable” if otherwise.

Similar to slice self-training, volume self-training involves iterative volume relabeling/pruning, random forest retraining (blue box in Fig. 1). The input to the random forest is the slice quality ratings predicted using the NR-Net. To reduce the influence of label noise, volumes satisfying the following criteria are retained:

1. Predicted labels identical to those predicted in the previous iteration;

2. Predicted labels with high-confidence, i.e., maximal probabilities beyond a threshold.

The labels of the selected volumes are replaced by the predicted labels. Volumes that do not meet these criteria are pruned from the training dataset. The random forest is then retrained for the next iteration until accuracy improvement is minimal.

IV Experimental Results

IV-A Training

We evaluated our automatic IQA framework on T1- and T2-weighted MR images of pediatric subjects from birth to six years of age [46]. The images were separated into three datasets:

1. Training dataset with noisy labels (annotated volume-wise by an expert);

2. Testing dataset with reliable labels (annotated volume-wise by multiple experts); and

3. Unlabeled dataset.

See Table I for a summary. Note that, as in practical scenarios, the unlabeled dataset is much larger. Fig. 4 shows examples of slices labeled as “pass” (no/minor artifacts), “questionable” (moderate artifacts), and “fail” (heavy artifacts).

In total, 3600, 2400, and 26040 axial slices were extracted, respectively, from the 60 T1-weighted training volumes, 40 T1-weighted testing volumes, and 434 T1-weighted unlabeled volumes. On the other hand, 3600, 2400, and 22800 axial slices were extracted, respectively, from the 60 T2-weighted training volumes, 40 T2-weighted testing volume, and 380 T2-weighted unlabeled volumes. Each slice was uniformly padded to 256$\times$256, min-max intensity normalized, and labeled according to the volume it belongs to. For both slice and volume assessments, the T1-/T2-weighted slice/volume training sets were divided into training and validation subsets with a ratio of 9:1.

To implement NR-Net, we employed Keras with Tensorflow backend. To avoid overfitting, the data were augmented via rotation and horizontal flipping. In addition, $L_{2}$ regularization was used for the Conv and DSConv layers. To deal with data imbalance, a multi-class balanced focal loss [47] with $L_{2}$ regularization was used:

[TABLE]

where $p_{t}$, $t=1,2,3$, are the predicted probabilities for “pass”, “questionable”, and “fail”. $\kappa\geq 0$ is a focusing parameter, $w$’s are the weight matrices of NR-Net, $\lambda=0.01$ is a tuning parameter for $L_{2}$ regularization, and $n_{w}$ is the number of weight matrices. Here, the class weights $\alpha_{t}=\max(N_{1},N_{2},N_{3})/N_{t}$ are used for balancing the contributions of imbalanced datasets. $N_{t}$ is the number of slices in association with the $t$-th class. The RMSprop optimizer was employed to learn the network weights, with the initial learning rate set to $1\times 10^{-5}$ and the decay rate set to $5\times 10^{-8}$. Slice self-training was repeated twice.

Random forest for volume prediction was implemented using Scikit-Learn. The random forest consisted of 50 trees with entropy as a measure of quality. Balanced class weights were used to counter volume data imbalance. Volume self-training was repeated twice.

IV-B Determination of Thresholds

Utilizing semi-supervised learning (Section IV-A), we obtained 29640 and 26400 T1-weighted and T2-weighted slices labeled with “pass”, “questionable” or “fail”. These slices were then used for slice and volume self-training, considering 4 probability thresholds $p_{\text{slice}}$, $p_{\text{volume}}\in\{0.6,0.7,0.8,0.9\}$. Table III(b) shows that the prediction accuracy is stable when the threshold is 0.7 and greater. Thus, in subsequent experiments, we set both $p_{\text{sliice}}$ and $p_{\text{volume}}$ to 0.8.

IV-C Ablation Study

To verify the effectiveness of DSRes and NRes blocks, we compared modified versions of NR-Net:

• •

Convolution residual (CRes) network, which substitutes the DSRes blocks and NRes block with CRes blocks with channel numbers 128, 256 and 512;

• •

CRes+NRes network, which substitutes DSRes blocks with CRes blocks with channel numbers 128 and 256;

• •

DSRes network, which substitutes the NRes block with DSRes block with channel number 512.

Note that NR-Net is a DSRes+NRes network. These networks were trained in a manner similar to the NR-Net.

IV-C1 Computational Efficiency

We compared the computational efficiency of NR-Net (DSRes+NRes network) with the other networks using several metrics: Number of parameters (NoP), the maximal dimension (MD), GPU and CPU time costs (TCs) of slice and volume. An NVIDIA GeForce GTX 1080Ti (GPU) and an Intel i7-8700K (CPU) were used for evaluation.

As shown in Table III, the NoP and TCs of NR-Net (DSRes+NRes network) are much smaller than the other networks (e.g., 14$\times$ times smaller than CRes) and the CPU TC of each slice and volume is $\sim$1.5 times lower. Compared with CRes and CRes+NRes, the NRes block reduces the maximal dimension from 512 to 256 and reduces the NoP by over three times. The GPU and CPU TCs of slice and volume are reduced by over 3%, and particularly both the CPU slice and volume TC are reduced by over 11.8%. Compared with the NoP and TCs of DSRes, the NoP of DSRes+NRes is reduced by over 2 times. The GPU and CPU TCs of slice and volume are all reduced by over 5%, particularly the CPU volume TC, which is reduced by over 15%. CRes and DSRes have the same maximal dimension. The NoP of DSRes is over 6 times smaller than CRes. Comparing the TCs of CRes and DSRes, the reduction of TCs on GPU is only $\sim$2%, but on CPU it is quite significant, i.e., $\sim$23% and $\sim$33% reduction of slice and volume TCs, respectively. Similarly, comparing CRes+NRes and DSRes+NRes, the NoP reduction reaches almost 4 times and the reduction of GPU and CPU TCs reaches over 3% and 20%, respectively. The analysis above shows that DSRes block and NRes block improve the computational efficiency of the overall network, making it suitable for real-time IQA.

IV-C2 Network Efficiency

Tables V(d) and VI(d) show the confusion matrices, along with the sensitivity and specificity of the different methods for T1-weighted testing images. The corresponding results for T2-weighted testing images are shown in Tables VII(d) and VIII(d). The detailed IQA results for the testing T1- and T2-weighted images are shown in Figs. 5 and 6, respectively.

It can be observed from Tables V(d) and VI(d) that the proposed method yields the best volume IQA performance than the other methods in terms of sensitivity and specificity. The slice prediction results in Tables V(d) and VI(d) and Fig. 5 show that the sensitivity of “fail” slices for DSRes+NRes method is lower than the other methods. This however does not affect the sensitivity of volume IQA. Similar conclusions can be drawn for T2-weighted images from Tables VII(d) and VIII(d) and Fig. 6, showing that the proposed method consistently yields the best performance for volume IQA.

IV-D Discordant Case Analysis

It can be observed from Table V(d) and Fig. 5-D that “fail” T1-weighted slices can be mistakenly predicted as “questionable”. By retrospectively inspecting all “questionable” and “fail” T1-weighted training slices, we found that this confusion is caused by the existence of ringing and motion artifacts in most “questionable” and “fail” slices, as shown in Fig. 7. Similarly, it can be observed from Table VII(d) and Fig. 6-D that “pass” T2-weighted slices can sometimes be mispredicted as “questionable”. Subtle degradation such as contrast reduction and local fuzziness can confuse the IQA network and causes mispredictions as shown in Fig. 8.

IV-E Performance Comparison

To verify the effectiveness of the proposed IQA method, we compared it with several deep learning IQA methods:

• •

Pretrained and fine-tuned [31]: InceptionV4 [48], ResNet [49]

• •

Trained from scratch: CNN [29]

For volume-wise IQA, our random forest is applied to the slice-wise quality ratings given by these methods.

Figs. 9 and 10, respectively, shows the evaluation results in terms of sensitivity (SEN), specificity (SPE), and accuracy (ACC) for slice- and volume-wise IQA of the T1- and T2-weighted testing data. From Figs. 9 and 10, our method significantly outperforms all compared methods for all evalution metrics except sensitivity to “fail” T1-weighted slices (i.e., SEN-S-F).

IV-F Effectiveness of Semi-Supervised Learning

We evaluated whether semi-supervised learning is able to correctly assess of the large amount of unlabeled data. The unlabeled data were annotated by an expert and the labels were compared with the predictions given by our method. As can be observed from Table IX(b), the automated and predicted volume IQA results are largely consistent. This implies that our method is able to effectively harness the large amount of unlabeled data for increasing sample size and hence improving network training.

V Discussion and Conclusion

In this paper, we have proposed a deep learning method for IQA of pediatric T1- and T2-weighted MR images. The network consists of a nonlocal residual network (NR-Net) for slice IQA and a random forest for volume IQA. Our method requires only a small amount of quality-annotated images for pre-training the NR-Net for initial automated annotation of a large amount of unlabeled dataset. This is refined via subsequent self-training processes. Our method copes with label noise effectively, affords tolerance to inevitable rater errors, minimizes the amount of required label data, and reduces the number of manual labeling hours. Experimental results verify that the proposed method yields near perfect IQA accuracy at a very low computational cost.

The proposed two-stage method is flexible and can be adjusted according to data availability. When only slice labels are available, NR-Net can be trained for slice IQA. When only volume labels are available, the two-stage framework can be used to identify slices that are useful for network training for slice and volume IQA. When a large labeled dataset is available, the semi-supervised learning process can be omitted. When a small labeled dataset and a large unlabeled dataset are available, the whole proposed framework can be utilized.

The proposed method can be combined with quality enhancement methods, such as artifact removal methods, for integrated quality control. The proposed framework can also be adapted to other IQA tasks, for instance, the IQA of diffusion MRI, where the size of unlabeled data is typically larger than that of labeled data, and where label noise is a significant issue. Future efforts will be directed towards such extension of our method.

Acknowledgement

This work was supported in part by NIH grants (AG053867, EB006733, MH117943, MH104324, and MH110274) and the efforts of the UNC/UMN Baby Connectome Project Consortium.