Deep learning in radiology: an overview of the concepts and a survey of the state of the art

TL;DR

This paper provides an overview of deep learning concepts and surveys its recent applications in radiology, highlighting its potential to transform medical image analysis and clinical practice.

Contribution

It offers a comprehensive review of deep learning methods applied to radiology, including basic concepts, research organization, and future opportunities and challenges.

Findings

Deep learning has achieved performance comparable to or exceeding human experts in image analysis.

Numerous studies have applied convolutional neural networks to various radiological tasks.

The field is rapidly evolving with significant potential for clinical integration.

Abstract

Deep learning is a branch of artificial intelligence where networks of simple interconnected units are used to extract patterns from data in order to solve complex problems. Deep learning algorithms have shown groundbreaking performance in a variety of sophisticated tasks, especially those related to images. They have often matched or exceeded human performance. Since the medical field of radiology mostly relies on extracting useful information from images, it is a very natural application area for deep learning, and research in this area has rapidly grown in recent years. In this article, we review the clinical reality of radiology and discuss the opportunities for application of deep learning algorithms. We also introduce basic concepts of deep learning including convolutional neural networks. Then, we present a survey of the research in deep learning applied to radiology. We organize the studies by the types of specific tasks that they attempt to solve and review the broad range of utilized deep learning algorithms. Finally, we briefly discuss opportunities and challenges for incorporating deep learning in the radiology practice of the future.

Figures

Full text

0.1 Terminology

To understand deep learning, it is helpful to first understand the related concepts of artificial intelligence and machine learning. Artificial intelligence is a set of computer algorithms that are able to perform complicated tasks or tasks that require intelligence when conducted by humans. Machine learning is a subset of artificial intelligence algorithms which, to perform these complicated tasks, are able to learn from provided data and do not require pre-defined rules of reasoning. The field of machine learning is very diverse and has already had notable applications in medical imaging [Erickson2017]. Deep learning is a sub-discipline of machine learning that relies on networks of simple interconnected units. In deep learning models, these units are connected to form multiple layers that are capable of generating increasingly high level representations of the provided input (e.g., images). Below, in order to explain the architecture of deep learning models, we introduce the artificial neural network in general and one specific type: the convolutional neural network. Then, we detail the ”learning” process of these networks, which is the process of incorporating the patterns extracted from data into deep neural networks.

0.2 Artificial Neural Networks

Artificial neural networks (ANNs) are machine learning models based on basic concepts dating as far as 1940s, significant development in 1970s and 1980s and a period notable popularity in 1990s and 2000s followed by a period of being overshadowed by other machine learning algorithms. ANNs consist of a multitude of interconnected processing units, called neurons, usually organized in layers. A traditional ANN typically used in the practice of machine learning contains 2 to 3 layers of neurons. Each neuron performs a very simple operation. While many neuron models were proposed, a typical neuron simply multiplies each input by a certain weight, then adds all the products for all the inputs and applies a simple nondecreasing function at the end. Even though each neuron performs a very rudimentary calculation, the interconnected nature of the network allows for the performance of very sophisticated calculations and implementation of very complicated functions.

0.3 Convolutional Neural Networks

Deep neural networks are a special type of an ANN. The most common type of a deep neural network is a deep convolutional neural network (CNN). A deep convolutional neural network, while inheriting the properties of a generic ANN, has also its own specific features. First, it is deep. A typical number of layers is 10\nobreakdash-30 but in extreme cases it could exceed 1 000. Second, the neurons are connected such that multiple neurons share weights. This effectively allows the network to perform convolutions (or template matching) of the input image with the filters (defined by the weights) within the CNN. Other special feature of CNNs is that between some layers, they perform pooling which makes the network invariant to small shifts of the images. Finally, CNNs typically use a different activation function of the neurons as compared to traditional ANNs.

Figure 1 shows an example of a small architecture for a typical CNN. One can see that the first layers are the convolutional ones which serve the role of generating useful features for classification. Those layers can be thought of as implementing image filters, ranging from simple filters that match edges to those that eventually match much more complicated shapes such as eyes, or tumors. Further from the network input are so called fully connected layers (similar to traditional ANNs) which utilize the features extracted by the convolutional layers to generate a decision (e.g., assign a label). A variety of deep learning architectures have been proposed, often driven by characteristics of the task at hand (e.g., fully convolutional neural networks for image segmentation). Some of these are described in more detail in the section of this paper that reviews the current state of the art.

0.4 The learning process in convolutional neural networks

Above, we described general characteristics of traditional neural networks and deep learning’s flagship: the convolutional neural network. Next, we will explore how to make those networks perform useful tasks. This is accomplished in the process referred to as learning or training. The learning process of a convolutional neural network simply consists of changing the weights of the individual neurons in response to the provided data. In the most popular type of learning process, called supervised learning, a training example contains an object of interest (e.g., an image of a tumor) and a label (e.g., the tumor’s pathology: benign or malignant). In our example, the image is presented to the network’s input, and the calculation is carried out within the network to produce a prediction based on the current weights of the network. Then, the network’s prediction is compared to the actual label of the object and an error is calculated. This error is then propagated through the network to change the values of the network’s weights such that the next time the network analyzes this example, the error decreases. In practice, the adaptation of the weights is performed after a group of examples (a batch) are presented to the network. This process is called error backpropagation or stochastic gradient descent. Various modifications of stochastic gradient descent algorithm have been developed [ruder2016overview]. In principle, this iterative process consists of calculations of error between the output of the model and the desired output and adjusting the weights in the direction where the error decreases.

The most straightforward way of training is to start with a random set of weights and train using available data specific to the problem being solved (training from scratch). However, given the large number of parameters (weights) in a network, often above 10 million, and a limited amount of training data (common in medical imaging), a network may overfit to the available data, resulting in poor performance on test data. Two training methods have been developed to address this issue: transfer learning [yosinski2014transferable] and off-the-shelf features (a.k.a. deep features) [sharif2014cnn]. A diagram comparing training from scratch with transfer learning and off-the-shelf deep features is shown in Figure 2.

In the transfer learning approach, the network is first trained using a different dataset, for example an ImageNet collection. Then, the network is ”fine-tuned” through additional training with data specific to the problem to be addressed. The idea behind this approach is that solving different visual tasks shares a certain level of processing such as recognition of edges or simple shapes. This approach has been shown successful in, for example, prediction of survival time from brain MRI in patients with glioblastoma tumor [ahmed2017fine] or in skin lesion classification [Esteva]. Another approach that addresses the issue of limited training data is the deep ”off-the-shelf” features approach which uses convolutional neural networks which have been trained on a different dataset to extract features from the images. This is done by extracting outputs of layers prior to the network’s final layer. Those layers typically have hundreds or thousands of outputs. Then, these outputs are used as inputs to ”traditional” classifiers such as linear discriminant analysis, support vector machines, or decision trees. This is similar to transfer learning (and is sometimes considered a part of transfer learning) with the difference being that the last layers of a CNN are replaced by a traditional classifier and the early layers are not additionally trained.

0.5 Deep learning vs ”traditional” machine learning

Increasingly often we hear a distinction between deep learning and ”traditional” machine learning (see Figure 3). The difference is very important, particularly in the context of medical imaging. In traditional machine learning, the typical first step is feature extraction. This means that to classify an object, one must decide which characteristics of an object will be important and implement algorithms that are able to capture these characteristics. A number of sophisticated algorithms in the field of computer vision have been proposed for this purpose and a variety of size, shape, texture, and other features were extracted. This process is to a large extent arbitrary since the machine learning researcher or practitioner often must guess which features will be of use for a particular task and runs the risk of including useless and redundant features and, more importantly, not including truly useful features. In deep learning, the process of feature extraction and decision making are merged and trainable, and therefore no choices need to be made regarding which features should be extracted; this is decided by the network in the training process. However, the cost of allowing the neural network to select its own features is a requirement for much larger training data sets.