Deep Residual Learning for Instrument Segmentation in Robotic Surgery

TL;DR

This paper introduces a deep residual learning approach that improves binary segmentation and extends it to multi-class segmentation for robotic surgical instruments, enhancing accuracy in minimally invasive surgery tasks.

Contribution

It advances surgical instrument segmentation by applying deep residual networks and dilated convolutions, and extends the method to multi-class segmentation of different instrument parts.

Findings

Improved segmentation accuracy over previous methods

Effective multi-class segmentation of surgical tools

Validated on MICCAI Endoscopic Vision Challenge dataset

Abstract

Detection, tracking, and pose estimation of surgical instruments are crucial tasks for computer assistance during minimally invasive robotic surgery. In the majority of cases, the first step is the automatic segmentation of surgical tools. Prior work has focused on binary segmentation, where the objective is to label every pixel in an image as tool or background. We improve upon previous work in two major ways. First, we leverage recent techniques such as deep residual learning and dilated convolutions to advance binary-segmentation performance. Second, we extend the approach to multi-class segmentation, which lets us segment different parts of the tool, in addition to background. We demonstrate the performance of this method on the MICCAI Endoscopic Vision Challenge Robotic Instruments dataset.

Figures

Tables

Table 1. Table 1 : Shows comparison of our results with previous state-of-the art  [ 7 ] in binary segmentation of robotic tools. Our method provides a 4% improvement in balanced accuracy.

	Sensitivity	Specificity	Balanced Accuracy
FCN-8s [7]	$87.8\%$	$88.7\%$	$88.3\%$
Our work	$85.7\%$	$98.8\%$	92.3%

Table 2. Table 2 : Table reports IoU results for the task of multi-class image segmentation. C1, C2 and C3 corresponds to Manipulator, Shaft, and Background, respectively.

	C1	C2	C3	Overall Mean
Video 1	79.6	68.2	96.5	81.4
Video 2	82.2	70.2	98.6	83.7
Video 3	80.4	66.4	98.0	81.6
Video 4	75.0	44.9	97.1	72.3
Video 5	72.3	56.0	96.3	74.9
Video 6	70.7	50.2	95.7	72.2

Equations

$$y_l=g(x_l,w_l),$$

$$x_{l+1}=f(y_l).$$

$$y_l=h(x_l)+F(x_l,W_l),$$

$$x_{l+1}=f(y_l),$$

$$y[i]=\sum _{k=1}^{K}x[i+rk]w[k]$$

Full text

1 Introduction

Robot-assisted Minimally Invasive Surgery (RMIS) overcomes many of the limitations of traditional laparoscopic Minimally Invasive Surgery (MIS), providing the surgeon with improved control over the anatomy with articulated instruments and dexterous master manipulators. In addition to this, 3D-HD visualization on systems such as da Vinci enhances the surgeon’s depth perception and operating precision [3]. However, complications due to the reduced field-of-view provided by the surgical camera limit the surgeon’s ability to self-localize. Traditional haptic cues on tissue composition are lost through the robotic control system [11].

Overlaying pre- and intra-operative imaging with the surgical console can provide the surgeon with valuable information which can improve decision making during complex procedures [14]. However, integrating this data is a complex task and involves understanding spatial relationships between the surgical camera, operating instruments and patient anatomy. A critical component of this process is segmentation of the instruments in the camera images which can be used to prevent rendered overlays from occluding the instruments while providing crucial input to instrument tracking frameworks [12, 2].

Segmentation of surgical tools from tissue backgrounds is an extremely difficult task due to lighting challenges such as shadows and specular reflections, visual occlusions such as smoke and blood, and due to complex background textures (see Fig. 2). Early methods attempted to simplify the problem by modifying the appearance of the instruments [15]. However, this complicates clinical application of the technique as sterilization can become an issue. Segmentation of the instruments using natural appearance is a more desirable approach as it can be applied directly to pre-existing clinical setups. However, this defines a more challenging problem. To solve it, previous work has relied on machine learning techniques to model the complex discriminative boundary. The instrument-background segmentation can be modeled as a binary segmentation problem to which discriminative models, such as Random Forests [4], maximum likelihood Gaussian Mixture Models [12] and Naive Bayesian classifiers [13], all trained on color features, have been applied. A more recent work, showing state-of-the-art performance [7], applies Fully Convolutional Networks (FCNs), more specifically FCN-8s model [10] for the task of binary segmentation of robotic tools. Although most approaches treat the problem as a binary segmentation problem, for different applications of instrument tracking, it is important to discriminate between different parts of the instrument, particularly the rigid shaft and the metallic clasper [2]. To the best of our knowledge, no previous work has performed multi-class robotic tool segmentation on the MICCAI Endoscopic Vision Challenge Robotic Instruments dataset [1].

In this work, we adopt the state-of-art residual image classification Convolutional Neural Network (CNN) [8] for the task of semantic image segmentation by casting it into Fully Convolutional Network (FCN) [10]. However, the transformed model delivers prediction map of significantly reduced dimension compared to the input image [10]. To account for that and recover full resolution feature map, we reduce in-network downsampling, employ dilated (atrous) convolutions to enable initialization with the parameters of the original classification network, and perform simple bilinear interpolation of the feature maps to obtain the original image size [5] [16]. This approach is a powerful alternative to using deconvolutional layers and “skip architecture” as in FCN-8s model [10]. By employing it, we advance the state-of-the-art in binary segmentation of tools in the aforementioned dataset and extend our approach for multi-class tool segmentation.

2 Method

The goal of this work is to label every pixel of an image I with one of $C$ semantic classes, representing surgical tool part or background. In case of binary segmentation, the goal is to label each pixel into $C=2$ classes, namely surgical tool and background. In this work, we also consider a more challenging multi-class segmentation with $C=3$ classes, namely tool’s shaft, tool’s manipulator and background.

Each image $\textbf{I}_{i}$ is a three-dimensional array of size $h\times w\times d$, where $h$ and $w$ are spatial dimensions, and $d$ is a channel dimension. In our case, $d=3$ because we use RGB images. Each image $\textbf{I}_{i}$ in the training dataset has corresponding annotation $\textbf{A}_{i}$ of a size $h\times w\times C$ where each element represents one-hot encoded semantic label $a\in\{0,1\}^{C}$ (for example, if we have classes 1, 2, and 3, then the one-hot encoding of label 2 is $(0,1,0)^{T}$).

We aim at learning a mapping from I to A in a supervised fashion that generalizes to previously unseen images. In this work, we use CNNs to learn a discriminative classifier which delivers pixel-wise predictions given an input image. Our method is built upon state-of-the-art deep residual image classification CNN (ResNet-101, Section 2.1), which we convert into fully convolutional network (FCN, Section 2.2).

CNNs reduce the spatial resolution of the feature maps by using pooling layers or convolutional layers with strides greater than one. However, for our task of pixel-wise prediction we would like dense feature maps. We set the stride to one in the last two layers responsible for downsampling, and in order to reuse the weights from a pre-trained model, we dilate the subsequent convolutions (Sec. 2.3) with an appropriate rate. This enables us to obtain predictions that are downsampled only by a factor of $8\times$ (in comparison to the original downsampling of $32\times$).

We then apply bilinear interpolation to regain the original spatial resolution. With an output map of the same resolution as an input image, we perform end-to-end training by minimizing the normalized pixel-wise cross-entropy loss [10].

2.1 Deep Residual Learning

Traditional convolutional networks learn filters that process the input $x_{l}$ and produce a filtered response $x_{l+1}$, as shown below

[TABLE]

Here, $g(.,.)$ is a standard convolutional layer with $w_{l}$ being the weights of the layer’s convolutional filters and biases, $f(.)$ is a non-linear mapping function such as the Rectified Linear Unit (ReLU). Many state of the art CNNs employ such convolutional layer followed by a non-linear rectification as a basic building block (AlexNet, VGG16, etc.). However, He et al. [8] recently showed that significant gains in performance can be obtained by employing “residual units” as a building block of a deep CNN, and called such networks Residual Networks (ResNets). In this work, we use a residual network to perform image segmentation. Deep residual networks (ResNets) [8] consist of many stacked “Residual Units”. Fig. 2 shows the architecture of a residual unit. Each unit can be expressed in the following general form,

[TABLE]

where $x_{l}$ and $x_{l+1}$ are input and output of the $l$-th unit, and $F(.,)$ is a residual function to be learnt. In [8], the function $h(.)$ is a simple identity mapping, $h(x_{l})=x_{l}$ and $f(.)$ is a rectified linear unit activation (ReLU) function. Because $h(x_{l})$ is chosen to be an identity mapping, it is easily realized by attaching an identity skip connection (also known as a “shortcut” connection).

Assuming that the desired underlying mapping for $y_{l}$ is $H(x_{l})$, a residual block fits a mapping of $F(x_{l})=H(x_{l})-x_{l}$, which is called a residual function. The original mapping is recast into $F(x_{l})+x_{l}$. It was experimentally shown in [8] that learning residual functions with reference to the layer inputs, instead of learning unreferenced functions allows to train deeper models which gain accuracy from considerably increased depth.

ResNets that are over 100-layers deep have shown to produce state-of-the-art accuracy for several challenging Image Classification and Image Segmentation tasks [8] [5]. This motivates our choice of using ResNet architecture over others. In our work, we adopt ResNet-101 architecture for Image Segmentation and apply it for the task of tool segmentation.

2.2 Fully Convolutional Networks

Deep CNNs (e.g. AlexNet, VGG16, ResNets, etc.) are primarily trained for the task of image classification. Hence, they are originally designed to solve recognition problems on the scale of entire image, by assigning one of many class labels to it. However, to obtain the output granularity required for a task such as image segmentation the network should be modified. This modification consists of converting fully connected layers into convolutions with kernels that are equal to their fixed input regions [10]. Such a network is called a Fully Convolutional Network (FCN). FCN operates on inputs of any size, and produces an output with reduced spatial dimensions [10]. The reduction in the spatial dimension is due to the presence of either pooling (VGG16) or convolutional (ResNets) layers with a stride greater than one pixel.

In order to convert our Image Classification CNN (ResNet-101) into FCN we follow the recent line of work by Long et al. [10] and Chen et al. [5] by removing the final average pooling layer and replacing the fully connected layer with a $1\times 1$ convolutional layer. Doing so casts the network into FCN that takes input of any size and produces an output with predictions over a spatial grid of smaller resolution. This transformation is illustrated in Fig. 3b.

Fully convolutional models deliver prediction maps with significantly reduced dimensions (for both VGG16 and ResNets, the spatial dimensions are reduced by a factor of $32$). In the previous work [10], it was shown that adding a deconvolutional layer to learn the upsampling with factor $32$ provides a way to get the prediction map of original image dimension, but the segmentation boundaries delivered by this approach are usually too coarse. To tackle this problem, two approaches were recently developed which are based on modifying the architecture. (i) By fusing features from layers of different resolution to make the predictions [10]. (ii) By avoiding downsampling of some of the feature maps [5] [16] (removing certain pooling layers in VGG16 and by setting the strides to one in certain convolutional layers responsible for the downsampling in ResNets). However, since the weights in the subsequent layers were trained to work on a downsampled feature map, they need to be adapted to work on the feature maps of a higher spatial resolution. To this end, [5] employs dilated convolutions. In our work, we follow the second approach: we mitigate the decrease in the spatial resolution by using convolutions with strides equal to one in the last two convolutional layers responsible for downsampling in ResNet-101 and by employing dilated convolutions for subsequent convolutional layers (Sec. 2.3).

2.3 Dilated Convolutions

In order to account for the problem stated in the previous section, we use dilated (atrous) convolution. Dilated convolution222We follow the practice of previous work and use simplified definition without mirroring and centering the filter [5]. in one-dimensional case is defined as

[TABLE]

where, $x$ is an input 1D signal, $y$ output signal and $w$ is a filter of size $K$. The rate parameter $r$ corresponds to the dilation factor. The dilated convolution operator can reuse the weights from the filters that were trained on downsampled feature maps by sampling the unreduced feature maps with an appropriate rate.

In our work, since we choose not to downsample in some convolutional layers (by setting their stride to one instead of two), convolutions in all subsequent layers are dilated. This enables initialization with the parameters of the original classification network, while producing higher-resolution outputs. This transformation follows [5] and is illustrated in Fig. 3c.

2.4 Training

Given a sequence of images $\{\textbf{I}_{t}\}^{n_{t}}_{t=0}$, and sequence of ground-truth segmentation annotations $\{\textbf{A}_{t}\}^{n_{t}}_{t=0}$, we optimize normalized pixel-wise cross-entropy loss [10] using Adam optimization algorithm [9] with learning rate set to $10^{-4}$ ($n_{t}$ stands for the number of training examples). We choose the learning rate of $10^{-4}$ after performing a grid search over five different learning rates and found that $10^{-4}$ helps produce the best score on the validation dataset. Other parameters of Adam optimization algorithm were set to the values suggested in [9].

3 Experiments and Results

We test our method on the MICCAI Endoscopic Vision Challenge’s Robotic Instruments dataset. This dataset consists of four $45$-second 2D stereo image sequences with Large Needle Driver (LND) instruments in an ex-vivo setup that is used for training. Each pixel is labeled as either background, shaft or articulated head. The test data consists of four $15$-second sequences with similar background to training sequence. Two $1$-minute 2D image sequences of $2$ instruments in an ex-vivo setup are also in the test dataset. These sequences also contain tool that is not present in the training dataset. The sequences contain occlusions and articulations.

3.1 Results

We report our results in Tab. 1 using standard metrics such as sensitivity and specificity and compare with the previous state-of-the-art [7] for the task of binary segmentation. We can see that our method outperforms the previous work by 4%. We also report results using the Intersection Over Union (IoU) metric for the task of multi-class segmentation in Tab. 2. IoU is a standard metric used for quantifying segmentation results [6]. To the best of our knowledge, we are the first to report segmentation results on the multi-class segmentation task on this dataset. Fig. 4 shows some qualitative results for both the binary segmentation and the multi-class segmentation tasks.

4 Discussion and Conclusion

In this work, we propose a method to perform robotic tool segmentation. This is an important task, as it can be used to prevent rendered overlays from occluding the instruments or to estimate the pose of a tool [2]. We use deep network to model the mapping from the raw images to the segmentation maps. Our use of a state-of-the-art deep network (ResNet-101) with dilated convolutions helps us achieve 4% improvement in binary tool segmentation over the previous stat-of-the-art. In addition, we extend the binary segmentation task to multi-class segmentation task (segmenting out tool parts). We are the first to do this on the MICCAI Endoscopic Vision Challenge’s Robotic Instruments dataset. Our results show the benefit of using deep residual networks for this task and also provide a solid baseline for future work on multi-class segmentation.