Automatic microscopic cell counting by use of deeply-supervised density regression model

TL;DR

This paper introduces a novel deeply-supervised density regression model using a multi-scale FCNN framework with auxiliary supervision to enhance automatic microscopic cell counting accuracy.

Contribution

It proposes a deeply-supervised, multi-scale FCNN approach with auxiliary supervision for improved cell counting in microscopic images.

Findings

Enhanced counting accuracy demonstrated over existing methods

Multi-scale feature integration improves density map quality

Auxiliary supervision accelerates training convergence

Abstract

Accurately counting cells in microscopic images is important for medical diagnoses and biological studies, but manual cell counting is very tedious, time-consuming, and prone to subjective errors, and automatic counting can be less accurate than desired. To improve the accuracy of automatic cell counting, we propose here a novel method that employs deeply-supervised density regression. A fully convolutional neural network (FCNN) serves as the primary FCNN for density map regression. Innovatively, a set of auxiliary FCNNs are employed to provide additional supervision for learning the intermediate layers of the primary CNN to improve network performance. In addition, the primary CNN is designed as a concatenating framework to integrate multi-scale features through shortcut connections in the network, which improves the granularity of the features extracted from the intermediate CNN layers and further supports the final density map estimation.

Tables

Table 1. Table 1 : Performance of the proposed cell counting method

Performance	PriCNN+AuxCNN	PriCNN-only	FCRN [7]
MAE	32.89	42.17	44.90
STD	26.35	30.97	35.30

Equations

$$Y_{i,j}=\sum _{k=1}^{N_c}{G}_{\sigma }(i-s_{k_x},j-s_{k_y}),\forall i\in M,j\in N,$$

$${\hat{Y}}_{i,j}=F(\varphi (X_{i,j});\Theta ),$$

$$N_c\approx \widehat{N_c}=\sum _{i=1}^M\sum _{j=1}^N{\hat{Y}}_{i,j}.$$

$$Y\approx \hat{Y}=F(X;\Theta ),$$

$$\{\begin{array}{ll}{L}_k({\Theta }_1,...,{\Theta }_k,{\theta }_k)& =\frac{1}{B}{\sum }_{i=1}^B{\Vert A_k({\varphi }_k(X_i;{\Theta }_1,...,{\Theta }_k);{\theta }_k)-Y_i^k\Vert }^2,k=1,2,3,\\ L(\Theta )& =\frac{1}{B}{\sum }_{i=1}^B{\Vert F(X_i,\Theta )-Y_i\Vert }^2,\end{array}$$

$$L_{overall}(\Theta ,{\theta }_1,{\theta }_2,{\theta }_3)=L(\Theta )+\sum _{k=1}^3{\lambda }_k{L}_k({\Theta }_1,...,{\Theta }_k,{\theta }_k),$$

$$\widehat{N_c}=\sum _{i=1}^{M^{\prime }}\sum _{j=1}^{N^{\prime }}[F(X^t;{\Theta }^{\ast }){]}_{i,j},$$

Full text

1 INTRODUCTION

Accurately counting the number of cells in microscopic images greatly aids medical diagnoses and biological studies [1]. However, manual cell counting, which is slow, expensive, and prone to subjective errors, is not practically feasible for high-throughput processes. An automatic and efficient solution with improved counting accuracy is highly desirable, but automatic cell counting is challenging due to the low image contrast, strong tissue background, and significant inter-cell occlusions in 2D microscopic images [2, 3, 4, 5].

To address these challenges [6, 7], density-based counting methods have received increasing attention, due to their superior performance to those traditional detection-based methods [8, 9, 5, 10]. Generally speaking, density-based methods employ machine-learning tools to learn a density regression model (DRM) that estimates the cell density distribution from the characteristics/features of a given image. The number of cells can be subsequently estimated by integrating the estimated density map. For example, Lempitsky et al. [6] proposed a supervised learning framework to learn a linear DRM and employ it for visual object counting tasks. Differently, Xie et al. [7] utilized a fully convolutional regression network (FCRN) to learn a DRM for regressing cell spatial densities over the image. The FCRN, as a specific fully convolutional neural network (FCNN), integrates informative feature extraction and powerful function learning to estimate the density map of a given image. It demonstrates promising performance for cell counting tasks, especially for counting overlapped cells.

Generally, the layers in the FCRN are hierarchically constructed, and the output of each layer relies on the outputs of its previous layers. There are two potential shortcomings of this traditional network design. First, the intermediate layers are optimized based on the gradient information back-propagated only from the final layer of the network, not directly from the adjacent layer. Second, this design allows only adjacent layers to be connected, while limiting the integration of multi-scale features (or information) and overall network performance. These two shortcomings might lead to sub-optimized intermediate layers, and can eventually affect the overall cell counting accuracy. Thus, the need to improve the accuracy of automated cell counting methods remains.

Recently, in order to improve the effectiveness of learning intermediate layers in a designed deep neural networks, deeply-supervised learning (or deep supervision) has been proposed and has shown promising performance for addressing various computer vision tasks, such as classification [11] and segmentation [12, 13]. In addition, concatenating CNN frameworks have also attracted great attention. These networks can concatenate multi-scale features by shortcut connections of non-adjacent layers within the network, and so achieve better results than the traditional networks in such computer vision tasks as segmentation [14] and detection [15].

Motivated by these works, this study proposes a novel density regression-based automatic cell counting method. A FCNN is used as a primary FCNN (PriCNN) to learn the density regression model (DRM) that performs an end-to-end mapping from a cell image to the corresponding density map. A set of auxiliary CNN (AuxCNNs) are built to assess the features at the intermediate layers in the PriCNN and to directly supervise the training of these layers. In addition, by use of concatenation layers, the multi-scale features from non-adjacent layers are integrated to improve the granularity of the features extracted from the intermediate layers for further supporting final density map estimation. Experimental results, evaluated on a set of immunofluorescent images of human embryonic stem cells (hESC), have demonstrated the superior performance of the proposed deep supervision-based DRM method compared to other state-of-the-art methods.

2 Methodology

2.1 Background: Density-Based Automatic Cell Counting

The goal of density regression-based cell counting methods is to learn a density regression function $F$, which can be employed to estimate the density map of a given image [6, 7]. Given an image $X\in\mathbb{R}^{M\times N}$ which includes $N_{c}$ cells, the density map $Y\in\mathbb{R}^{M\times N}$ of $X$ can be considered as the superposition of a set of $N_{c}$ normalized 2D discrete Gaussian kernels that are placed at the centroids of the $N_{c}$ cells. Therefore, the number of cells can be counted by integrating the density map over the image.

Let $S=\{({s_{k_{x}}},{s_{k_{y}}})\in\mathbb{N}^{2}:k=1,2,...,N_{c}\}$ represent the cell centroid positions in $X$. Each pixel $Y_{i,j}$ on the density map $Y$ can be expressed as:

[TABLE]

where $G_{\sigma}(n_{x},n_{y})=C\cdot e^{-\frac{n_{x}^{2}+n_{y}^{2}}{2\sigma^{2}}}\in\mathbb{R}^{(2K_{G}+1)\times(2K_{G}+1)},n_{x},n_{y}=-K_{G},-K_{G}+1,...,K_{G}$, is a normalized 2D Gaussian kernel that satisfies $\sum_{n_{x}=-K_{G}}^{K_{G}}\sum_{n_{y}=-K_{G}}^{K_{G}}G_{\sigma}(n_{x},n_{y})=1$. Here, $\sigma^{2}$ is the isotropic covariance, $(2K_{G}+1)$ is the kernel size, and $C$ is a normalization constant.

The density regression-based cell counting process generally includes three steps: (1) map an image to a feature map, (2) estimate a cell density map from the feature map, and (3) integrate the density map for automatic cell counting. In the first step, each pixel $X_{i,j}$ in $X$ can be assumed to be associated with a real-valued feature vector $\phi(X_{i,j})\in R^{Z}$. The feature map $P\in\mathbb{R}^{M\times N\times Z}$ of $X$ can be generated using specific feature extraction methods, such as the dense scale invariant feature transform (SIFT) descriptor [16], ordinary filter banks [17], or codebook learning [18]. In the second step, the estimated density $\hat{Y}_{i,j}$ of each pixel $X_{i,j}$ in $X$ can be obtained by applying a pre-trained density regression function $F$ on the given $\phi(X_{i,j})$:

[TABLE]

where $\Theta$ is a parameter vector that determines the function $F$. Finally, in the third step, the number of cells in $X$, $N_{c}$, can be counted by integrating the estimated densities $\hat{Y}$ over the image region:

[TABLE]

A key task in density regression-based cell counting methods is learning the function $F$ by use of training datasets. The learning of $F$ and the related cell counting method proposed in this study are described below.

2.2 The Proposed Automatic Cell Counting Framework

We propose a novel automatic cell counting method that employs deeply-supervised density regression model in this study. The framework, shown in Figure 1, includes two phases: 1) DRM training and 2) cell counting by use of the trained DRM. The network architecture of the proposed DRM is described in Section 2.2.1. The two phases in the framework are described in Sections 2.2.2 and 2.2.3, respectively.

2.2.1 The DRM Network Architecture

The DRM is built as a primary FCNN (PriCNN) with the purpose of estimating the density map $\hat{Y}$ of an image $X$, such that:

[TABLE]

where $F(X;\Theta)$ is a density regression function, and $\Theta$ is a parameter vector that determines $F$.

Motivated by the network architecture of FCRN [7], the designed PriCNN (DRM) includes $8$ chained blocks. As shown in the upper row of Figure 1, each of the first three blocks includes a convolutional (CONV) layer, a ReLU layer, and a max-pooling (Pool) layer; the fourth block in the PriCNN includes a CONV layer and a ReLU layer; each of the fifth to seventh blocks includes a up-sampling (UP) layer, a CONV layer, and a ReLU layer; and the last block includes a chain of a CONV layer and a ReLU layer.

In addition, concatenation layers are employed in the PriCNN to integrate multi-scale features and thus improve the granularity of the features, which assists in the final density map estimation. This design is motivated by a network architecture described by Ronneberger et. al [14]. As shown in Figure 1, the outputs from each of the first three blocks are multi-resolution, low-dimension, and highly-representative feature maps. Three shortcut connections are established to connect the first and seventh blocks, the second and sixth blocks, and the third and fifth blocks, respectively. With these shortcut connections, multi-resolution features can be concatenated between non-adjacent layers. The integration of multi-scale features can further improve the performance of the network compared to the traditional FCRN, which allows only adjacent layers to be connected.

2.2.2 DRM Training Process

1. AuxCNN-supported DRM Training

Training the designed DRM (or PriCNN) with such a hierarchical structure is a challenging task. As described in Section 1, all the layers in the original FCRN [7] are learned based on the feedback only from the final layer. Therefore, the intermediate layers might be sub-optimized, which can significantly affect the accuracy of the final estimated density map.

Innovatively, three auxiliary FCNNs (AuxCNNs) are employed to provide additional supervision for learning the intermediate layers of the PriCNN. As shown in Figure 2, each AuxCNN contains two CONV-ReLU blocks for estimating a low-resolution density map from the feature map generated at an intermediate layer of the PriCNN. By jointly minimizing the errors between the estimated density maps and the corresponding ground truth density maps at different resolution levels, the optimization of the intermediate layers in the PriCNN can be improved, which eventually improves the overall performance of the PriCNN.

2. Jointly Training the PriCNN and AuxCNN

The parameters of the PriCNN can be learned by jointly training the PriCNN and the AuxCNNs with a set of given training data $D=\{(X_{i},Y_{i})\}_{i=1,2,..,B}$, where $X_{i}$ and $Y_{i}$ represent the $i$-th image and its associated ground-truth density map, respectively. The training is completed by minimizing the differences between the estimated density maps at different resolution levels and the ground truth density maps.

As shown in Figure 2, the PriCNN can be denoted mathematically as $F(X;\Theta)$, where $\Theta$ is the parameter vector of $F$ and $X$ is an input image. All the trainable parameters in the first four blocks, the $5$th, the $6$th, and the last two blocks can be denoted as $\Theta_{1}$, $\Theta_{2}$, $\Theta_{3}$, and $\Theta_{4}$, respectively. Therefore, $\Theta=(\Theta_{1},\Theta_{2},\Theta_{3},\Theta_{4})$ and $F(X;\Theta)=F(X;\Theta_{1},\Theta_{2},\Theta_{3},\Theta_{4})$. Also, the output feature maps of the $4$-th, $5$-th, and $6$-th blocks can be denoted as $\phi_{1}(X;\Theta_{1})$, $\phi_{2}(X;\Theta_{1},\Theta_{2})$, and $\phi_{3}(X;\Theta_{1},\Theta_{2},\Theta_{3})$, respectively. Similarly, the three AuxCNNs can be denoted as $A_{k}(\phi_{k};\theta_{k})(k=1,2,3)$, where $\theta_{k}$ is the parameter vector of the $k$-th AuxCNN, and $A_{k}$ is the density map estimated by use of the $k$-th AuxCNN. Therefore, the cooperative training of the PriCNN and AuxCNNs is performed by jointly minimizing four loss functions, defined below:

[TABLE]

where $Y^{k}_{i}$ is the ground truth low-resolution density map (GTLR) generated from $Y_{i}$. $Y^{k}_{i}\in\mathbb{R}^{M_{k}\times N_{k}}$ is generated from the original ground-truth density map $Y_{i}\in\mathbb{R}^{M\times N}$ by summing every adjunct $a\times b$ in $Y$, with $a_{k}=\frac{M}{M_{k}}$ and $b_{k}=\frac{N}{N_{k}}$. $L(\Theta)$ is the average mean square error (MSE) between the estimated density maps and their ground truths. $L_{k}(\Theta_{1},...,\Theta_{k},\theta_{k})$ is the average MSE between the low-resolution density maps estimated by $k$-th AuxCNN and their corresponding GTLR density maps.

To improve the computational efficiency of the optimization of the PriCNN and AuxCNNs, we construct a combined loss function, defined as below:

[TABLE]

where $\lambda_{k}\in[0,1]$ is a parameter that controls the relative strength of the supervision under the $k$-th AuxCNN for learning the intermediate layers in the PriCNN. Eqn.(6) is numerically minimized via stochastic gradient descent (SGD) methods [19].

2.2.3 Density estimation and cell counting

During the cell counting phase of the framework (Figure 1), the number of cells in a to-be-tested image $X^{t}\in\mathbb{R}^{M^{\prime}\times N^{\prime}}$ can be estimated by use of the trained DRM represented by $F(X;\Theta^{*})$:

[TABLE]

where $[F(X^{t};\Theta^{*})]_{i,j}$ is the estimated density at pixel $(i,j)$. In this step, the dimensions of the to-be-tested image can be different because arbitrary input image sizes are allowed by the trained PriCNN.

3 Experimental Results

3.1 Dataset

A set of $49$ immunofluorescent images of human embryonic stem cells (hESC) was employed in this study to test the performance of the proposed method. Each image was $512\times 512$ pixels, and the $49$ hESC images were manually annotated by identifying the centroid of each cell within each image. Statistically, the cell number among these images is about $518\pm 316$.

For each annotated image in the training dataset, the corresponding ground truth density maps were generated by placing a normalized 2D discrete Gaussian kernel with isotropic covariance, $\sigma^{2}$, at each annotated cell centroid in the image (details shown in Section 2.2). The values of $\sigma$ and $2K_{G}+1$ were set to $3$ pixels and $21$ pixels, respectively. A pair consisting of an image and a density map was considered as a training sample for training the PriCNN and AuxCNNs. In this study, 5-fold cross validation was employed to evaluate the cell counting performance.

3.2 Method Implementation

We compared the performance of the proposed method (denoted as PriCNN+AuxCNN) with a state-of-the-art method, FCRN [7]. The FCRN used the same network architecture as the PriCNN, but without concatenation layers. In addition, a PriCNN without AuxCNNs (or a FCRN with concatenation layers) was also compared to illustrate the performance improvement by use of AuxCNNs.

In the PriCNN, the convolution kernel size in the first $7$ blocks was set to $3\times 3$, while that in the last block was set to $1\times 1$. The numbers of kernels in the first to $8$th CONV layers are set to 32, 64, 128, 512, 128, 64, 32, and 1, respectively. The pooling size in each pool layer was set to $2\times 2$, and the Up layers performed bi-linear interpolation.

In the first block of the AuxCNN, the kernel size was set to $3\times 3$ and the number of kernels was $32$, while the comparable values in the second block were $1\times 1$ and $1$, respectively. In addition, the ground truth low-resolution density map (GTLR) $Y^{k}_{i}\in\mathbb{R}^{M_{k}\times N_{k}}$ was generated from the original ground-truth density map $Y_{i}\in\mathbb{R}^{M\times N}$ by summing local regions with size of $(a_{1},b_{1})=(8,8)$, $(a_{2},b_{2})=(4,4)$, and $(a_{3},b_{3})=(2,2)$, respectively.

All the three methods, PriCNN+AuxCNN, PriCNN-only, and FCRN were trained under the same hyper-parameter configurations, including a learning rate of $0.0001$ and a batch size of 100. In addition, all the parameters were orthogonally initialized [20].

3.3 Results

In this study, mean absolute error (MAE) and standard deviation of absolute errors (STD) were employed to evaluate the cell counting performance. MAE measures the mean of the absolute errors (MAE) between the estimated cell counts and their ground truths for all images in the validation set. The STD measures the standard deviation of the absolute errors. Table 1 shows that the proposed method yields superior cell counting performance to the other two methods in terms of MAE and STD. In addition, Figure 3 presents the estimated density maps of one hESC image example estimated by the three methods. The numbers of cells counted from density maps are indicated below each density map. From the figure, we can see that the proposed method (PriCNN+AuxCNN) can estimate a density map that is more similar to the ground truth, compared to the other two methods. Also, our estimated cell count is closer to the ground truth.

4 Discussion

Convolutional neural networks (CNN) have succeeded in computer vision tasks, including image classification [21], segmentation [14, 22], and object detection [23]. The success is because that CNNs can integrate informative feature extraction and powerful nonlinear function learning. Furthermore, fully convolutional neural networks (FCNN), such as FCN [24] and U-Net [14], allow flexible input image sizes, and have been employed to perform an efficient end-to-end mapping from an image (one domain) to a probability map (another domain). Both the PriCNN (the proposed DRM in the study) and FCRN [7] are some specific FCNNs, which explain the descent cell counting accuracy they have achieved in this study.

In this study, only a set of experimental immunofluorescent hESC images was employed for DRM training and validation. However, the generalization of the proposed method should not be limited to only the experimental immunofluorescent hESC images. In future, we will evaluate the proposed method on image sets of other modalities. In addition, other competing general object counting methods will also be compared with our proposed method.

5 Conclusion

In this study, for the first time, a deeply-supervised density regression framework is proposed for automatic cell counting. The results obtained on experimental hESC images demonstrate the superior cell counting performance of the proposed method, compared with the state of the art.

Acknowledgements.

This work was supported in part by award NIH R01EB020604, R01EB023045, R01NS102213, and R21CA223799.