Red blood cell image generation for data augmentation using Conditional Generative Adversarial Networks

TL;DR

This paper presents a method using conditional GANs to generate realistic blood cell images and segmentation masks, enhancing small medical datasets for improved segmentation and detection tasks.

Contribution

The paper introduces a novel application of conditional GANs for generating synthetic blood smear images and masks, aiding data augmentation in medical imaging.

Findings

Generated images are photorealistic and diverse.

Data augmentation improves segmentation accuracy.

Method effectively increases dataset size with high-quality samples.

Abstract

In this paper, we describe how to apply image-to-image translation techniques to medical blood smear data to generate new data samples and meaningfully increase small datasets. Specifically, given the segmentation mask of the microscopy image, we are able to generate photorealistic images of blood cells which are further used alongside real data during the network training for segmentation and object detection tasks. This image data generation approach is based on conditional generative adversarial networks which have proven capabilities to high-quality image synthesis. In addition to synthesizing blood images, we synthesize segmentation mask as well which leads to a diverse variety of generated samples. The effectiveness of the technique is thoroughly analyzed and quantified through a number of experiments on a manually collected and annotated dataset of blood smear taken under a microscope.

Tables

Table 1. Table 1: Statistics of red blood cells in the train set.

	Mean	Std
RBCs per image	$669$	$149$
RBC shape	$46\times 46$	$5\times 5$
Colors (in RGB)	$154\times 151\times 162$	$20\times 17\times 21$

Table 2. Table 2: Quantitative evaluation on various tasks. Column names represent a different combination of data used for training of the networks: real data(RD), synthetic data(SD), and augmentations (aug).

Methods \Tasks	Segmentation task (Section 5.2). Metric: Dice score.
	RD	SD	RD+aug	RD+SD	RD+SD+aug
FCN [19]	0.961	0.848	0.962	0.948	0.964
	Detection task (Section 5.3). Metric: AP.
Faster R-CNN [24]	0.781	0.852	0.985	0.986	0.993
	Detection from segmentation (Section 5.4). Metric: AP.
DCAN [6]	0.853	0.749	0.885	0.848	0.895

Equations

$$\{(s_1,l_1),(s_2,l_2),...,(s_n,l_n),\text{background}\},$$

$$l_i\sim \mathcal{P}(i),$$

$$\mathcal{P}(i)=\{\begin{array}{ll}\mathrm{uniform},& i\le n_{init}\\ \frac{1}{n_{init}}{\sum }_{j=1}^{n_{init}}z(l_j),& i=n_{init}+1\\ (1-a_i)\mathcal{P}(i-1)+a_{i}z(l_{i-1}),& i>n_{init}+1\end{array}$$

$$\displaystyle\min_{G}\bigg{(}\Big{(}\max_{D_{1},D_{2}}\sum_{k=1,2}\mathcal{L}_{\text{GAN}}(G,D_{k})\Big{)}+\lambda_{\text{FM}}\sum_{k=1,2}\mathcal{L}_{\text{FM}}(G,D_{k})$$

$$\displaystyle+\lambda_{\text{PR}}~{}\mathcal{L}_{\text{PR}}(G(x,E(x)),y)\bigg{)},$$

$${\mathbb{E}}_{(x,y)}[\log D_k(x,y)]+{\mathbb{E}}_x[\log (1-D_k(x,G(x,E(x)))],$$

$${\mathbb{E}}_{(x,y)}\sum _{i=1}^T\frac{1}{N_i}[||D_k^{(i)}(x,y)-D_k^{(i)}(x,G(x,E(x)))|{|}_1],$$

$$\sum _{i=1}^N\frac{1}{M_i}[||F^{(i)}(y)-F^{(i)}(G(x,E(x)))|{|}_1]$$

$$DSC=\frac{2|p\cdot t|}{|p{|}^2+|t{|}^2},$$

Full text

1 Introduction

Deep learning based methods have had a great success on a number of typical visual perception tasks such as classification, segmentation, and object detection. While there are a number of important reasons that have made this progress possible, one of them is the availability of large-scale datasets. On the contrary to the traditional computer vision tasks, not much effort has been put to the creation of large-scale medical image datasets. There are several reasons these datasets are hard to create.

First of all, there is a limited number of data annotators available. In contrast to traditional image annotation (e.g. class label, segmentation mask, bounding box) where fairly any person is able to perform the annotation, medical data requires a specialized professional—often with an advanced medical degree—to perform a reliable annotation.

Secondly, medical data sharing is not a straightforward process. In order to democratize medical dataset, an agreement from a number of involved parties such as patients, doctors, hospitals, and data users should be reached. Additionally, it requires a detailed guideline on what grounds and for what purposes the data can be utilized.

Lastly, different hospitals and countries around the world might use different medical protocols, devices, or mechanisms. Thus, this data could not be easily transferable, requiring special managing to standardize it for the creation of a large-scale dataset.

All the of aforementioned reasons greatly increase the cost of the annotation while overall making a large-scale medical dataset creation extremely time and resource consuming. Fortunately, with the advent of the Generative Adversarial Networks (GANs) [11] a powerful image synthesis has become possible. Recently, Wang et al. have proposed a model named pix2pixHD [31] that performs high-resolution photorealistic image generation given the instance segmentation mask of the scene. Since the segmentation masks are easy to be manipulated, this method allows interactive object manipulation leading to numerous new samples.

In this paper, we focus on generating new training samples of blood images taken with a microscope. For this purpose, we have collected a dataset and manually annotated every single blood cell in images with an instance level segmentation label. Next, we train GAN to perform translation of a segmentation mask to a photorealistic blood image.

Once the network is trained, we automatically generate diverse instance segmentation masks on run (see Figure 1), and create a defined number of various data samples which are further used as an additional training data for segmentation and object detection tasks. These generated samples serve as a powerful data augmentation technique that could boost the performance of relevant tasks. The graphical summary of the algorithm is presented in Figure 2. To sum up, the contributions of this work are the following:

• •

pipeline to generate red blood cell images from a synthetically created instance segmentation masks

• •

extensive quantified analysis on the effect of generated images on segmentation and detection tasks

This paper is structured in the following way. In Section 2, we cover the related works which similarly have utilized GANs for medical data generation. Then, Section 3 describes the dataset creation. Section 4 specifies a procedure for a synthetic instance segmentation mask generation as well as details on the utilized GAN model. Lastly, a number of experiments and discussion are presented in Section 5 followed by conclusion and future plan (Section 6).

2 Related Work

The use of GANs with medical image data is not new [37, 16, 32]. GANs have demonstrated promising results with a medical data in a number of tasks such as segmentation [17, 34, 20, 29, 35, 38], detection [25, 2, 7, 26, 30], and image synthesis [10, 33, 3, 1, 23].

Since existing medical datasets are often limited in size, several works have examined the use of GANs for data augmentation purposes to increase the number of training samples. For instance, Calimeri et al. [5] have concluded the possibility of employing synthetically generated MRI for inexpensive and fast data augmentation. Furthermore, Han et al. [13] have studied the effect of synthetic data on brain metastases detection task in MRI. In addition, several other works have observed an improved performance on segmentation of various data sources such as MRI [21, 27], CT [4], and X-ray [22].

Similar conclusions of favorable use of GANs as data augmentation technique have been observed for classification of CT images [9] and liver lesion classification [8].

While there are many works targeting MRI, CT, and X-ray image data, blood smear images taken with a microscope have got less attention. Thus, in this paper, we explore the use of GANs for blood image synthesis and study the effect of utilizing GANs as a data augmentation method for segmentation and object detection tasks.

3 Dataset

In order to obtain the dataset with Red Blood Cells (RBCs), we have collected blood from $100$ patients and prepared stained blood smear slides. Then, we have manually selected $100$ images to ensure every image is distinctive to diversify the dataset. This results in a great variation of a number of RBCs per image, their shape, and color values (see Table 1). For example, the largest cells could reach $71\times 87$ resolution, while the smallest could be of several pixels (i.e. partially visible cells on the border of the image). Every image is of $1920\times 1200$ resolution in RGB format. The dataset is randomly split into training and testing sets with a ratio of 60 to 40 respectively. Every image is of $1920\times 1200$ resolution in RGB format.

The annotation technique of the dataset is inspired by the 2015 MICCAI Gland Segmentation Challenge [28]. Specifically, given a microscope image with blood smear, we aim to create an instance segmentation mask which would allow us to extract every single cell in a given image. For this purpose, qualified professionals manually draw precise segmentation mask for each RBC.

This drawing process consists in putting a color on every single pixel belonging to a blood cell. Color values are selected by the user from a predefined list of color values. In order to be able to extract every single blood cell without an overlap with others, any touching cells have different segmentation mask colors assigned to it, hence, no touching cells having the same color.

While this dataset also includes annotation for other blood cells and noise (e.g. dust), in this paper, we primarily focus on RBCs. The representative images and corresponding instance segmentation masks are shown in Figure 3.

4 Methodology

The creation of new samples is composed of two stages: synthetic mask generation and translating generated mask to a photorealistic image of blood cells. The graphical visualization of the whole pipeline during training and testing phases are shown in Figure 2.

4.1 Synthetic mask generation

In order to generate new and meaningful photorealistic samples of the blood cells, we first need to produce synthetic instance segmentation masks in which the blood cells have unique shapes and location distribution. For this purpose, we have designed a synthetic segmentation mask generator that combines sampled cell shapes and their distributions to produce synthetic segmentation masks. More formally, we formulate a synthetic segmentation mask as a set of sampled cell shapes which are located at their corresponding locations on the background:

[TABLE]

where $s_{i}$ denotes the shape of the cell and $l_{i}$ determines the location of the cell. The total number of cells in an image $n$ is drawn from a normal distribution as $n\sim\textrm{Norm}(\mu_{n},\sigma_{n})$, where $\mu_{n}$ and $\sigma_{n}$ are determined from the statistics of training data (see Table 1).

4.1.1 Cell shape sampling

To model the natural variety and similarity of cell shapes in blood, we perform exemplar-based cell shape sampling. During the training stage, segmented boundaries of each cell in the training data are accumulated to build the cell shape database. This database serves as a blood cell shape supplier during the inference stage. In the inference stage (i.e. generating new samples), the cell shape sampler iteratively selects random cell shape $s_{i}$ from the database and puts it on the instance segmentation mask which is composed of numerous cells. Moreover, whenever we pick $s_{i}$ from the database, a set of probabilistic augmentations are applied to diversify the appearance. The augmentation includes rotation, scale, horizontal and vertical flipping.

4.1.2 Cell distribution sampling

The most straightforward way to locate the sampled cell shapes in the synthetic segmentation mask is simply to iteratively sample the coordinates at random and place the cell shape masks at the sampled location if the place is not occupied. However, such an approach results in a nonrealistic cell distribution. Specifically, this leads to a uniform random distribution of cells all over the image with many cells being placed solo without touching other cells. In reality, due to cell adhesion [12], cells tend to stick to each other, consequently, forming clusters.

Therefore, in order to generate the segmentation masks which incorporate the aforementioned aspect, our cell distribution algorithm sequentially samples the appropriate location of each cell from the probability density function defined on the 2D discrete space. The location of i-th cell $l_{i}$ is sampled from the probability map $\mathcal{P}(i)$ as:

[TABLE]

where each pixel value in $\mathcal{P}(i)$ denotes the probability of being selected as a location of a cell’s center at time step $i$. $\mathcal{P}(i)$ is initially uniform during the sampling of the first $n_{init}$ cells, but changes its landscape as $i$ increases in order to simulate inherent cell adhesion. We have modeled this evolving nature of $\mathcal{P}(i)$ as a Markov random process.

[TABLE]

Specifically, this explains that the probability map $\mathcal{P}(i)$ incrementally changes as accumulating the function $z(\cdot)$ to its previous state. The excitation function $z(l_{i})$ is calculated by applying a 2D Gaussian function with $\sigma=\sigma_{x}=\sigma_{y}$ around the sampled cell center $l_{i}$ and reverting the values within the $cell_{size}$ around it. This is done to impose low probability within boundaries of already allocated cells to prevent a cell being placed at the occupied location. The amount of increment is controlled by the normalizing coefficient $a_{i}$, which is in the form of a harmonic progression of $i$. (i.e. $a_{i}=1/i$). At any time stamp, $\mathcal{P}(i)$ always maintains the ‘sum to unity’ property.

In practice, during the cell placement on the synthetic mask, at any time the cell is placed on the canvas, a specific color is assigned to it to satisfy the condition that no touching cells have the same color. If such constraint is impossible to maintain—many touching cells or all predefined colors are used—the coordinate sampling is repeated. Since every touching cell has a different color, produced synthetic mask can be treated as an instance segmentation mask with a possibility to extract every single cell.

This synthetic mask generation process, as well as the comparison of randomly distributed cells against the proposed strategy, is visually described in Figure 4.

4.2 Synthetic blood image generation

For the purpose of synthesizing photorealistic blood cell image given the instance segmentation mask, we have utilized a recent pix2pixHD framework proposed by Wang et al. [31]. For our scenario, the pix2pixHD framework is composed of Generator $G$ which tries to translate segmentation mask to a photorealistic blood cell image. At the same time, two multi-scale generators $D=(D_{1},D_{2})$ are trying to distinguish real images from the generated ones. The full training objective of the network is the following:

[TABLE]

where:

• •

$\mathcal{L}_{\text{GAN}}(G,D_{k})$ is the adversarial loss defined as:

[TABLE]

• •

$\mathcal{L}_{\text{FM}}(G,D_{k})$ is the feature matching loss that aims to stabilize training and produce more visually appealing results at multiple scales. It is defined as:

[TABLE]

• •

$\mathcal{L}_{\text{PR}}(G(x),y)$ is the perceptual reconstruction loss aiming to further improve the performance of high-quality image generation:

[TABLE]

As suggested in [31], we incorporate feature encoder network $E$ and combine its output with original input $x$ to be able to manipulate image synthesis style easily. Since it is a relatively easy task to generate blood cells, we only use the global generator without a local enhancement.

During the inference stage, we feed synthetically generated mask to the generator to obtain a synthetic image of blood cells. The style of the output image is influenced by randomly sampling features from one of $10$ clusters which are created by running K-means clustering on the outputs of the encoder $E$ supplied with the training images.

For a detailed description of the losses, feature encoding, and clustering please refer to the original paper [31]. Implementation details, as well as training procedures, are described in Section 5.1.

5 Experiments and Results

In this section, first, we cover specific details on parameters and training strategy for blood cell images synthesis. Later, the majority of this section describes various experiments on segmentation and object detection tasks with a focus on the effect of the use of synthetic images during training on the performance of the networks.

5.1 Synthetic blood image generation

In order to decide the number on cells to be placed on the synthetic mask, we sample the number from a normal distribution with a mean and standard deviation taken from the dataset statistics (see Table 1 “RBC count per image” row). In the case of the probability map creation, during the synthetic mask generation, the standard deviation value $\sigma$ of 2D Gaussian distribution is related to half width at half maximum (HWHM)111It is half of the distance between points on the curve at which the function reaches half its maximum value. value. Specifically, we want HWHM value to be equal to $cell_{size}$. Hence, we can derive that $\sigma=cell_{size}/\sqrt{2\ln{2}}$, where $cell_{size}=46$ (taken from Table 1). Initially, when a synthetic segmentation mask is empty, $n_{init}=20$ cells are placed at random without considering the probability map.

The training of the pix2pixHD is performed in two stages due to the GPU memory constraint. In the first stage, the images are downsized to $1024\times 640$ and global generator $G$, discriminators $D$, and feature encoder $E$ are trained simultaneously for $120$ epochs. Then, feature encoder $E$ is used to precompute image features of the training data. In the second stage, the training of $G$ and $D$ is performed using full-resolution images (i.e. $1920\times 1200$) for additional $120$ epochs. After training is completed, 10 clusters are created using encoded features for each semantic category. These clusters are used to simulate different blood cells style leading to the diversification of generated images.

In all experiments, we have utilized original pix2pixHD network with several modifications. For instance, the number of filters in the first convolutional layer of $G$, $D$, and $E$ are set to $16$. The primary goal for such channel tuning is to fit the model within the GPU memory capacity. Horizontal and vertical flipping are utilized as data augmentation.

The pix2pixHD network is trained with a train set solely. Figure 6 shows network outputs on the test set images, hence, we can compare synthesized and corresponding real images. Certainly, the network is able to learn shapes, color, and boundaries of the blood cells given their segmentation mask and generally to produce realistic blood images. Noticeably, when synthetic and real images are compared one-to-one, they do not necessarily have the same colors and intensities values. Furthermore, due to the feature encoding which influences the style of individual cells, synthetic images often have excessive noize embedded to generated blood cells (see Figure 6 (last column)). While the style is selected at random during the synthesis stage, we could manually define the cluster where features are sampled from. For example, Figure 7 shows different styles of the generated blood cell image given the segmentation mask from the first column of the Figure 6.

The output results of the whole pipeline including synthetic mask generation and blood image synthesis can be seen in Figure 5. Noticeably, the proposed method is able to generate synthetic segmentation mask which holds cell adhesion rule (i.e. intercellular forces make cells group with each other). Furthermore, utilizing these synthetic segmentation masks as an input to the blood image generator, the results in realistic blood images of a different style (e.g. color and intensity values, noise level).

Unoptimized Python implementation of synthetic mask generation on average takes $60$ seconds (using Intel Xeon) per mask and it is heavily dependent on the number of cells to be placed on the mask. The image generation takes on average $0.5$ seconds per image (using GeForce 1080Ti). Therefore, in the current implementation form, proposed augmentation method better not be used for a real-time data augmentation, but rather for an offline technique to increase the training samples.

5.2 Segmentation

For this task, we have utilized FCN-8s [19] model. Loss function is formulated as $1-DSC$, where $DSC$ stands for a Dice score that is defined as:

[TABLE]

where it is stated as vector operation over binary vectors $p$ and $t$ corresponding to the network prediction and corresponding target (i.e. ground truth) respectively.

The Dice score is used as an evaluation metric as well. The model is trained from scratch without any pretraining or transfer learning until the training loss is converged and no improvement is observed on the test set. The best scoring weights on the test set are selected for reporting results.

Table 2 demonstrates the quantitative results with different data used as training instances. For instance, “RD” column represents results where the network is only trained with real data (RD). Similarly, in the “SD” case, exclusively synthetic data (SD) is used for the training. In both of these cases, the number of training samples is intentionally set equal to $60$ samples. The remaining columns “RD+aug”, “RD+SD”, and “RD+SD+aug” represent results where training set is composed of real data with augmentations, real data mixed with synthetic data, and augmented real and synthetic data respectively. The data augmentation includes horizontal and vertical flips, Gaussian blur, sharpening, embossing, Gaussian noise, color channel inversion, brightness change (addition and multiplication), contrast normalization, and grayscale augmentation applied randomly. For these scenarios, the number of training samples is identical and equals $960$ samples.

FCN is a powerful model capable of reaching a high dice score of $0.961$ by utilizing real data exclusively without any augmentations. Though, when the real data is replaced with synthetic one, the network reaches a substantially lower score of $0.848$. This could imply that the real data has richer semantic information compared to the completely synthetic images that tend to approximate the real data.

Real data augmentation slightly helps to boost performance by about $0.1$ percent points. However, utilizing synthetic data as an augmentation method alongside real data actually harms the performance of the model decreasing the score by roughly $1\%$. Lastly, if the real and synthetic data is trained with augmentations together, the maximum score of $0.964$ is reached.

To sum up, this experiment implies that the use of generated synthetic blood cell images is beneficial to the performance of the network in this particular circumstances. The similar conclusions have been observed in the previous works [21, 27, 4, 22] as well.

5.3 Detection

For this task we have utilized Faster R-CNN [24] based on ResNet-101 [14] feature extractor. The model is pretrained on Visual Genome [18], and we have fine-tuned the model with blood cell data sets mentioned in Table 2. For this task average precision (AP) is used as an evaluation matric. The implementation of the network is provided by [36].

Faster R-CNN is a heavy network which requires lots of training data. Hence, utilizing only limited real data for training results in a relatively low AP score of 0.781 (see Table 2). Surprisingly, the network performs better (about $9\%$ relative improvement) when synthetic data alone is used for training while every other factor is kept equal. While this is counter-intuitive, it could be explained by the fact that, in order to produce synthetic blood images, we produce synthetic segmentation masks which comprise of blood cells shapes derived from a whole training set. This results in new cell shape distribution for each synthetic image. Furthermore, when the cell shape is placed on the segmentation mask, we additionally apply augmentations for each cell shape which leads to even more diverse samples compared to the real training set.

Contrarily to the segmentation task, augmentation provides a drastic improvement (about $26\%$) to the generalization capabilities of the model for the cell detection task. Similarly, a noticeable improvement is observed if synthetic data is used alongside the real data. Lastly, using real and synthetic data with augmentations provides even further improvement (about $1\%$) to the performance compared to the one without synthetic images.

Overall, this experiment shows a marginal benefit of synthetic data on the detection task when used alongside real data. Utilization of synthetic data during the training performs on par with traditional data augmentation techniques and could be used alongside real data and augmentations to maximize the model performance.

5.4 Detection from segmentation

While Faster R-CNN model has shown a great performance on the cell detection task, this model has several significant obstacles. First of all, it struggles with cluttered objects which highly overlap with each other. This is a potentially difficult problem because it involves a large number of small highly-overlapping objects (i.e. small object detection problem). Secondly, such a complex object detection models are often quite computationally expensive, which might be a problem for point-of-care medical devices which often require a fast and efficient computation on the edge. Lastly, this model is relatively slow and hard to stably train with numerous parameter tuning required.

In order to implement a method which is light-weight and fast, while maintaining satisfactory detection score, we utilize an idea proposed by Chen et al. [6]. Specifically, the segmentation network is modified to predict objectness (i.e traditional segmentation mask) as well as contours (i.e. boundary) of the blood cells. These two predictions are further processed to isolate individual cells from each other which simplify the detection problem to a blob detection on the binary image. Therefore, this method is suitable for the detection task and can be evaluated with average precision (AP) score as in the previous experiment.

Similarly to the segmentation task described in Section 5.2, we have utilized FCN network but with the modified head to output objectness and contour predictions. The network is trained from scratch, with a loss which is simply a sum of objectness and contour losses. The best performing (defined by the AP score) model on a test set is selected for results stated in this paper.

The evaluation results are stated in Table 2. Since this method relies on segmentation, the detection performance generally follows the trend of segmentation prediction from Section 5.2. For example, utilizing only synthetic data harms the performance of the network by $0.104$ AP score. Also, the use of data augmentation helps to marginally boost to the network performance. Finally, using real and synthetic data with augmentations helps to reach the highest AP score of $0.895$. When this approach is compared to Faster R-CNN model, only in the case the real data is used for training, this method is able to achieve better performance. However, in every other scenario, Faster R-CNN outperforms this approach by a large margin.

While with this approach we have not succeeded to outperform powerful Faster R-CNN model, we confirm the possibility of achieving blood cell detection from the segmentation mask. The quality of detection is directly related to the quality of predicted segmentation mask. Therefore, we believe that the detection performance could be greatly increased by utilizing more recent neural networks such as [15] for more accurate medical image segmentation.

6 Conclusion

In this paper, we have developed a method to synthesize photorealistic microscopy blood images by utilizing conditional generative adversarial networks. These synthetic images are used alongside real data to meaningfully increase small datasets. The effect of such data augmentation technique is studied through a number of experiments on several tasks. While the use of synthetic images is shown to be marginally beneficial for the segmentation task, the performance on a detection task demonstrates a slightly stronger relative improvement.

To sum up, based on the experiments we have performed, for our specific strategy and algorithm design, the use of GANs as a synthetic data generator and further utilization of generated samples as an augmentation technique is usually beneficial for the model performance. However, the additional overhead which comes from designing GAN model, long and unstable training, heavy computational requirements, and other challenges might not justify the marginal improvement to the overall performance for a specific task.

In the current version, the proposed method is limited to generating microscopy red blood cell images. In future work, we plan to extend this method to synthesize other blood cells such as white blood cells, and platelets. Additionally, we would like to be able to synthesize parasite-infected cells —red blood cells with a visible parasite inside them —which would be beneficial for identification of various diseases such as malaria.