Deep neural networks for classifying complex features in diffraction images

TL;DR

This paper demonstrates that deep neural networks significantly improve the classification and feature recognition of complex diffraction images from nano-sized objects, streamlining data analysis in high-throughput diffraction experiments.

Contribution

It presents a systematic setup, modifications, and benchmarking of deep neural networks for diffraction image classification, showing superior performance over previous methods.

Findings

Deep neural networks outperform traditional algorithms in classifying diffraction patterns.

The approach significantly reduces the effort in post-processing large diffraction datasets.

Neural networks generalize well across different experimental conditions.

Abstract

Intense short-wavelength pulses from free-electron lasers and high-harmonic-generation sources enable diffractive imaging of individual nano-sized objects with a single x-ray laser shot. The enormous data sets with up to several million diffraction patterns represent a severe problem for data analysis, due to the high dimensionality of imaging data. Feature recognition and selection is a crucial step to reduce the dimensionality. Usually, custom-made algorithms are developed at a considerable effort to approximate the particular features connected to an individual specimen, but facing different experimental conditions, these approaches do not generalize well. On the other hand, deep neural networks are the principal instrument for today's revolution in automated image recognition, a development that has not been adapted to its full potential for data analysis in science. We recently published in Langbehn et al. (Phys. Rev. Lett. 121, 255301 (2018)) the first application of a deep neural network as a feature extractor for wide-angle diffraction images of helium nanodroplets. Here we present the setup, our modifications and the training process of the deep neural network for diffraction image classification and its systematic benchmarking. We find that deep neural networks significantly outperform previous attempts for sorting and classifying complex diffraction patterns and are a significant improvement for the much-needed assistance during post-processing of large amounts of experimental coherent diffraction imaging data.

Tables

Table 1. Table 1: Statistics of the helium nanodroplets dataset. Non-exclusive labels assigned by a researcher. One image can be in multiple classes. Total dataset size is 7264 7264 7264 . Note that Spherical/Oblate as a class also contains Round patterns, only Prolate shapes are excluded from this class (see also caption of Figure 2 ).

Class	Nr. of labels	% of the whole dataset
Spherical/Oblate	$6589$	$90.7$
Round	$5792$	$79.7$
Elliptical	$796$	$11.0$
Newton rings	$460$	$6.3$
Prolate	$453$	$6.2$
Bent	$390$	$5.4$
Asymmetric	$367$	$5.1$
Streak	$242$	$3.3$
Double Rings	$218$	$3.0$
Layered	$47$	$0.7$
Empty	$222$	$3.1$

Table 2. Table 2: The iterative optimization routine for a deep feed-forward neural network.

$1$.	Forward pass: Propagate $bs$ images through the network.
$2$.	Evaluate the predictions: At the output layer calculate
	the loss between the ground truth and the output of the
	deep neural network (Equation 6).
$3$.	Construct the backpropagation rule: Include all
	gradients w.r.t. the weights of all layer according
	to Equation 9.
$4$.	Backward correction: Update all weights in the
	network using gradient descent, see Equation 7.

Table 3. Table 3: Overall evaluation metrics for the ResNet18 architecture (vanilla configuration) and both datasets. The table gives the max values during training for Accuracy , Precision , and Recall . The training time after which the neural network achieved the highest Accuracy score on the evaluation dataset is labeled Train Time M ​ a ​ x subscript Train Time 𝑀 𝑎 𝑥 \text{Train Time}_{Max} and the time for training the full 200 200 200 epochs is labeled Train Time F ​ u ​ l ​ l subscript Train Time 𝐹 𝑢 𝑙 𝑙 \text{Train Time}_{Full} . See also appendix A for further details.

Architecture	ResNet18
Dataset	CXIDB	Helium
Accuracy	$0.967$	$0.955$
Precision	$0.932$	$0.918$
Recall	$0.933$	$0.866$
${\text{Train Time}}_{Max}$ [h]	$0.278$	$0.231$
${\text{Train Time}}_{Full}$ [h]	$0.668$	$0.694$

Table 4. Table 4: Evaluation metrics for the ResNet18 network with and without the logarithmic activation function. We benchmark three values for α 𝛼 \alpha . Results are shown for both datasets and are the maximum value recorded during training. Bold numbers indicate the best scores across their respective category.

Architecture	ResNet18
$\alpha$	$0.2$	$0.5$	$1.0$	unmodified
Dataset	Helium
Accuracy	$0.965$	$0.960$	$0.959$	$0.955$
Precision	$0.922$	$0.920$	$0.922$	$0.918$
Recall	$0.870$	$0.870$	$0.868$	$0.867$

Table 5. Table 5: Evaluation metrics of the ResNet18 α = 0.2 subscript ResNet18 𝛼 0.2 \text{ResNet18}_{\alpha=0.2} network with the logarithmic activation function and an α 𝛼 \alpha value of 0.2 0.2 0.2 . Results are shown for the helium datasets and reflect the maximum achieved value reached throughout the training process, assessed on the evaluation dataset. Bold numbers indicate the best scores across their respective category.

Architecture	${\text{ResNet18}}_{\alpha =0.2}$
Training set size	$6174$	$4631$	$3088$	$1544$
Dataset	Helium
Accuracy	$0.965$	$0.915$	$0.829$	$0.797$
Precision	$0.922$	$0.821$	$0.740$	$0.673$
Recall	$0.870$	$0.771$	$0.679$	$0.593$

Table 6. Table 6: Evaluation results when training a ResNet18 α = 0.2 subscript ResNet18 𝛼 0.2 \text{ResNet18}_{\alpha=0.2} on the original diffraction images and on CCF maps calculated from them. The results reflect the maximum value achieved throughout the training process, assessed on the evaluation dataset. Bold numbers indicate the best scores across their respective category.

Architecture	${\text{ResNet18}}_{\alpha =0.2}$
Noise added	None		Mean		Mean + Std.		Max.
Input data	CCF Maps	Diff. Imgs.	CCF Maps	Diff. Imgs.	CCF Maps	Diff. Imgs.	CCF Maps	Diff. Imgs.
Dataset	Helium
Accuracy	$0.950$	$0.965$	$0.948$	$0.954$	$0.946$	$0.944$	$0.944$	$0.926$
Precision	$0.901$	$0.922$	$0.897$	$0.910$	$0.893$	$0.887$	$0.905$	$0.865$
Recall	$0.838$	$0.870$	$0.833$	$0.853$	$0.823$	$0.815$	$0.814$	$0.808$

Table 7. Table 7: The deep neural network architecture of the VGG variant D and E. conv(a, b, c) is a convolutional layer with filter size a × a a a \text{a}\times\text{a} , number of filters b 𝑏 b and stride c 𝑐 c . max pooling(d, e) is a max pooling layer with filter size d × d d d \text{d}\times\text{d} and stride e 𝑒 e . Note that we changed the fully connected layer of the original architecture to a convolutional layer.

Variant	D	E
Depth	16	19
Input	$2\times \text{conv}(3,64,1)$
Pooling	$\text{max pooling}(2,2)$
Block 1	$2\times \text{conv}(3,128,1)$
Pooling	$\text{max pooling}(2,2)$
Block 2	$3\times \text{conv}(3,256,1)$	$4\times \text{conv}(3,256,1)$
Pooling	$\text{max pooling}(2,2)$
Block 3	$3\times \text{conv}(3,512,1)$	$4\times \text{conv}(3,512,1)$
Pooling	$\text{max pooling}(2,2)$
Block 4	$3\times \text{conv}(3,512,1)$	$4\times \text{conv}(3,512,1)$
Pooling	$\text{max pooling}(2,2)$
Out block	$2\times \text{conv}(7,4096,1),\text{conv}(1,N,1)$

Table 8. Table 8: Used ResNet variants, see also 18 18 18 , 50 50 50 and 101 101 101 layer layout in He  et al. ( 2015 ) . Note that we added the pre-activated layer layout from Zagoruyko and Komodakis ( 2016 ) . conv(a, b, c) is a convolutional layer with filter size a × a a a \text{a}\times\text{a} , number of filters b 𝑏 b and stride c 𝑐 c . max pooling(d, e) is a max pooling layer with filter size d × d d d \text{d}\times\text{d} and stride e. avg pooling is a global average pooling layer, and fc(f) is a fully connected layer with output size f. Layers in bold emphasis have a stride of 2 2 2 during their first iteration, therefore reducing the dimension by a factor of 2 2 2 .

Variant	Classic	Bottleneck	Bottleneck
Depth	18	50	101
Input	conv(7, 64, 2)
Pooling	max pooling(3 , 2)
Block 1	$2\times \left[\begin{array}{c}\text{conv}(3,64,1)\\ \text{conv}(3,64,1)\end{array}\right]$	$3\times \left[\begin{array}{c}\text{conv}(1,64,1)\\ \text{conv}(3,64,3)\\ \text{conv}(1,256,1)\end{array}\right]$	$3\times \left[\begin{array}{c}\text{conv}(1,64,1)\\ \text{conv}(3,64,3)\\ \text{conv}(1,256,1)\end{array}\right]$
Block 2	$2\times \left[\begin{array}{c}\text{conv}(3,128,1)\\ \text{conv}(3,128,1)\end{array}\right]$	$4\times \left[\begin{array}{c}\text{conv}(1,128,1)\\ \text{conv}(3,128,3)\\ \text{conv}(1,512,1)\end{array}\right]$	$8\times \left[\begin{array}{c}\text{conv}(1,128,1)\\ \text{conv}(3,128,3)\\ \text{conv}(1,512,1)\end{array}\right]$
Block 3	$2\times \left[\begin{array}{c}\text{conv}(3,256,1)\\ \text{conv}(3,256,1)\end{array}\right]$	$6\times \left[\begin{array}{c}\text{conv}(1,256,1)\\ \text{conv}(3,256,3)\\ \text{conv}(1,1024,1)\end{array}\right]$	$36\times \left[\begin{array}{c}\text{conv}(1,256,1)\\ \text{conv}(3,256,3)\\ \text{conv}(1,1024,1)\end{array}\right]$
Block 4	$2\times \left[\begin{array}{c}\text{conv}(3,512,1)\\ \text{conv}(3,512,1)\end{array}\right]$	$3\times \left[\begin{array}{c}\text{conv}(1,512,1)\\ \text{conv}(3,512,3)\\ \text{conv}(1,2048,1)\end{array}\right]$	$3\times \left[\begin{array}{c}\text{conv}(1,512,1)\\ \text{conv}(3,512,3)\\ \text{conv}(1,2048,1)\end{array}\right]$
Output block	avg pooling, fc(N)

Table 9. Table 9: Overall evaluation metrics for all architectures and both datasets. The training time after which the neural network scored the highest accuracy score on the evaluation dataset is labeled Train Time M ​ a ​ x subscript Train Time 𝑀 𝑎 𝑥 \text{Train Time}_{Max} and Train Time F ​ u ​ l ​ l subscript Train Time 𝐹 𝑢 𝑙 𝑙 \text{Train Time}_{Full} is the time for training the full 200 200 200 epochs. The table gives the max values during training for accuracy, precision, and recall. Bold scores are the best results in their respective category.

Architecture	ResNet			VGG
Depth	$18$	$50$	$101$	$16$	$19$
Dataset	Helium
Accuracy	$0.955$	$0.958$	$0.964$	$0.958$	$0.959$
Precision	$0.918$	$0.917$	$0.925$	$0.923$	$0.920$
Recall	$0.866$	$0.864$	$0.878$	$0.867$	$0.867$
${\text{Train Time}}_{Max}$ [h]	$0.231$	$0.605$	$0.940$	$3.271$	$6.615$
${\text{Train Time}}_{Full}$ [h]	$0.694$	$1.814$	$2.820$	$6.541$	$6.726$
Dataset	CXIDB
Accuracy	$0.967$	$0.973$	$0.978$	$0.970$	$0.970$
Precision	$0.932$	$0.937$	$0.949$	$0.944$	$0.943$
Recall	$0.933$	$0.937$	$0.941$	$0.904$	$0.904$
${\text{Train Time}}_{Max}$ [h]	$0.278$	$1.205$	$1.093$	$4.374$	$4.480$
${\text{Train Time}}_{Full}$ [h]	$0.668$	$1.807$	$2.623$	$6.562$	$6.720$

Table 10. Table 10: Per-class accuracy, precision and recall values for the best performing ResNet configuration with 101 101 101 layers. Samples are the number of images whose ground truth label is positive in the evaluation dataset. Results are shown for both datasets and reflect the maximum achieved value reached throughout the training process, assessed on the evaluation dataset.

Class	Accuracy	Precision	Recall	Samples
Oblate	0.9681	0.9770	0.9965	988
Spherical	0.9166	0.9247	0.9849	869
Elliptical	0.9231	0.8054	0.4836	119
Newton rings	0.9352	0.6325	0.2282	69
Prolate	0.9690	0.9274	0.6777	68
Bent	0.9657	0.8161	0.6487	59
Asymmetric	0.9458	0.6044	0.2207	55
Streak	0.9898	0.9372	0.9876	36
Double Rings	0.9768	0.7708	0.6788	33
Layered	0.9896	0.9062	0.6170	7
Empty	0.9904	0.9537	0.9763	32

Equations

$$a_j=\sum _{k=1}^m{x}_k{w}_{kj},$$

$$a_{i,j,k}=\sum _{l,m,n}{x}_{l,j+m-1,k+n-1}{k}_{i,l,m,n},$$

$$h\left(x\right)=\frac{1}{1+\exp \left(-x\right)}$$

$$h(x)=\{\begin{array}{ll}x& \text{if x≥0}\\ \gamma x& \text{if x<0},\end{array}$$

$$H(p,q)=H(p)+D_{\mathrm{KL}}(p\Vert q)$$

$$H(x_{out},x)=\sum _i^M{x}_i^{out}-x_i^{out}{x}_i+\log \left(1+\exp \left(-x_i^{out}\right)\right),$$

$${\mathbf{w}}_{\tau +1}={\mathbf{w}}_{\tau }-\eta {\nabla }_{{\mathbf{w}}_{\tau }}H\left(x_{out},x\right).$$

$${\nabla }_{{\mathbf{w}}_{\tau }}H\left(x_{out},x\right)=\frac{\partial H(x_{out},x)}{\partial w_{j\tau }^N}=\frac{\partial H(x_{out},x)}{\partial h^N\left(a_j^N\right)}\frac{\partial h^N\left(a_j^N\right)}{\partial w_{j\tau }^N},$$

$$\frac{\partial H(x_{out},x)}{\partial h^N\left(a_j^N\right)}=\frac{\partial H(x_{out},x)}{\partial h^{N-1}\left(a_j^{N-1}\right)}\frac{\partial h^{N-1}\left(a_j^{N-1}\right)}{\partial h^N\left(a_j^N\right)}.$$

$$H(x_{out},x{)}_{reg}=H(x_{out},x)+\alpha ||\mathbf{w}|{|}_1+\beta ||\mathbf{w}|{|}_2$$

$$\text{Accuracy}=\frac{\text{True Positives}+\text{True Negatives}}{\text{Condition Positives}+\text{Condition Negatives}},$$

$$\text{Precision}=\frac{\text{True Positives}}{\text{True Positives}+\text{False Positives}},$$

$$\text{Recall}=\frac{\text{True Positives}}{\text{True Positives}+\text{False Negatives}}.$$

$$h(x)=\{\begin{array}{ll}\alpha \left(\log \left(x+c_0\right)+c_1\right)& \text{if x≥0}\\ -\alpha \left(\log \left(c_0-x\right)+c_1\right)& \text{if x<0},\end{array}$$

$$\frac{\partial h(x)}{\partial x}=\{\begin{array}{ll}\frac{\alpha }{x+c_0}& \text{if x≥0}\\ \frac{\alpha }{c_0-x}& \text{if x<0}.\end{array}$$

$$C_{i,j}\left(q_i,q_j,\Delta \right)={\int }_{-\infty }^{\infty }{I}_i^{\ast }\left(q_i,\varphi \right)I_j\left(q_j,\varphi +\Delta \right)d\varphi ,$$

$$C_{i,j}^n\left(q_i,q_j\right)=I_i^{n\ast }\left(q_i\right)I_j^n\left(q_j\right),$$

$$I_i^n\left(q_i\right)=\frac{1}{2\pi }{\int }_0^{2\pi }I\left(q_i,\varphi \right)\exp \left(-in\varphi \right)d\varphi$$

$$H(p,q)=H(p)+D_{\mathrm{KL}}(p\Vert q)$$

$$H(p,q)=-\sum _i{p}_i\log q_i,$$

$$p\left(x\right)=\{\begin{array}{ll}y\left(x\right)& \text{if k=1}\\ 1-y\left(x\right)& \text{if k=0}\end{array}$$

$$q\left(x\right)=\{\begin{array}{ll}\hat{y}\left(x\right)& \text{if k=1}\\ 1-\hat{y}\left(x\right)& \text{if k=0}\end{array}$$

$$\hat{y}{\left(x\right)}_{\text{sigmoid}}=\frac{1}{1+\exp \left(-x\right)},$$

$$\begin{array}{rl}H(p,q)& =-\sum _i^2{p}_i\log q_i,\\ & =-y\left(x\right)\log \left(\hat{y}\left(x\right)\right)-\left(1-y\left(x\right)\right)\log \left(1-\hat{y}\left(x\right)\right)\\ & =-y\left(x\right)\log \left(\frac{1}{1+\exp \left(-x\right)}\right)-\left(1-y\left(x\right)\right)\\ & \log \left(1-\frac{1}{1+\exp \left(-x\right)}\right)\\ & \text{makebox}[7.7778pt]\text{hfil}⋮\text{hfil}\\ & =x-xy\left(x\right)+\log \left(1+\exp \left(-x\right)\right)\end{array}$$

$${\mu }_{\text{mb}}=\frac{1}{m}\sum _{i=1}^m{x}_i$$

$${\sigma }_{\text{mb}}^2=\frac{1}{m}\sum _{i=1}^m{\left(x_i-{\mu }_{\text{mb}}\right)}^2$$

$$\widehat{x_i}=\frac{x_i-{\mu }_{\text{mb}}}{\sqrt{{\sigma }_{\text{mb}}^2+ϵ}}$$

$$y_i=\gamma \widehat{x_i}+\beta$$

$$Y^c=\sum _k{w}_k^c\sum _i\sum _j{A}_{ij}^k$$

$$w_k^c=\sum _i\sum _j{a}_{ij}^{kc}\text{LeakyReLu}\left(\frac{\partial Y^c}{\partial A_{ij}^k}\right).$$

$$a_{ij}^{kc}=\frac{\frac{{\partial }^2{Y}^c}{{\left(\partial A_{ij}^k\right)}^2}}{2\frac{{\partial }^2{Y}^c}{{\left(\partial A_{ij}^k\right)}^2}+{\sum }_a{\sum }_b{A}_{ab}^k\frac{{\partial }^3{Y}^c}{{\left(\partial A_{ij}^k\right)}^3}}$$

Full text

residual convolutional deep neural networks, coherent diffraction imaging, image classification

††preprint: v1.0

1 Introduction

Coherent diffraction imaging (CDI) experiments of single particles in free flight have been proven to be a significant asset in the pursuit of understanding the structural composition of nano-scaled matter Seibert et al. (2011); Loh et al. (2012); Bostedt et al. (2010); Gomez et al. (2014); Chapman and Nugent (2010); Rupp et al. (2017). While traditional microscopy methods are able to image fixated, substrate-grown or deposited individual particles Li et al. (2008); Farges et al. (1986); Clemmer and Jarrold (1997); Kostko et al. (2007); Sakdinawat and Attwood (2010), only CDI can combine high-resolution images with single particles in free flight in one experiment Bostedt et al. (2009); Gorkhover et al. (2012); Bostedt et al. (2016a). CDI became possible due to the recent advent of short wavelength free-electron lasers (FELs) producing coherent high-intensity x-ray pulses with femtosecond duration with a single x-ray laser shot Emma et al. (2010). However, CDI also comes with its own set of new challenges.

One of the growing problems of CDI experiments is the sheer amount of recorded data that has to be analyzed. The LINAC Coherent Light Source (LCLS), for instance, has a repetition rate of $120\text{\,}\mathrm{Hz}$ with typical hit-rates ranging from $1\text{\,}\mathrm{\char 37\relax}30\text{\,}\mathrm{\char 37\relax}$ Emma et al. (2010); Bostedt et al. (2016b); Calvey et al. (2016), greatly depending on the performed experiment. The newly opened European XFEL will have an even higher maximum repetition rate of $27\,000\text{\,}\mathrm{Hz}$ Schneidmiller (2011), which may add up to several million diffraction patterns in a single $12$-hour shift. The idea of using neural networks for classification of large number of scattering patterns was born out of the significant difficulties of analyzing large data sets of clusters Rupp et al. (2014), in particularly metal clusters Barke et al. (2015). Moreover, the ability to analyze such data sets is sought after by the community in general Lundholm et al. (2018). For example, for the successful determination of 3D-structures from a CDI data set using the expansion-maximization-compression algorithm Flamant et al. (2016); Ekeberg et al. (2015); Lundholm et al. (2018), it is necessary to sample the 3D Fourier space up to the Nyquist rate for the desired resolution and this for all sub-species contained in the target under study. The achievable resolution, as well as the chance for successful convergence of the algorithm, correlates directly with the number of diffraction patterns with a high signal-to-noise ratio Flamant et al. (2016). Thus, huge data sets are taken and as a consequence of the sheer amount of data, it is getting increasingly complicated to distill the high-quality data subsets that are suitable for subsequent analysis steps.

The enormous success of neural networks in the regime of image processing and classification provides a unique way of facing the imminent data-analysis bottleneck and reduces the impending problem to a mere domain adaptation from datasets used throughout the industry to ones that are used in CDI research. This work aims to be a stepping-stone towards this adaptation by providing an introduction to the theory of deep neural networks and analyzing how to best transfer and optimize these algorithms to the domain of scattering images. As a new baseline, we train a widely used deep neural network architecture, a residual convolutional deep neural network He et al. (2016), in a supervised manner with a training set of manually labeled data. We then adapt the neural network to the domain of diffraction images and improve on the baseline performance by addressing the following issues:

Modification of the architecture to account for the specificities of diffraction images and thus optimize the prediction capabilities. 2. 2.

Determination of the appropriate size of the training dataset in order to keep the manual work of a researcher to a moderate level. 3. 3.

Mitigation of experimental artifacts, in particular noisy diffraction images.

Experience has shown that a researcher is able to relate diffraction patterns produced by similarly shaped particles of different sizes and orientations in context with each other. However, a programmatic description for a classification and sorting of these mostly similar patterns is almost impossible to achieve.

Figure 1 illustrates the case of two diffraction patterns captured from almost identical particles but under different orientations. Both patterns clearly show an elongated and bent streak, but the bending is differently pronounced and directed. If we wanted to handcraft an algorithm that detects this feature, we would need to describe it via some appropriate metric that must take into account the various grades of inflection, direction, brightness, and completeness of this feature within every image. Furthermore, we would need to redo it for every characteristic feature in a diffraction image of which we want to find similar ones.

In addition to that, poor signal-to-noise ratios, stray-light, a beam stop or central hole of multichannel plates or pnCCDs Meidinger et al. (2006) and overall poor image quality can even further increase the difficulty to make an automatized classification of all images coherent Kurta et al. (2016); Bobkov et al. (2015); Atla et al. (2011).

Therefore, we need a robust classification routine that is insusceptible to the described artifacts, just as a researcher is, to tackle the upcoming data volume. Deep neural networks provide a way out of this situation, and we show in this paper that they outperform the current state-of-the-art classification and sorting routines.

Current state-of-the-art automatic classification routines for diffraction experiments employ so-called kernel methods Bobkov et al. (2015); Yoon et al. (2011). Bobkov et al. (2015) trained a support-vector-machine on a public small-angle x-ray scattering dataset with an Accuracy of $87\text{\,}\mathrm{\char 37\relax}$, but only on selected images (we will use this approach as a reference in section 4). Yoon et al. (2011) were able to achieve an Accuracy of up to $90\text{\,}\mathrm{\char 37\relax}$ using unsupervised spectral clustering on a non-public small-angle x-ray scattering dataset.

Deep neural networks, on the other hand, have already been applied to a broad range of physics-related problems ranging from predicting topological ground states Deng et al. (2017), distinguish different topological phases of topological band insulators Zhang et al. (2018a), enhancing the signal-to-noise at hadron colliders Field et al. (1996), differentiate between so-called known-physics background and new-physics signals at the Large Hadron Collider Bhimji et al. (2017) and to help solve the Schrödinger equation Mills et al. (2017); Manzhos et al. (2009). Their ability to classify images has also been utilized in cryo-electron microscopy Zhu et al. (2017), medical imaging Gao et al. (2017) and even for hit-finding in serial x-ray crystallography Ke et al. (2018). However, to our knowledge, this paper is the first application of deep neural networks for classifying complex features within diffraction patterns. We show that deep neural networks outperform the current state-of-the-art classification and sorting routines, while being insusceptible to typical artifact features of diffraction measurements. Furthermore a deeper analysis of the trained network shows that it can understand complex concepts of what constitutes a characteristic feature in a diffraction pattern.

The paper is organized as follows: In section 2, the data set is presented and a few experimental details are discussed. Section 3 provides the fundamental theory to understand the basics of neural networks; it has two subsections. Subsection 3.1 covers the theory, and algorithmic underpinnings of deep neural networks and how to train these models and subsection 3.2 presents three common metrics to evaluate the quality of the neural network’s predictions.

Section 4 establishes our starting point, while the full benchmark report on the baseline neural network can be found in appendix A. We introduce the chosen network architecture and provide baseline results on the data presented in section 2 but also on a reference dataset for which classification results are already published Kassemeyer et al. (2012).

In section 5, we discuss solutions for the above stated issues of applying neural networks to diffraction data. In subsection 5.1 we discuss the choice of the activation function for the neural network and present a novel logarithmic activation function that enhances the prediction performance with diffraction image data. Subsection 5.2 benchmarks the dependence of neural networks on training data size, asking essentially how much manually labeled data is needed for the neural network to give acceptable results and subsection 5.3 presents an approach to harden the neural network against very noisy data using a custom two-point cross-correlation map.

In section 6 we then provide more profound insights into the output of the neural network by showing and discussing calculated heatmaps that visualize the gradient flow within the neural network. These images directly correlate with what the neural network sees; they are created using an advanced visualization algorithm called GradCam++ Chattopadhyay et al. (2017).

Finally, we give a summary of the principal results and unique propositions of this paper and conclude with an outlook on further modifications as well as future directions.

2 The data

Helium nanodroplets Langbehn et al. (2018) were imaged using extreme ultraviolet (XUV) photon energies between $19\text{\,}\mathrm{eV}35\text{\,}\mathrm{eV}$ using the experimental setup of the LDM beamline Lyamayev et al. (2013); Svetina et al. (2015) at the Free Electron Laser FERMI Allaria et al. (2012). Scattering images were recorded with a multi-channel-plate (MCP) detector combined with a phosphor screen which was placed $65\text{\,}\mathrm{mm}$ downstream from the interaction region; this defines the maximum scattering angle of $30\text{\,}\mathrm{\SIUnitSymbolDegree}$. Single shot diffraction images in the XUV regime are in some respect a special case, as they cover large scattering angles and can contain 3D structural information Barke et al. (2015), manifesting as complex and pronounced characteristic features, such as the bent streaks in Figure 1. Out of $2\text{\times}{10}^{5}$ laser shots, about $38\,000$ images were obtained. The images were corrected for straylight background and the flat detector (see also Langbehn et al. (2018))

For the neural network training dataset, we selected $7264$ diffraction images randomly out of all recorded patterns. The size of the subset was chosen to be the maximum a researcher could classify manually given one week time. From this subset we manually identified $11$ distinct but non-exclusive classes (see Figure 2 for examples as well as a description and Table 1 for statistics about every class). We chose each of the diffraction patterns shown in Figure 2 for being a strong candidate for its class, but it is important to note that almost all diffraction patterns belong to multiple classes since this is a multi-class labeling scenario. These patterns are therefore not always clearly distinguishable from each other and can exhibit multiple characteristics from different classes. For example, the Newton rings in Figure 2d) are superimposed on a concentric ring pattern that falls into the category Spherical/Oblate, but Newton rings can also occur in other classes, e.g. streak patterns. Furthermore, labeling all images is itself prone to systematic errors because the researcher has to learn-to-label Frénay and Kabán (2014). This means that the labeling process itself is to some extent ill-posed, as the researcher does not know the characteristics of a feature a priori which results in a changing perception of features and classes along the labeling process and thus a systematically decreased consistency for every class.

We uploaded all available data alongside our assigned labels to the public CXI database (CXIDB, Maia (2012)) under the public domain CC0 waiver 111http://cxidb.org/id-94.html.

3 Basic Theory

3.1 What is a deep neural network

We concentrate in this paper solely on deep feed-forward neural networks. They are a classification model consisting of a directed acyclic graph that defines a set of hierarchically structured non-linear functions.

A fundamental example can be constructed by arranging n non-linear functions ($z_{1},~{}z_{2},~{}\dots~{}z_{n}$) in a chain-like manner: $z_{\text{output}}=z_{n}\left(z_{n-1}\left(\dots\left(z_{2}\left(z_{1}\left(\bm{x}\right)\right)\right)\dots\right)\right)$, where $\bm{x}$ is the input, which is in our case a diffraction image. The first function, $z_{1}\left(\bm{x}\right)$, is called the input layer. We then pass the output of $z_{1}$ to $z_{2}$ and so on; this goes on until the last layer ($z_{n}$) which is called the output layer. The nomenclature is that all layers except the output layer ($z_{n}$) and the input layer ($z_{1}$) are called hidden layers.

For illustrative purposes, Figure 3 shows a convolutional neural network. There, we schematically show the layer functions $z_{1},~{}\dots~{}z_{n}$ where every layer consists of two stages; A linear layer-specific operation on its inputs followed by a so-called activation function, which is always non-linear. We address the choice of layer-specific operations in section 3.1.1 and then introduce the activation functions in section 3.1.2. In general, the layer-specific operation is always the name-giving component for the layer, so for example if we compute a 2D convolution as the layer-specific operation on the input and then apply an activation function, we call the set of these two stages a convolutional layer. Figure 3 shows a neural network whose first layers are convolutional layers followed by a fully connected layer that produces the predictions.

3.1.1 Affine transformations

All common choices for layer-specific operations are affine transformations. They all introduce trainable weights; free parameters that are adjustable during the training process and are sometimes called neurons due to the intuition that in a fully connected layer they share some similarity to the dendrites, soma, and axon of a biological neuron Arbib (1987). These trainable weights are the name-giving components in a neural network.

Now, the goal of training a neural network is to optimize all these weights for all layers, so, that the predictions for all images in the training data match their accompanying original labels. The original labels are called ground truth and define the upper limit of how good a network can fit a domain. No neural network is better than its training data. In this section, we briefly illustrate the affine transformations of the fully connected layer and the convolutional layer and then explain in the next section the role of the activation function.

Fully connected layer

The name-giving operation for the fully connected layer is a matrix multiplication performed on a flattened input, for example, a $m\times n$ sized input image would be flattened into a $m\cdot n$ sized vector. Mathematically this is a matrix multiplication between a matrix and a vector:

[TABLE]

where $\bm{x}$ is the flattened input and $\bm{w}$ is the weight matrix of a fully connected layer. Here, all input vector elements (e.g., the pixels of an image, now arranged in one large row $x_{k}$) contribute to all output matrix elements and are therefore connected. Furthermore, by convention is $x_{0}$ defined as $1$ and $w_{0j}=b_{j}$, where $b_{j}$ is a free and trainable bias parameter.

Convolutional layer

In a convolutional layer, the trainable weights are parameters of a kernel that slides over the inputs, this is visualized in Figure 3. The general idea of a convolutional layer is to preserve the spatial correlations in the input image when going to a lower dimensional representation (the next layer). This is achieved by using a kernel with a spatial extent larger than $1$ px. The kernel size is then also the extent to which one kernel can correlate different areas of an input and is called its local receptive field. Each kernel produces one output which is called a feature map or filter. Multiple feature maps from multiple kernels are grouped within one convolutional layer. For example, the first convolutional layer in Figure 3 produces $9$ feature maps out of the input diffraction image and hence has $9$ kernels that get optimized during training. Since we usually only have in the input layer a 2-dimensional diffraction image as input and a high number of feature maps for every subsequent convolutional layer as their inputs, we define the output of a convolutional layer with a $4$-dimensional kernel $k$ that produces $i$ feature maps of size $j\times k$:

[TABLE]

here the input $x$ has $l$ dimensions of size $j\times k$ and we slide a kernel of size $m\times n$ across all these $l$ dimensions. In the given example for the input layer, $l$ is simply $1$ and the summation is just across one input image, as shown in Figure 3.

3.1.2 Activation functions

Regardless of the affine transformation that is used, all layer-specific operations produce trainable weights which are passed through an activation function. This function is always non-linear. We only address $2$ activation functions here as they are the most common used by the community and the only ones we use; The sigmoid and the LeakyRelu function. The first one is a logistic regression function used mostly at the outputs of neural networks, and the second one is a piecewise linear activation function used between layers for numerical reasons Nair, V.;Hinton (2010); Maas et al. (2016). The sigmoid function is given as:

[TABLE]

and the LeakyRelu function is given as:

[TABLE]

where in both functions $x\in\bm{a}$ are the trainable weights of the affine transformation (the convolutional or the fully connected layer operation, i.e., the output of Equation 1 or 2) and $\gamma$ is the slope for the negative part in the LeakyRelu function and is called leakage.

In Figure 3 the last activation function of the neural network, denoted by Logistic function, is a sigmoid function, because its output can be interpreted as a probability in a Bernoulli distribution, yielding a probability for how likely it is that a given event (an image in our case) is part of a class (in our case, the pre-defined classes from Table 1). Sigmoid functions always give an output between $1$ and [math]. In our case, we have $11$ distinct classes which are mutually non-exclusive, which means every image has a probability of being part of every class. Using a sigmoid function at the end of the neural network yields therefore 11 distinct Bernoulli distributions. The generalization from the single-case Bernoulli distribution to its multi-case n-class distribution equivalent is called categorical distribution.

Interpreting the output of the neural network, as well as the original labels, as a categorical distribution is key to train the neural network because only then we can use statistical measures to evaluate the quality of the neural network’s prediction, which allows us to optimize it iteratively.

However, due to the non-linearity of all activation functions, optimizing a neural network is a non-convex problem where no global extrema can be found with certainty. The general procedure is that of a forward pass and then a backward correction. Meaning, we feed the neural network several images, take the network’s prediction and compare this prediction to the ground truth; This is the forward pass. Then we calculate a loss function which is a metric for how bad or good the predictions were, see the next section, and correct the weights of the network in a way that it would be better equipped to predict the labels for the images it just saw. This correction step is starting at the end of the network using an algorithm called backpropagation; hence the name backward correction, see section 3.1.4.

3.1.3 The forward pass: Assess the network’s predictions

Optimizing a neural network always starts by feeding it multiple images and evaluate what the neural network made of it. For assessing the quality of the network’s prediction a so-called loss function is used. It is the defining metric that we seek to minimize during the training of the neural network. In every training step, we compare the output of the neural network to the real labels provided by the researcher and calculate the so-called loss. Lower loss values correspond to a higher prediction quality of the neural net.

Therefore, the goal during the training process is to adjust all weights and biases within the network so, that the loss is minimal for all input training images. There are various possible loss functions which often serve a specific purpose. For classification tasks, such as the present case, primarily the cross-entropy is used Schmidhuber (2014); LeCun et al. (2015); Szegedy et al. (2016); Hinton et al. (2015). Cross-entropy is a concept from information theory giving an estimate about the statistical distance between a true distribution $p$ and an unnatural distribution $q$. In our case, $p$ is the categorical distribution over the ground truth labels, and $q$ is the output of the neural network.

Cross-entropy is calculated as the sum of the Shannon entropy Shannon (1948) for the true distribution $p$ and the Kullback-Leibler divergence Kullback and Leibler (1951) between $p$ and $q$. The former is a measure of the total amount of information of $p$, and the latter is a typical distance measure between two probability distributions.

If the Kullback-Leibler divergence is zero, then the cross-entropy is just the Shannon entropy of $p$, and we have $p=q$. Then, the predictions of the neural network are not distinguishable from the labels of all training images.

Cross-entropy can be formally written as:

[TABLE]

where $H(p)$ is the Shannon entropy of $p$, and $D_{\mathrm{KL}}(p\|q)$ is the Kullback–Leibler divergence of $p$ and $q$ Goodfellow et al. (2016).

When using a sigmoid function as activation function on the output layer, the final loss function can be defined as:

[TABLE]

where $M$ is the number of all images in the training data, $x^{out}_{i}$ is the prediction for one image from the deep neural network and $x_{i}$ is the original label of the image, assigned by the researcher. Please see appendix B for a complete derivation.

Using Equation 6 as it is, would require us to pass all images through the network for one training step, as the sum runs over all images. This is computational intractable. Therefore, we use a variant of Equation 6 where the sum runs only over a stochastically chosen subset of size $bs$, called a batch. The size of that batch is called batch size and is an important hyperparameter that needs to be chosen prior to training, see section 3.1.5. One iteration step now involves only $bs$ images from the dataset, and we define an epoch as the number of iteration steps it takes the network during the training to see all images one time.

To summarize, minimizing the cross-entropy is the goal during the training process in a neural network. The network learns to link the user-defined labels to the provided images. All that’s left to understand the basic training process of a neural network is a way to adjust the weights in all layers.

3.1.4 The backward correction: Gradient descent and backpropagation

Optimizing the weights within the neural network so that they give minimal loss for all training images is done using two distinct algorithms; gradient descent and backpropagation. In principal, gradient descent works by evaluating the gradient at some point and then moving a certain step-size in the opposite direction; This is done iteratively until the gradient is smaller than some pre-defined threshold, which is the numerical equivalent of calculating the extrema of a function analytically.

The basic gradient descent step is given by:

[TABLE]

where $\eta$ is the afore mentioned step-size, called learning rate, $\nabla_{\bm{w}_{\tau}}$ is the gradient w.r.t. the weights at step $\tau$ and $H\left(x_{out},x\right)$ is the loss function from Equation 6. With Equation 7 we already could update the weights within the output layer of the neural network ($z_{n}(\cdot)$), since for the output layer we can calculate the numerical gradients. But we can’t do this for the layers that come before the output layer since we’re lacking a way to include these. In order to propagate the gradient descent correction throughout the network an algorithm called backpropagation is used Rumelhart et al. (1986):

First, we define the gradient of $H(x_{out},x)$, w.r.t. the weights at the output of the deep neural network, using the chain rule:

[TABLE]

where $N$ denotes the layer depth of the output layer, $h^{N}(\cdot)$ is the used activation function in that layer and $a^{N}_{j}$ are the outputs of the layer-specific operation, as in Equation 1 and 2. Starting from there we include the layer, preceding the output layer ($z_{n-1}(z_{n}(\cdot))$), by making use of the chain rule again:

[TABLE]

This can be iteratively repeated until the input layer ($z_{1}(\cdot)$) is included in the calculation. By making use of the chain rule until we reach the input layer we can include all trainable weights of all layers into the correction term of the gradient descent algorithm. With this, we conclude the full optimization routine in Table 2.

3.1.5 Training setup

Of significant importance is the way how the network is constructed; How deep should the network be and of what should it consist? For nomenclature, the combination of all used layers, the depth of the network and the used activation functions is called an architecture.

We benchmarked the performance of various architectural choices when used with diffraction images as input and provide the results in appendix A and not in the main paper, due to its rather technical character. In short, all architectures are established through extensive empirical research. So far, not only the leading A.I. research institutes, like the Massachusetts Institute of Technology (MIT) or the University of Toronto, but also large companies like Google, Facebook and Microsoft have invested significant amounts of resources to establish well working out-of-the-box solutions Szegedy et al. (2016); Schmidhuber (2014); LeCun et al. (2015).

Building on this and after extensively benchmarking the most common architectures on our own, we settled on an architecture called pre-activated wide residual convolutional neural network in its $18$-layer configuration, called ResNet18 He et al. (2015, 2016); Zagoruyko and Komodakis (2016). In essence, it is a convolutional neural network much like the example in Figure 3 but it employs so-called residual skip connections which increase Accuracy while decrease training time, see appendix A for further details as well as comparisons with other architectures.

After settling on an architecture, training a neural network requires fine-tuning of multiple free parameters. Four of them are critical: The learning rate $\eta$, the batch size bs and so-called regularization parameters of which we have two (which will be introduced at the end of this section).

We set the initial learning rate for the gradient descent algorithm to $\eta=0.1$, see also Equation 7. Throughout the training we multiply $\eta$ with $0.1$ every $50$ epochs, this increases the chance for the gradient descent algorithm to get numerically closer to a minimum in the loss function He et al. (2015). Furthermore, we use a batch size of $48$ for all training procedures, see also the explanations for Equation 6.

We split the manually classified part of the helium dataset into a training and an evaluation subset, where we shuffle the order of all images and then select $85\text{\,}\mathrm{\char 37\relax}$ for the training set while the rest serves as an evaluation set.

We rescale all diffraction images to $224\text{\times}224\text{\,}\mathrm{px}$ which is necessary to fit the deep neural net on two Nvidia $1080$Ti GPUs, each having $11$ GB memory. The image dimensions are chosen to be a compromise between file size and resolution. All features we are training the neural network on are still clearly visible and distinguishable after the rescaling.

Furthermore, we face the problem of having a comparatively small training set, consisting of only $\approx 6000$ classified images, which could result in a phenomenon called over-fitting. Meaning the network memorizes the training set without learning to make any meaningful prediction from it. Therefore we employ two additional techniques called regularization and data augmentation:

Regularization means adding a so-called penalty term to the loss function. There are two regularizations we use, L1 and L2 Zou et al. (2005). These penalty terms are dependent on the weights themselves and not on the labels, making the loss function explicitly dependent on the weights of the neural network. This dependency encourages the neural network to reduce the values of all weights according to the two penalty terms and ultimately find a sparser solution which in return helps to prevent over-fitting. Formally we add these two terms to the loss in Equation 6:

[TABLE]

where $H(x_{out},x)$ is the cross-entropy loss function, $||\bm{w}||_{1}$ and $||\bm{w}||_{2}$ are the L1- and the L2-norm applied on the sum of all trainable weight parameters and $\alpha$ and $\beta$ are so-called regularization coefficients. In our experiments we set $\alpha$ and $\beta$ to $1\text{\times}{10}^{-5}$ during training. Using L1 and L2 regularization in combination is commonly referred to as elastic net regularization Zou et al. (2005). 2. 2.

Data augmentation means creating artificial input images by randomly applying image transformations on the original image like flipping the vertical or the horizontal axes and adjusting contrast or brightness values randomly. This greatly increases the robustness to over-fitting and is used as a standard procedure when facing small training datasets Hinton et al. (2012); Perez and Wang (2017).

We were able to train deep neural networks with a depth of up to $101$ layers without over-fitting using regularization and data augmentation, see appendix A. In all experiments reported here we choose a depth of $18$ layers for the neural network, due to numerical, memory and time reasons. We trained all deep neural networks variants for $200$ epochs.

3.2 Evaluating a deep neural network

We use three metrics to assess the quality of the predictions from the neural network, Accuracy, Precision, and Recall. We calculated these metrics every $2500$ training iteration steps ($\approx 52$ epochs) using the evaluation dataset. Accuracy is formally defined as:

[TABLE]

where condition positives/negatives is the real number of positives/negatives in the data and true positives/negatives is the correct overlap of the prediction from the model and the condition positives/negatives. An Accuracy of $1$ corresponds to a model that was able to predict all classes of all images correct. Therefore, Accuracy is a good measure for evaluating the prediction capabilities of a model when true positives and true negatives are of importance. Predicting negative labels correct is in the case of the helium dataset of particular interest because we want to estimate if the neural network was able to understand the complex inter-class relationships imposed by the researcher. The network should realize that if, for example, one prediction is Spherical/Oblate, it cannot simultaneously be Prolate. Therefore, the network has to produce a true negative for either one of these predictions. However, using only Accuracy as a metric has several downsides. The most important one is the decreased expressiveness of Accuracy when working in a multi-class scenario. In order to understand this, we first introduce Precision and Recall, and then provide an example:

[TABLE]

Precision, also-called positive predictive value, is a measure for how reasonable the estimates of the model were when it labeled a class positive, and Recall is a measure for how complete the model’s positive estimates were.

For example, if the model would predict all training images in the helium dataset to be Spherical/Oblate and nothing else (out of $7264$ images, $6589$ are indeed Spherical/Oblate) then Accuracy would be $0.767$, which translates to $77\text{\,}\mathrm{\char 37\relax}$ of all labels correctly assigned. However, if the model estimated all images to be part of no class (setting every label to negative), then Accuracy would be $0.801$, because out of $79\,904$ possible labels ($11$ independent classes for $7264$ images), $64\,339$ are negative. Therefore, we would have a useless model that still was able to predict $80\text{\,}\mathrm{\char 37\relax}$ of all labels correct.

Using Precision in these both examples would give $0.907$ for the Spherical/Oblate example and $0.000$ for the all-negative example. Precision is, therefore, a metric that quantifies how well the positive predictions were assigned. Since $91\text{\,}\mathrm{\char 37\relax}$ of all images are indeed Spherical/Oblate, setting all labels positive in the Spherical/Oblate class can make sense, and Precision also provides insight when the model makes no positive prediction at all which would be a useless model for our purpose. However, Precision alone is not sufficient as a metric. At this point we don’t know if our model predicted almost every possible positive label correct or if only a small fraction of all positive labels were assigned correctly, we, therefore, need an additional measure for the generalization capabilities of our model. For that reason, Precision is always used in combination with Recall. The Recall for our first example is $0.423$ and for the second one $0.000$. Recall relies on False Negatives instead of the False Positives, used by Precision , which provides a measure about the completeness of all positive predictions compared to all positive labels within our data. Recall states that our model only captured $42\text{\,}\mathrm{\char 37\relax}$ of all possible positive labels in the Spherical/Oblate example, showing that generalization of the model would not be sufficient for a real-world application.

Therefore, a balanced interpretation of these three metrics is necessary to estimate the quality of the models tested here.

4 Baseline performance of neural networks with CDI data

In this section we briefly report on what we call baseline results. We used the previously described ResNet He et al. (2016) neural network architecture in its basic configuration with a depth of 18 layers, termed vanilla configuration or ResNet18 (see section 3.1.5) and trained it with the helium diffraction data set as described in section 2 as well as with a reference data set from the literature Kassemeyer et al. (2012). This reference data set was made freely available on the CXIDB by Kassemeyer et al. (2012) 222https://www.cxidb.org/id-10.html. It contains diffraction patterns of a number of prototypical diffraction imaging targets, namely the Paramecium bursarium Chlorella virus (PBCV-1), bacteriophage T4, magnetosomes and nanorice. For further experimental details see Kassemeyer et al. (2012).

We selected this dataset because of a previous publication dealing with this dataset Bobkov et al. (2015), that describes, to our knowledge, the current state-of-the-art method for classification and sorting of diffraction images Bobkov et al. (2015). Bobkov et al. (2015) trained a support-vector-machine on the CXIDB dataset and inferred the particle type directly from the diffraction images. Overall, they achieved an Accuracy of up to $0.87$, but only on selected high quality images with a high confidence score of the support-vector-machine above $0.75$.

Table 3 shows the overall evaluation metrics as well as the training wall time. $\text{Train Time}_{Max}$ is the time when the neural network achieved the highest Accuracy score on the evaluation dataset, and $\text{Train Time}_{Full}$ is the time for training 200 epochs. In practice, we achieved optimal convergence after training for 70 to 100 epochs.

We achieved an Accuracy of $0.967$ on not only a high quality subset of the CXIDB data, like in Bobkov et al. (2015), but on all available data (see table 3), using a vanilla ResNet18 architecture, proving that using a neural network significantly outperforms the current state-of-the-art approach in Bobkov et al. (2015).

In the case of the helium dataset we face a much more complicated multi-class learning problem (one image can belong to multiple classes compared to one image belongs to exactly one class as it is in the CXIDB data). However, we reach a comparable Accuracy score of $0.955$. Even more promising, Precision and Recall are very high for the helium and the CXIDB dataset, proving that the neural network not only predicted the true positives with high confidence and reliability (high Precision ), it did so for almost all true positive labels in the evaluation dataset (high Recall).

In the next section we show how to further improve on the baseline performance of neural networks with diffraction images as input data.

5 Adapting neural networks for CDI data

Here, we describe our contribution for using neural networks in combination with diffraction images.

First, we show in section 5.1 that the performance of a neural network can be enhanced when using a special activation function after the input layer.

Second, in section 5.2 we benchmark the performance of the neural network when using a smaller amount of training data. The idea is to provide an intuition about how much the prediction capabilities deteriorate when a smaller training dataset is used. This is useful because so far a researcher still has to invest a lot of time preparing the training dataset and, more general, minimizing the time spent looking through the raw data is the ultimate goal for using a neural network in the first place.

Third, in section 5.3 we propose a novel data augmentation in the form of a custom two-point cross correlation map that hardens the network against very noisy data. We show that when using this augmentation the network is more robust to noise from a uniform distribution added on top of the original diffraction image. This simulates the experimental scenario in which a very low signal-to-noise ratio is unavoidable, e.g., during CDI experiments with very limited photon flux Rupp et al. (2017) or very small scattering cross sections as it is the case with upcoming CDI experiments on single biomolecules Ikeda and Kono (2012); Shintake (2008).

5.1 The logarithmic activation function

One of the key additions of this paper is the proposed activation function, formally stated in Equation 11. It is designed to account for the inherent property of diffraction images of scaling exponentially. More general, the intensity distribution of scattered light on a flat detector follows two laws, depending on the scattering angle that is recorded. For very small angles (SAXS and USAXS experiments) the Guinier approximation is the dominant contribution to the recorded intensity, while for larger scattering angles (SAXS and WAXS experiments) Porod’s law becomes dominant Hammouda (2010); Sinha et al. (1988). Where the scattering intensity in the Guinier approximation is proportional to $\approx\exp{\left(-q^{2}\right)}$, in Porod’s law the intensity scales with $\approx q^{-d}$. $q$ is the scattering vector (function of the scattering angle and of the wavelength in use) and $d$ is the so-called Porod coefficient, which can vary significantly depending on the object from which the light was scattered Hammouda (2010).

In any case, the recorded detector intensity for diffraction images scale exponentially. For this reason we propose a logarithmic activation function of the form:

[TABLE]

where $\alpha>0$ is a tunable scaling parameter, $c_{0}=\exp{\left(-1\right)}$, $c_{1}=1$ and $x$ is the input.

We define $c_{0}$ and $c_{1}$ so that the activation function is anti-symmetric around 0, which helps speed up training and avoids a bias shift for succeeding layers Clevert et al. (2015); Szegedy et al. (2014).

Since we are using a gradient-based optimization technique we need to take care that the gradient can propagate throughout the whole network, otherwise it would lead to so-called gradient flow problems, which befalls deep architectures Kolen and Kremer (2001); Nair, V.;Hinton (2010). There are two possibilities for insufficient gradient flow, either the gradients are getting too small (vanishing gradient) or too large (exploding gradient) when propagating throughout the network. Both scenarios lead to numerical instabilities during training making convergence for large architectures very hard or even impossible. The reason for this is the backpropagation algorithm which invokes the chain rule for calculating the gradients. Every gradient is therefore also a multiplicative factor for the gradient of a succeeding layer. For our case the derivative of Equation 11 w.r.t. $x$ is given by:

[TABLE]

It shows that the gradient scales with $x^{-1}$ with a discontinuity of size $\alpha~{}c_{0}^{-1}$ at [math].

If we used this activation function for all activations throughout the network, the gradient would have an increased probability to vanish - or explode - the deeper the architecture gets. In addition to that, the discontinuity at $x=0$ could lead to gradient jumps, which would further decrease numerical stability. Therefore, we use the logarithmic activation function only for the first convolutional layer and use a LeakyRelu activation with leakage of $0.2$ on all hidden layers. This compromise still captures the exponential scale of the diffraction images but without losing numerical stability.

Since $\alpha$ is a tunable hyperparameter, we conduct experiments with three values for $\alpha\in\left[0.2,0.5,1.0\right]$ and evaluate its impact on the performance of the neural network.

In Table 4 we provide the evaluation metrics for ResNet18 used with the logarithmic activation function, trained with three different values for $\alpha$. For comparison, we also provide the results of the unmodified ResNet18 labeled unmodified. The best performing configuration is with an $\alpha$ value of $0.2$, maxing out with an Accuracy of $0.965$. Therefore, providing a boost in Accuracy of a full percentage point compared to the unmodified ResNet18. The lowest value for the maximum Accuracy was reached without the logarithmic activation function, topping at $0.955$. Precision and Recall both increase with the addition of the logarithmic activation function. These improvements all come without increasing training time or complexity of the model.

The maximum achieved Accuracy seems to be anti-correlated to $\alpha$, with the $\text{ResNet18}_{\alpha=1.0}$ variant performing worst. We suspect that this is related to the smaller size of the discontinuity of the derivative of $h(a_{j})$ when choosing a small value for $\alpha$, see Equation 12.

However, choosing even smaller values for $\alpha$ did not improve the Accuracy further, either because the benefit from the activation function plateaus there or because we reached the classification capacity of this ResNet layout.

These results show convincingly that the addition of the logarithmic activation function improves the overall performance and generalization of the deep neural network. This is in so far expected because we imposed a form of feature engineering on the network, by exploiting a known characteristic of the dataset. Therefore, without increasing the complexity, the depth or the training time, we showed that using the logarithmic activation improves all relevant evaluation metrics. For this reason, we use the logarithmic activation function with an $\alpha$ value of $0.2$ as default for all following experiments.

5.2 Size of the training set

In this section, we evaluate the impact of the training set size on the evaluation metrics, we trained the $\text{ResNet18}_{\alpha=0.2}$ with a varying amount of labeled images. The reason for this is to provide intuition for how many images are needed to be classified manually before the employment of a neural network is useful. We uniformly select images from the training set but kept the same evaluation dataset described in section 3.1.5. We decreased the size of the training set in three stages (to $75\text{\,}\mathrm{\char 37\relax}$ $\equiv$ $4631$, to $50\text{\,}\mathrm{\char 37\relax}$ $\equiv$ $3088$ images, to $25\text{\,}\mathrm{\char 37\relax}$ $\equiv$ $1544$ images).

Table 5 shows the performance of $\text{ResNet18}_{\alpha=0.2}$ when trained with datasets of different sizes. For the helium dataset, the maximum achieved Accuracy is dropping from $0.9650.797$ when using only $1544$ images instead of the full $6174$ images. Even more pronounced is the decline in Precision and Recall from $0.922$ and $0.870$ to $0.673$ and $0.593$ for the smallest training set size. The steeper decline rate for Precision and Recall, compared to Accuracy, can be understood as the helium dataset predominantly consists of Negative ground truth labels ($64\,339$ out of $79\,904$ labels) to which the neural networks resorts in the absence of sufficient training data. Precision and Recall, on the other hand, provide only information about the positive prediction capabilities and their completeness and therefore decrease faster when a smaller training set size is used.

This shows that the number of images is critical for the prediction capabilities of the neural network. The drastic decrease in training set size results in a much worse generalization of the model, detecting only those images that are very close to the ones from the training set, missing most from the evaluation set. The network has not learned the characteristics of a particular class to a point where it can transfer the gained knowledge to other images, which is the one critical property for which we employed a neural network in the first place.

Therefore, if time is limited, one may be well advised to concentrate efforts on preparing a sufficiently large, high-quality training dataset while using e.g. our here presented neural network approach in its standard configuration.

5.3 Using two-point cross-correlation maps to be more robust to noise

This section introduces an image augmentation based on the two-point cross-correlation function, which increases the resistance to noise. We prepare four training sets, each with an increasing amount of noise sampled from a uniform distribution and analyze the noise dependence of the neural network.

One of the principal problems in CDI experiments, or imaging experiments in general, is recorded noise. Noise often leads to computational problems due to noise resistance being a known weak point for a significant fraction of predictive algorithms Atla et al. (2011). In particular, deep neural networks are known to be easily fooled by noise. When adding noise to an image, whose addition may be invisible to the human eye, a neural network can come to entirely different conclusions and this even with high confidence; Seeing a panda where there was a wolf Nguyen et al. (2014); Moosavi-Dezfooli et al. (2016). Therefore, we propose an additional pre-processing step for the input images to increase the noise resistance of the neural network.

To quantify the quality of an image, the signal-to-noise ratio is often used. It is a measure for how much noise is present when compared to some information content, where low values indicate that information might be indistinguishable from noise. It has been shown that higher orders of the two-point cross-correlation function (CCF) can act as a frequency dependent noise filter and increase the quality of a reconstruction of a diffraction image even in the presence of recorded noiseKurta et al. (2017); Donatelli et al. (2015). And since the CCF can be interpreted as an image, see Figure 4 e) to h), we employ this method in a similar manner to optimize the use-case with a convolutional deep neural network, expecting that the higher-order terms make the neural network more resistant to the presence of noise.

In general, the CCF is defined as:

[TABLE]

where $\Delta$ is the angular separation, $\phi$ is the angular coordinate, and $\left(i,j\right)$ denotes the index of the two scattering vectors $q_{i}$ and $q_{j}$. For discrete $\phi$ and written as Fourier decomposition, Equation 13 yields Kurta et al. (2017):

[TABLE]

where $n$ denotes the order of the CCF. $I_{i}^{n}$ is given by:

[TABLE]

Since $C_{i,j}=C_{j,i}$, we can split the final correlation map into an upper and a lower triangle matrix. To maximize information, and to optimally use the local receptive fields of the convolutional layers, we merge the lower triangle from the full CCF calculation, Equation 13 with $\Delta=0$, and the upper triangle of order $n=8$ from Equation 14. Therefore, we combine a plain correlation map with a higher order map that is more resistant to noise, see Figure 4 e) to h) for a full example.

To test the robustness of this method, we use the $\text{ResNet18}_{\alpha=0.2}$ and train it with various pre-processed datasets.

From our original dataset we derive three additional datasets that only differ in the amount of noise added. We do this as follows; First, we calculate the mean, the standard deviation (std) and the maximum intensity values of each image in the original dataset. From these values we calculate the median, instead of the mean (due to increased robustness against outliers); ending up with three statistical characteristics describing the intensity distribution throughout all diffraction images. With that, we define three continuous uniform distributions to sample noise from. A continuous uniform distribution is fully defined by an upper and a lower boundary; $a$ and $b$, respectively. The probability for a value to be drawn within these boundaries is equal and non-zero everywhere. For our three noise distributions we always use a lower boundary of [math] and vary the upper boundary so that $b$ is either the mean, the mean + the std. or the maximum of the intensity distribution on the images (the three statistical characteristics described above).

For example, for creating the maximum noise dataset, we looped through every diffraction image and added noise sampled from the maximum noise distribution. We do this for all three noise distributions. From these three noise embedded datasets, as well as our original dataset, we calculate the here proposed CCF maps. This leads to a total of eight data sets; for each of them we train a $\text{ResNet18}_{\alpha=0.2}$. An example of one image in all eight datasets is in Figure 4.

The results for these eight data sets are given in Table 6. The performance of the neural network without added noise is much stronger when using the original diffraction images instead of the CCF maps. However, as soon as noise is added, the performance of the neural network trained on diffraction images deteriorates much faster as compared to the performance with CCF maps as input. When the upper boundary of the added noise excels the median values of mean + std., the neural network is performing better with the CCF maps instead of the original diffraction images. Especially with the noisiest dataset the differences in performance are significant. Precision is increased by $4$ percentage points when using the CCF maps as input, showing that our data augmentation may serve as a helpful asset when dealing with very noisy data.

In general, it is a viable alternative to use the CCF maps as input to the convolutional deep neural network, which should be considered an option in the case of very noisy data where it provides a boost to classification results. The downside is, calculating the CCF for every image comes at an additional computational cost. It took us three full days to calculate the CCF maps for all $39\,879$ images of both datasets on an Intel 6700K quad-core machine using a multi-threaded Python script (Also released on Github).

6 What the neural network saw

Neural networks are often considered being a black box approach. We usually do not impose a-priori knowledge on our model, the network learns this on its own. Although this is part of the reason why they are so successful it also gives rise to doubts about the interpretability of their predictions. Some ways to interpret the processes of decision finding within a trained neural network have been presented in the literature Selvaraju et al. (2017); Chattopadhyay et al. (2017); Zhou et al. (2015); Li et al. (2018). In order to get a better understanding of why our deep neural network assigned images to certain classes, we calculated heatmaps using the GradCam++ algorithm Chattopadhyay et al. (2017). These heatmaps are making visible where the network has looked for in a particular class, which we do by tracing back the gradient flow from the output layer to the last convolutional layer. The network’s class-specific interest directly correlates with this gradient signal because, in essence, we simulate a training step using backpropagation and interpolate the feature maps from the last convolutional layer. A full description of this process is given in appendix D. The output of the GradCam++ algorithm provides contour maps whose amplitude is a normalized measure for how much the gradient would impose corrections on the weights if used during training. This gradient flow directly corresponds to what the network deemed the most relevant regions.

Figure 5 shows the GradCam++ results for the Streak and Bent classes using our best performing network - $\text{ResNet18}_{\alpha=0.2}$. We present results from these classes, because the distinct spatial characteristics are obvious to the human eye. Therefore they are an ideal candidate to test if the neural network understood these characteristics. In each row of Figure 2, a schematic sketch of the key feature together with five randomly selected images from this class are depicted.

The GradCam++ contour maps are overlaid on the image, in addition, the contour levels are also used as an $\alpha$ mask for the diffraction image so that the brightest areas in each plot correspond to the ones with the highest gradient flow. In the case of the Streak class, Figure 5 clearly shows that the neural network was able to identify the dominant streak feature regardless of its orientation or size. Results on the Bent class also show a strong correlation between the shape of the contour maps and the bent shape of the diffraction pattern.

Therefore, combining these metrics and the GradCam++ images we think that the Streak class feature identified by the neural network indeed corresponds to the one seen by the researcher. Also, the Bent class contour maps from the network show a clear resemblance of the feature intended by the researcher, albeit not so strongly pronounced. Although the deep neural network learned these representations on its own, they co-align with the intentions of the researcher. This demonstrates that neural networks are capable of learning these complicated patterns on their own.

7 Summary and outlook

In this paper, we give a general introduction on the capabilities of neural networks and provide results on the first domain adaption of neural networks for the use-case of diffraction images as input data. The main additions of this paper are (i) a novel activation function that incorporates the intrinsic logarithmic intensity scaling of diffraction images, (ii) an evaluation on the impact of different training set sizes on the performance of a trained network and (iii) the use of the point-wise cross-correlation function to improve the resistance against very noisy data. In addition, we provide a large benchmarking routine, utilizing multiple neural network architectures and layouts in appendix A.

We have shown that even in the most basic configuration, convolutional deep neural networks outperform previously established sorting algorithms by a significant margin. More importantly, we improved on these baseline results by modifying the activation function for the first layer. For the case of very noisy data, often a problem in diffraction imaging experiments, we showed that two-point cross-correlation maps as input data instead of the original diffraction images improve the robustness of the classification capabilities of the network. Our results set the stage for using deep learning techniques as feature extractors from diffraction imaging datasets. The ultimate goal will be establishing an unsupervised routine that can categorize and extract essential pieces of information of a large set of diffraction images on its own. We envision for the near future, that the gained insights lead to multiple new approaches regarding neural networks and diffraction data. For example, the MSFT algorithm used in Langbehn et al. (2018), can be used as a generative module in an end-to-end unsupervised classification routine using large synthetic datasets as training data for a neural network. This approach can be extended to utilize these trained networks as an online-analysis tool during the experiments. Furthermore, we hope to develop an unsupervised approach that connects the recent research from Generative Adversarial Network theory Zhang et al. (2018b); Miyato and Koyama (2018); Miyato et al. (2018); Goodfellow (2016) and mutual information maximization Chen et al. (2016) with the results of this paper. Such an approach would allow for finding characteristic classes of patterns within a data set without any a priori knowledge about the recorded data. All of the code, written in Python $3.6+$ and using the Tensorflow framework, is available at Github, free to use under the MIT License 333https://github.com/julian-carpenter/airynet. We hope the community uses and improves the code provided in this repository.

Acknowledgements.

We would like to thank K. Kolatzki, B. Senfftleben, R.M.P. Tanyag, M.J.J. Vrakking, A. Rouzée, B. Fingerhut, D. Engel and A. Lübcke from the Max-Born-Institute, Ruslan Kurta from the European XFEL and Christian Peltz as well as Thomas Fennel from the University of Rostock for fruitful discussions. This work received financial support by the Deutsche Forschungsgemeinschaft under Grant MO 719/13-1, 14-1 and STI 125/19-1 and by the Leibniz Grant SAW/2017/MBI4.

Appendix A Architectural design choices

In this section, we describe and explain our choices for neural network architecture to establish as baseline performance when working with diffraction patterns, before the inclusion of our diffraction specific activation function, see section 5.1 in the main manuscript. We present the theory and background on available architectures and provide results on two architectures with five depth layouts.

There are different layer styles from which we can build a neural network. Nomenclature is that a full arrangement of all layers is called architecture, or configuration, of the network.

For our tests, we use two different neural network architectures, a ResNet and a VGG- Net, both with multiple depth layouts. For the ResNet, we train and evaluate three depth variations ($18$, $50$ and $101$ layers), and for the VGG-Net we train two variants ($16$ and $19$ layers).

The structure of this section is as follows: First, we explain how a convolutional layer works in general. Second, we motivate the derivation of the VGG-Net from preceding architectures, and third, we show how the ResNet architecture can be explained by expanding the core ideas used in the VGG-Net. In the following section, we will then present the results for all the here trained configurations.

Almost every architectural design is empirically derived Szegedy et al. (2016); Schmidhuber (2014); LeCun et al. (2015) and constitutes of multiple combinations of only a few basic layer styles, namely the fully connected layer, convolutional layer, a pooling operation and a batch normalization operation. We discuss the pooling and batch normalization layer only in appendix C, because of their minor role within the neural network. The reader is also referred to the exhaustive overview in Schmidhuber (2014) and LeCun et al. (2015). Since the convolutional layer serves as a fundamental basis for image analysis with neural networks, we explain it here in more detail.

The very basic idea of a convolutional layer is that nearby pixels in an input image are more strongly correlated than more distant pixels, this is called a local receptive field. Therefore, by calculating a convolution over an input image with a trainable filter of size $>1\times 1$ we can approximate these correlations.

In a convolutional layer, $N$ filter, with size $M\times M$, slide over an input image and produce $N$ convolved maps, called feature maps. One filter uses the same weights on all parts of the input image for producing one feature map; this is called weight sharing. Weight sharing reduces not only the complexity of the model but provides a bridge towards the convolution function in mathematics. With weight sharing, we can identify the filter within the convolutional layer as a kernel function from the mathematical convolution function. Figure 6 a) shows a schematic of a convolutional layer with one filter.

This exemplary filter with size $3\times 3$ slides over an image of size $7\times 7$ and producing a feature map of size $3\times 3$. The feature map is smaller than the input image because the filter moves two pixels for each step. This step-size is called stride.

Hereafter we use the notation conv(a, b, c) for a convolutional layer with filter size $\text{a}\times\text{a}$, number of filters $b$ and stride $c$. The example from figure 6 a) could, therefore, be written as $\text{conv}\left(3,~{}1,~{}2\right)$ and would result in $9$ trainable weight parameters plus $1$ bias parameter (not shown in the figure).

This concept was introduced with the LeNet architecture by Lecun et al. (1998) which is considered the seminal work in the field and the first deep convolutional neural network. After Yann LeCun proposed the LeNet architecture, further research Krizhevsky et al. (2012) led to the now de-facto standard for plain convolutional networks, the VGG-Net. Simonyan and Zisserman (2014) proposed the original architecture which consists of up to $19$ weight layers of which $16$ are convolutional layers, and $3$ are fully connected ones.

It is easy to build, easy to train and provides in general good results Schmidhuber (2014); LeCun et al. (2015). For these reasons, we include two variations of it in our tests, namely version D and E, nomenclature is from Simonyan and Zisserman (2014)). Table 7 shows the details of the architecture, using the naming convention we introduced with the convolutional layer.

Simonyan and Zisserman (2014) derived the VGG-Net directly from the LeNet by arguing that three convolutional layers with filter size $3$ and stride $1$ (VGG-Net) achieve better results than only one filter with size $7$ and stride $2$ (LeNet), which equals to the same effective local receptive field size Simonyan and Zisserman (2014). Three layers perform better than one due to having $2$ additional non-linear activation functions and reduced complexity (less weight-parameter because of the smaller filter sizes), which enforces the neural network not only to be more discriminative but to find sparser solutions Simonyan and Zisserman (2014).

Building on the results achieved by the VGG-net, it was shown that the depth of a deep neural network directly relates to its classification capabilities Eldan and Shamir (2015); He et al. (2015); Szegedy et al. (2014). This led to the introduction of the so-called residual skip-connections which further exploit this depth-matters concept He et al. (2015); Szegedy et al. (2014). These residual skip connections are the name-giving components for the ResNet architecture.

In principle, a ResNet still uses the VGG architectural layout but exchanges the convolutional blocks $1$ to $4$ with residual skip connections, compare tables 7 and 8. This exchange drastically reduces the complexity of the whole network while increasing the number of layers.

The VGG-architecture can be broken down into six blocks, one input block, one output block, and four convolutional blocks (see table 7). Block $2$ is the first block in which there are distinctions between VGG variant D and E.

The VGG-net architecture proved that increasing the depth and decreasing the amount and size of the filters increases the accuracy, which ultimately gave rise to the plain skip connections: Blocks of few convolutional layers designed to replace the large amounts of filters in one layer for multiple layers with fewer, and smaller, filters. Two types exist: A classical and a bottleneck skip connection, both differ only in the amount of how much the depth is increased and the complexity decreased.

This addition has so far only modified the depth and complexity of the network and is called a plain network, see He et al. (2015). It performs reasonably well but not significantly better than VGG-net. A residual skip connection differs from a plain skip connection only in adding the identity of its inputs to its outputs. This way all the convolutional layers in a skip connection learn only a residual of their input. This simple technique enables a ResNet to outperform all other convolutional deep neural network architectures Szegedy et al. (2016); He et al. (2016). Figure 6 b) exemplifies a classical residual skip connection. There is still an ongoing debate about why a residual neural network performs so well He et al. (2015); Szegedy et al. (2014); Veit et al. (2016). Research has shown that ResNets find sparser solutions faster due to their layout, and that they behave like ensembles of shallower networks with information flow only activated on $10$ to $34$ layers even when the neural network has a depth of $101$ layers He et al. (2015); Szegedy et al. (2014); Veit et al. (2016).

However, besides empirical success, one of the critical advantages of ResNets is that reaching training convergence is not getting significantly harder when increasing the depth of the neural network, which is usually the case with other architectures. Therefore, the training of very deep residual neural networks is no more difficult than training shallow plain neural networks Li et al. (2016); Szegedy et al. (2016).

For these reasons, we train three variants, with $18$, $50$ and $101$ layers, of a further optimized version of the classical ResNet, called pre-activated ResNet He et al. (2016) (see table 8 for implementation details).

Table 9 shows the overall evaluation metrics on the helium and the CXIDB dataset. Table 10 shows the per-class evaluation metrics for the helium dataset, which are not needed for the CXIDB dataset because predictions on the helium dataset are a multi-class problem whereas predictions on the CXIDB data are single-class. Single-class - or one-hot - problems have identical overall- and per-class-evaluation metrics. We trained all models as described in section 3.1.5 in the main manuscript.

Table 9 shows the overall evaluation metrics as well as the training wall time. $\text{Train Time}_{Max}$ is the time when the neural network achieved the highest accuracy score on the evaluation dataset, and $\text{Train Time}_{Full}$ is the time for training 200 epochs. However, in practice we achieved optimal convergence after training for 70 to 100 epochs. After this the network showed overfitting.

Both VGG models took significantly longer to train than the ResNet variants, needing between $6.5\text{\,}\mathrm{h}6.7\text{\,}\mathrm{h}$, for $200$ epochs on both datasets, whereas training ResNet101 took only $2.8\text{\,}\mathrm{h}$ and $2.6\text{\,}\mathrm{h}$ respectively. Furthermore, the maximum reached accuracy of both VGG networks is more than half a percentage point below the maximum of ResNet101 - $0.959$ compared to $0.964$ for the helium data and $0.970$ vs. $0.978$ for the CXIDB data. Also, accuracy did not change much when increasing the depth from $1619$, precision even decreased slightly and recall remained unchanged.

On the other hand, increasing complexity within the ResNet architecture helped to boost the accuracy from $0.955$ (CXIDB data: $0.973$) with ResNet18 to $0.964$ (CXIDB data: $0.978$) with ResNet101.

For the multi-class results in Table 10, we chose the ResNet101 layout, as it is our best performing configuration. For classes Oblate, Spherical, Streak and Empty a precision of $0.92470.9770$ and a recall of at least $0.9763$ show that the majority of all predictions in these classes were correct and virtually no image was missed.

For the classes Prolate, Bent, Double Rings and Layered, the ResNet reached a good precision, but a recall score of $\approx 0.65$ shows that it missed almost a third of all available images, indicating we failed to generalize the network for these classes.

For Elliptical, Newton Rings and Asymmetric images, the recall of $0.22070.4836$ shows that these images were a lot harder to find, observable in the relatively low precision scores for those classes. Elliptical is the only class of these three where precision is high enough for using the neural network as a predictor. For the Newton Rings and Asymmetric class, with precision scores around $0.6$, the neural network is effectively guessing.

The performance of all variants clearly shows the general good classification capabilities of a convolutional deep neural networks in the use case of diffraction patterns. Even the lowest performing neural network can outperform previous classification approaches by a large margin - compare with Bobkov et al. (2015). In particular, the results of ResNet18 are compelling; it is small, easy to train and has relatively low complexity. Although having only a fraction of trainable parameters, it performed almost always on-par with the much more complex VGG architectures and all this while taking only $0.2\text{\,}\mathrm{h}$ for reaching the maximum accuracy during training.

Therefore, we chose the ResNet18 layout as the default configuration for all the following experiments; it is an ideal compromise between complexity, training time and classification accuracy.

Appendix B Derivation of the binary cross-entropy

Here, we give an derivation for the binary cross entropy (Equation 6 in the main manuscript). We start with the most general form of the cross-entropy given by:

[TABLE]

where $H(p)$ is the Shannon entropy of $p$, and $D_{\mathrm{KL}}(p\|q)$ is the Kullback–Leibler divergence of $p$ and $q$ Goodfellow et al. (2016). This is equivalent to:

[TABLE]

where $p_{i}$ and $q_{i}$ are two probability distributions over the same set of events. $p_{i}$ is the “correct” distribution, and $q_{i}$ is the approximation of $p_{i}$ from the deep neural network. Since we are using a Bernoulli distribution as our probabilistic model there are only two outcomes that one event ($k$) can have: $k\in\left\{0,1\right\}$. The probability for both outcomes of one event and of both distributions can be written as:

[TABLE]

$x$ is some event, $y$ is the ground truth label and $\hat{y}$ is the approximate probability assigned by the deep neural network. Since we are using a sigmoid function at the output of our deep neural network, we can simplify equation 17. Using:

[TABLE]

we can write:

[TABLE]

where $x$ is an event (e.g. the activation in the output layer of the deep neural network) and $y$ is the real label of this event.

Appendix C Further building blocks of deep neural networks

This section describes the pooling layer and the batch normalization layer in more detail. Since these components are not critical for the neural network their explanation is only here in supplemental material.

C.1 Pooling

There are two commonly used variants of pooling layers, the max pool, and the average pool. The idea is to reduce the dimensionality of the output from a preceding layer ($\dim{\bm{x}}=\left(N\times X\right)$) by letting a filter, with size $a\times a$ slide over parts of the image with step size $b$, called stride, and let them perform a down-sample operation.

A max pool filter only takes the maximum value, and a avg pool filter averages over all values, within its perceptive field Lecun et al. (1998); Krizhevsky et al. (2012), this process is equivalent to a convolutional operation but instead of a matrix multiplication with a convolutional kernel the pooling operation is carried out.

C.2 Batch Normalization

Every layer within a deep neural network is to some point modeling the probability distribution given to it by its preceding layer. It is a hierarchical regression problem, which becomes harder if one layer changes key characteristics of the modeled probability distribution (e.g. the mean, variance or the kurtosis). This shift is then further multiplied in every succeeding layer and is therefore dependent on the depth of the network. This phenomenon is called a covariate shift Shimodaira (2000). Although this problem is solved in a deep neural network via domain adaptation, the costs of a covariate shift are usually much longer training times and reduced accuracy Jiang (2008).

For this reason a batch normalization layer (bn) is used to shift the mean of the mini-batch input to zero and to set the variance to one. This significantly reduces the amount of training time and increases accuracy Ioffe and Szegedy (2015). bn consists of 4 steps after which a normalized mini-batch is returned:

Calculated the mini-batch mean:

[TABLE] 2. 2.

Calculated the mini-batch variance:

[TABLE] 3. 3.

Normalize:

[TABLE] 4. 4.

Scale and shift according to adjustable parameter:

[TABLE]

where $y_{i}$ is the normalized output of input $x_{i}$ and $\gamma$ and $\beta$ are adjustable parameter.

Appendix D GradCam++

In chapter $6$ of the main manuscript we show what the neural network deemed the most relevant areas within an input image. We calculated these so-called heatmaps with an algorithm called GradCam++. The main idea is based on Cam Zhou et al. (2015) and Gradcam Selvaraju et al. (2017) and allows for a very intuitive explanation for the decisions made by a convolutional deep neural network Chattopadhyay et al. (2017).

The core principle is that the output of a convolutional deep neural network can be expressed as a linear combination of the globally average pooled feature maps of the last convolutional layer.

[TABLE]

where $A_{ij}^{k}$ is one feature map of all $k$ maps from the last convolutional layer and $w^{c}_{k}$ are the weights for a particular class prediction $c$ of feature map $k$. $Y^{c}$ is the predicted probability that the input image belongs to this certain class $c$. In the GradCam++ formalism the weights can be calculated:

[TABLE]

where $a_{ij}^{kc}$ are the gradient weights and $\text{LeakyReLu}\left(\cdot\right)$ is a rectified linear unit activation function, very similar to the one we used throughout the main manuscript. $a_{ij}^{kc}$ depends only on $A_{ij}^{k}$ and $Y^{c}$ via:

[TABLE]

The final heatmap, often called saliency map, can then be obtained:

[TABLE]

So the algorithm propagates an image forward through the network, then calculates the gradients until the last convolutional layer, and using equation 19 and 20, obtains a heatmap of the areas within the input image that shows the gradient flow from the convolutional layer.