Reducing Anomaly Detection in Images to Detection in Noise

TL;DR

This paper proposes a novel unsupervised method for anomaly detection in images by transforming the problem into detecting anomalies in residual noise images, simplifying background modeling and enabling rigorous thresholding.

Contribution

It introduces a noise-based anomaly detection approach that reduces complex background modeling to residual noise analysis, applicable to arbitrary images.

Findings

Effective in detecting anomalies across various image backgrounds

Enables rigorous threshold setting using a contrario detection theory

Unsupervised method applicable to diverse image types

Abstract

Anomaly detectors address the difficult problem of detecting automatically exceptions in an arbitrary background image. Detection methods have been proposed by the thousands because each problem requires a different background model. By analyzing the existing approaches, we show that the problem can be reduced to detecting anomalies in residual images (extracted from the target image) in which noise and anomalies prevail. Hence, the general and impossible background modeling problem is replaced by simpler noise modeling, and allows the calculation of rigorous thresholds based on the a contrario detection theory. Our approach is therefore unsupervised and works on arbitrary images.

Equations

$$\hat{P}=\frac{1}{Z}\sum _{i=1}^n\exp \left(-\frac{\Vert P-P_i{\Vert }_2^2}{h^2}\right)P_i$$

$$f(i,\mathbf{x})=N\mathbb{P}(|X_i|\ge |x_i|),$$

Full text

Index Terms—  Anomaly detection, Saliency, Self-similarity

1 Introduction

Anomalies are image regions not conforming with the rest of the image. Detecting them is a challenging image analysis problem, as there seems to be no straightforward definition of what is (ab)normal for a given image.

Anomalies in images can be high-level or low-level outliers. High-level anomalies are related to the semantic information presented in the scene. For example, human observers immediately detect a person inappropriately dressed for a given social event. In this work, we focus on the problem of detecting anomalies due to low or mid level rare local patterns present in images. This is an important problem in many industrial, medical or biological applications.

We introduce in this paper an unsupervised method for detecting anomalies in an arbitrary image. The method does not rely on a training dataset of normal or abnormal images, neither on any other prior knowledge about the image statistics. It directly detects anomalies with respect to residual images estimated solely from the image itself. We only use a generic, qualitative background image model: we assume that anything that repeats in an image is not an anomaly. In a nutshell, our method removes from the image its self-similar content (considered as being normal). The residual is modeled as colored Gaussian noise, but still contains the anomalies according to their definition: they do not repeat.

Detecting anomalies in noise is far easier and can be made rigorous and unsupervised by the a-contrario theory [1] which is a probabilistic formalization of the non-accidentalness principle [2]. The a-contrario framework has produced impressive results in many different detection or estimation computer vision tasks, such as, segment detection [3], spots detection [4], vanishing points detection [5], mirror-symmetry detection [6], among others. The fundamental property of the a-contrario theory is that it provides a way for automatically computing detection thresholds that yield a control on the number of false alarms (NFA). It favorably replaces the usual p-value when multiple testing is involved. It follows that not only one can detect anomalies in arbitrary images without complex modeling, but in addition the anomalies are associated an NFA which is often very small and therefore offers a strong guarantee of the validity of the detection. We shall show detections performed directly on the image residual, or alternatively on residuals extracted from dense low and mid-level features of the VGG neural net [7].

The paper is organized as follows. Section 2 discusses previous work while Section 3 explains the proposed method and its implementation. Section 4 presents results of the proposed method on real/synthetic data, and a comparison to other state-of-the-art anomaly detectors. We finally close in Section 5.

2 Related Work

The 2009 review [8] examining about 400 papers on anomaly detection considered allegedly all existing techniques and application fields. It is fairly well completed by the more recent [9] review. These reviews agree that classification techniques like SVM can be discarded, because anomalies are generally not observed in sufficient number and lack statistical coherence. There are exceptions like the recent method [10] which defines anomalies as rare events that cannot be learned, but after estimating a background density model, the right detection thresholds are nevertheless learned from anomalies. A broad related literature exists on saliency measures, for which learning from average fixation maps by humans is possible [11]. Saliency detectors try to mimic the human visual perception and in general introduce semantic prior knowledge (e.g., face detectors). This approach works particularly well with neural networks trained on a base of detect/non-detect with ground truth obtained by for example, gaze trackers[12].

Anomaly detection has been generally handled as a “one class” classification problem. In [13] authors concluded that most research on anomaly detection was driven by modeling background data distributions, to estimate the probability that test data do not belong to such distributions [4, 14, 15, 16]. Autoencoders neural networks can be used to model background [17, 18]. The general idea is to compute the norm between the input and a reconstruction of the input. Another successful background based method is the detection of anomalies in periodic patterns of textile [19, 20]. In [21, 22], center surround detectors based on color, orientation and intensity filters are combined to produce a final saliency map. Detection in image and video is also done in [23] with center-surround saliency detectors which stem from [24] adopting similar image features. In [14], the main idea is to estimate the probability of a region conditioned on the surroundings. A more recent non parametric trend is to learn a sparse dictionary representing the background (i.e., normality) and to characterize outliers by their non-sparsity [25, 26, 27, 28, 29].

The self-similarity principle has been successfully used in many different applications [30, 31]. The basic assumption of this generic background model, is that in normal data, features are densely clustered. Anomalies instead occur far from their closest neighbors. This idea is implemented by clustering (anomalies being detected as far away from the centroid of their own cluster), or by simple rarity measurements based on nearest neighbor search (NN) [32, 33, 34].

Background probabilistic modeling is powerful when images belong to a restricted class of homogeneous objects, like textiles. But, regrettably, this method is nearly impossible to apply on generic images. Similarly, background reconstruction models based on CNNs are restrictive and do not rely on provable detection thresholds. Center-surround contrast methods are successful for saliency enhancement, but lack a formal detection mechanism. Being universal, the sparsity and the self-similarity models are tempting and thriving. But again, they lack a rigorous detection mechanism, because they work on a feature space that is not easily modeled.

We propose to benefit of the above methods while avoiding their mentioned limitations. To this aim, we do construct a probabilistic background model, but it is applied to a new feature image that we call the residual. This residual is obtained by computing the difference between a self-similar version of the target image and the target itself. Being not self-similar, this background is akin to a colored noise. Hence a hypothesis test can be applied, and more precisely multiple hypothesis testing (also called a contrario method), as proposed in [4]. In that way, we present a general and simple method that is universal and detects anomalies by a rigorous threshold. It does not require learning, and it is easily made multiscale.

3 Method

Our method is built on two main blocks: a removal of the self-similar image component, and a simple statistical detection test on the residual based on the a contrario framework.

3.1 Construction of the residual image

The proposed self-similarity based background subtraction is inspired from patch-based non-local denoising algorithms, where the estimate is done from a set of similar patches [31]. This search is generally performed locally around each patch [35, 31] to keep computational cost low and to avoid noise overfitting. The main difference with non-local denoisers is that we forbid local comparisons. The nearest neighbor search is performed outside a square region surrounding each query patch. This square region is defined as the union of all the patches intersecting the query patch. Otherwise any anomaly with some internal structure might be considered a valid structure. What matters is that the event represented by the anomaly is unique, and this is checked away from it.

For each patch $P$ in the image the $n$ most similar patches denoted by $P_{i}$ are searched and averaged to give a self-similar estimate,

[TABLE]

where $Z=\sum_{i=1}^{n}\exp\left(-\frac{\|P-P_{i}\|_{2}^{2}}{h^{2}}\right)$ is a normalizing constant, and $h$ is a parameter.

Since each pixel belongs to several different patches, they will therefore receive several distinct estimates that can be averaged. Algorithm 1 gives a generic pseudocode for this process, which ends with the generation of a residual image $r(u)$ allegedly containing only noise and the anomalies (see Figure 1). The intuition is that it is much easier to detect anomalies in $r(u)$ than in $u$.

3.2 Statistical detection by the a contrario approach

Our goal is to detect structure in the residual image $r(u)=\hat{u}-u$. We are in a much better situation modeling $r(u)$ than $u$. Indeed, contrarily to $u$, $r(u)$ is by construction unstructured and akin to a colored noise (as illustrated in Fig. 1). In what follows we assume that $r(u)$ is a spatial stationary random process and follow [4], who proposed automatic detection thresholds in any colored Gaussian noise.

Given a set of random variables $(X_{i})_{i\in[|1,N|]}$ a function $f$ is called an NFA if it guarantees a bound on the expectation of its number of false alarms under the null-hypothesis, namely, $\forall{\epsilon>0},\mathbb{E}[\#\{i,f(i,X_{i})\leq\epsilon\}]\leq\epsilon$. In other words, thresholding all the $f(i,X_{i})$ by $\epsilon$ should give up to $\epsilon$ false alarms when $(X_{i})_{i\in[|1,N|]}$ verifies the null-hypothesis. In our case, we consider

[TABLE]

Where $i$ index among the $N$ executed tests (detailed below), $X_{i}$ is a random variable distributed as the residual at position $i$, and $x_{i}$ the actual measured value (pixel or feature value) at position $i$. The null-hypothesis is that the residual, represented by $(X_{i})_{i\in[|1,N|]}$, verifies that each $X_{i}$ follows a standard normal distribution. Independence is not required.

Residual distribution. In practice the distribution of the residual $r(u)$ is not necessarily Gaussian. A careful study of the residual distribution lead us to consider that it follows a generalized Gaussian distribution (GCD). We approximately estimate the GCD parameters, and then apply a non-linear mapping to make it normally distributed.

Choice of NFA. The choice of the NFA given in (2) enables to detect anomalies in both tails of the Gaussian distribution (i.e., very bright or very dark spots). To detect anomalies of all sizes, the detection is carried out independently at $N_{\text{scales}}$ scales computed from the residual at the original resolution (by Gaussian subsampling of factor two). Let us denote by $\Omega_{s}$ the set of pixels in the residual image at scale $s$ having $N_{\text{feat}}$ number of features. When working with colored noise, Grosjean and Moisan [4] propose to convolve the noise with a measure kernel to detect spots of a certain size. This corresponds to the generation of new image features $\bar{r}(u)=r(u)\ast K$, where $K$ is a disk of a given radius. This idea is used in our framework, where the residual is convolved with kernels of small sizes. Since we apply the detection at all dyadic scales, the tested radii are limited to a small set of $N_{\text{kernel}}$ values (1,2 to 3) at each scale. Because the residual is assumed to be a stationary Gaussian field, the result after filtering is also Gaussian. The variance is estimated and the filtered residuals are normalized to have unit variance. This is the input to the NFA (2) computation (i.e., $\textbf{x}_{i}$). Thus, the inputs to the detection phase are multi-channel images of different scales, where each pixel channel, representing a given feature, follows a standard normal distribution.

Then, the number of tests is $N=N_{\text{kernel}}\cdot N_{\text{feat}}\cdot\sum_{i=0}^{N_{\text{scales}}-1}|\Omega_{s}|.$

3.3 Choice of the image features

Anomaly detectors work either directly on image pixels or on some feature space but the detection in the residual, which is akin to unstructured noise, is fairly independent of the choice of the features. We used with equal success the raw image color pixels, or some intermediate feature representation extracted from the VGG convolutional neural network [7]. To compress the dynamical range of the feature space we apply a square root function to the network features.

In order to reduce the feature space dimension, we compute the principal components (PCA) and keep only the first five. This is done per input image independently.

Parameters. The main method parameter is the number of allowed false alarms in the statistical test. In all presented experiments, we set NFA=$10^{-2}$. Hence, an anomaly is detected at pixel $\mathbf{x}$ in channel $i$ iff the NFA function $f(i,\mathbf{x})$ is below $\epsilon=10^{-2}$. This implies a (theoretical) expectation of less than $10^{-2}$ “casual” detection per image under the null hypothesis that the residual image is noise. Obviously the lower the NFA the better. Most anomalies have a much lower NFA. For the basic method working on image pixels we used two disks of radius one and two, while for the neural network features, we add a third disk of radius three. The number of scales is set to $N_{scale}=4$ in all tests. The patch size in Alg. 1 is $8\!\times\!8\!\times\!3$ for the pixels variant, while when using neural nets features, we use a patch size of $5\!\times\!5\!\times\!5$. The number of nearest patches is always set to $n=16$, and $h=10$. Results presented herein use the outputs from VGG-19 layers conv1_1, conv2_1 and conv3_1.

4 Experiments

In absence of a valid test image database for anomalies, we used the most common images proposed in the literature (see Fig. 2) and we adopted the following comparison methodology, that was applied to our method and to other four state-of-the-art ones for comparison:

a) Sanity check: verifying that for toy examples proposed in the literature the sole detection is the anomaly;

b) Theoretical sanity check: verify the a contrario principle: ”no detection in white noise”

c) Classic challenging images: we verify the detector power on classic challenging images of the literature: side scan sonar, textile, mammography and natural images. In the case of the mammography where one paper computed an NFA, we verify crucially that by computing the NFA on the residual instead of the image, we gain a huge factor, the NFA being divided by eleven orders of magnitude.

We tested our proposed anomaly detector on two different input image representations: the basic one, pixels, directly applies the anomaly detection procedure to the residuals obtained from the color channels, and three different variants using as input features extracted at different levels from the VGG network [7], namely, very low level  (conv1_1), low level (conv2_1), and medium level  (conv3_1) features. As we shall check the four detections are similar and can be fused by a mere pixel union of all detections.

Existent anomaly detectors are often tuned for specific applications, which probably explains the poor code availability. We compared to Mishne and Cohen [36], a state-of-the-art anomaly detector with available code, to the salient object detector DRFI [37] (which is state-of-the-art according to [40]), and to the state-of-the-art human gaze predictor SALICON [12]. We also compared to the Itti et al. salient object detector [21], which works reasonably well for anomaly detection. All methods produce saliency maps where anomalies have the highest score. Anomalies for Mishne and Cohen are red-colored, while the other methods don’t have a threshold for anomalies. More results are available in the supplementary materials.

Synthetic images. The proposed method performs well on synthetic examples as shown in Figure 2). Some weak false detections are found when using as input features extracted at different layers of the VGG net. All the other compared methods miss some detections. SALICON successfully detects the anomalous density on the fourth example but misses several anomalies in others or introduces numerous wrong detections. Itti et al. method successfully detects the anomalous color structure in the first example, but fails to detect the other ones. Mishne and Cohen and DRFI methods do not perform well on any of the five synthetic examples.

Real images. The comparison on real images is more intricate and requires looking in detail to find out whether detections make sense (Figure 2). In the garage door (fourth row), there are two detections that stand out (lens flare and red sign), some others – less visible – can be found (door scratches or holes in the brick wall). For our method, the main detections are present in all the variants. There are also specific anomalies that can be detected only at a given layer of the neural network. For example, conv1_1 detects the holes in the brick wall and the gap between the garage door and the wall, in addition to the ones detected with pixels input. The variants conv2_1 and conv3_1 detect a missing part of a brick in the wall. Saliency methods detect the red sign but not the lens flare. Mishne and Cohen one only detects the garage door gap. The second real example is a man walking in front of some trees. Our method detects the man with pixels and conv1_1. DRFI and SALICON detect the man while Mishne and Cohen and Itti et al. do not. The third real example is a radar image showing a mine, while the last example is a defect in a periodic textile. All methods detect the anomalies, with more or less precision. Note that the detection in the top right corner for both pixels and conv1_1 (and only these) correspond to a defect inside the periodic pattern.

Comparison to the a contrario method of Grosjean and Moisan [4]. This a contrario method is designed to detect spots in colored noise textures, and was applied to the detection of tumors in mammographies. This detection algorithm is the only other one computing NFAs, and we can directly compare them to ours. The detection results on a real mammography (having a tumor) are shown in Figure 3. With our method the tumor is detected with a much significant NFA (NFA of $10^{-12}$ whereas in [4] NFA of $0.15$). Our self-similar anomaly detection method shows fewer false detections, actually corresponding to rare events like the crossings of arterials.

5 Conclusion

We have shown that anomalies are easier detected on the residual image, computed by removing the self-similar component, and then performing hypothesis testing. It is reassuring to see that our method finds all anomalies proposed in the literature with very low NFA. In addition, we have experimentally shown that the method verifies the non-accidentalness principle: no anomalies are detected in white noise. We plan to build a database of test images with anomalies to run extensive validation and comparison. We also plan to extend the method to videos, by analyzing anomalies in the motion field.