Camera-trap images segmentation using multi-layer robust principal component analysis

TL;DR

This paper introduces a novel multi-layer robust principal component analysis method for segmenting animals in camera-trap images, effectively handling environmental challenges and outperforming existing background subtraction techniques.

Contribution

The paper presents the first application of multi-layer RPCA for camera-trap image segmentation, optimizing parameters and demonstrating superior performance over state-of-the-art methods.

Findings

Achieved 76.17% F-measure on color sequences.

Achieved 69.97% F-measure on infrared sequences.

Outperformed existing background subtraction algorithms.

Abstract

The segmentation of animals from camera-trap images is a difficult task. To illustrate, there are various challenges due to environmental conditions and hardware limitation in these images. We proposed a multi-layer robust principal component analysis (multi-layer RPCA) approach for background subtraction. Our method computes sparse and low-rank images from a weighted sum of descriptors, using color and texture features as case of study for camera-trap images segmentation. The segmentation algorithm is composed of histogram equalization or Gaussian filtering as pre-processing, and morphological filters with active contour as post-processing. The parameters of our multi-layer RPCA were optimized with an exhaustive search. The database consists of camera-trap images from the Colombian forest taken by the Instituto de Investigaci\'on de Recursos Biol\'ogicos Alexander von Humboldt. We analyzed the performance of our method in inherent and therefore challenging situations of camera-trap images. Furthermore, we compared our method with some state-of-the-art algorithms of background subtraction, where our multi-layer RPCA outperformed these other methods. Our multi-layer RPCA reached 76.17 and 69.97% of average fine-grained F-measure for color and infrared sequences, respectively. To our best knowledge, this paper is the first work proposing multi-layer RPCA and using it for camera-trap images segmentation.

Figures

Tables

Table 1. Table 1 : β 𝛽 \beta values and algorithms for the best performances of each experiment.

Experiment	Algorithm	$𝜷$	Avg f-measure
Experiment 1	APG-PARTIAL	0.6	0.7539
Experiment 2	APG-PARTIAL	0.3	0.7393
Experiment 3	NSA2	0.45	0.7259
Experiment 4	NSA2	0.55	0.7261

Equations

$$M=L_0+S_0$$

$$||L|{|}_{\ast }+\lambda ||S|{|}_1$$

$$L+S=M$$

$$M(x,y)=\beta f_t(x,y)+(1-\beta )f_c(x,y)$$

$$\text{f-measure}=2\frac{\text{precision}\ast \text{recall}}{\text{precision}+\text{recall}}$$

Full text

**Index Terms— ** Camera-trap images, Multi-Layer Robust Principal Component Analysis, background subtraction, image segmentation.

1 Introduction

Studying and monitoring of mammals and birds species can be performed using non-invasive sampling techniques. These techniques allow us to observe animal species for conservation purposes, e.g. to estimate population sizes of endangered species. Camera trapping is a method to digitally capture wildlife images. This method facilitates the register of terrestrial vertebral species, e.g. cryptic species. Consequently, camera traps can generate large volumes of information in short periods of time. Thus, the contributions in camera trapping are important for better species conservation decisions.

Camera traps are devices to capture animal images in the wild. These devices consist of a digital camera and a motion detector. They are triggered when the motion sensor detects movement and dependent on the temperature of the source in relation to the environment temperature. Biologist can monitor wildlife with camera traps for detecting rare species, delineating species distributions, monitoring animal behavior, and measuring other biological rates [1]. Camera traps generate large volumes of information, for example a camera trapping study can generate until 200000 images, where 1% of the information is valuable [2]. As a consequence, biologists have to analyze thousands of photographs in a manual way. Nowadays, software solutions cannot handle the increment of the number of images in camera trapping [3]. Accordingly, it is important to develop algorithms to assist the post-processing of camera-trap images.

Background subtraction techniques could help to segment animals from camera-trap images. There is a significant body research of background subtraction focused in video surveillance [4]. Nevertheless, there are not enough methods that can handle the complexity of natural dynamic scenes [5]. Camera-trap images segmentation is important for animal detection and classification. Camera-traps images usually have ripping water, moving shadows, swaying trees and leaves, sun spots, scene changes, among others. Consequently, the models used to segment those types of images should have robust feature extractors. There are some segmentation methods in the literature applied to camera-trap images segmentation. Reddy and Aravind proposed a method to segment tigers on camera-trap images, using texture and color features with active contours [6]. They do not make an objective evaluation of their method. Ren et al. developed a method to segment images from dynamic scenes, including camera-trap images; the method uses Bag of Words (BoW), Histogram of Oriented Gradients (HOG), and graph cut energy minimization [7]. They do not show the results on camera-trap images. Zhang et al. developed a method to segment animals from video sequences, using camera-trap images; the method uses BoW, HOG, and graph cut energy minimization [8]. They obtained 0.8695 of average f-measure on their own camera-trap data set.

Robust Principal Component Analysis (RPCA) is a method derived from Principal Component Analysis. RPCA assumes that a data matrix can be decomposed in a low-rank and a sparse matrix. RPCA has newly seen significant activity in many areas of computer sciences, particularly in background subtraction. As a result, there are some algorithms to solve the RPCA problem [9, 10, 11, 12, 13]. In this work, we proposed a Multi-Layer RPCA approach in order to segment animals from camera-trap images. Our method combines color and texture descriptors as feature extractor, and solve the RPCA problem with some state-of-the-art algorithms. To our knowledge, this paper is the first work in proposing a Multi-Layer RPCA approach and using it for camera-trap images segmentation.

The paper is organized as follows. Section 2 shows material and methods. Section 3 describes the experimental framework. Section 4 presents the experimental results and the discussion. Finally, Section 5 shows the conclusions.

2 Materials and Methods

This section shows a brief explanation of the algorithms and metrics used in this paper.

2.1 Robust Principal Component Analysis

An image can be decomposed in a low-rank and sparse matrix. Equation 1 shows the RPCA problem, where $M$ is the data matrix, $L_{0}$ is the low-rank matrix, and $S_{0}$ is the sparse matrix. The low-rank matrix is the background and the sparse matrix is the foreground in background subtraction.

[TABLE]

The RPCA problem can be solved with the convex program Principal Component Pursuit (PCP). This program computes $L$ and $S$, taking the objective function in the Equation 2, where $||L||_{*}$ denotes the nuclear norm of the low-rank matrix, $||S||_{1}$ denotes the $l_{1}$-norm of the sparse matrix, and $\lambda$ is a regularizing parameter. There are some algorithms to perform PCP such as Accelerated Proximal Gradient (APG), and Augmented Lagrange multiplier (ALM) [14].

[TABLE]

2.2 Multi-Layer Robust Principal Component Analysis

Equation 3 shows the data matrix $M$ in our Multi-Layer RPCA method, where $\beta\in[0,1]$ is a weight value indicating the contribution of the texture function to the overall data matrix. Function $f_{t}(x,y)$ denotes the texture descriptor extracted from each image, using the classic Local Binary Pattern (LBP) [15]. LBP describes the texture of an image using the neighborhood of each pixel. Function $f_{c}(x,y)$ denotes the color transformation for each image, converting to gray scale in this case. Our Multi-Layer RPCA computes the $L$ and $S$ matrices from the $M$ matrix in the Equation 3. Texture descriptors can work robustly into rich texture regions with light variation. However, they do not work in a efficient way on uniform regions such as water, the sky, and others. Color descriptors could overcome the texture descriptor limitation [16]. The Multi-Layer approach proposed in this paper was tested in camera traps for wildlife image segmentation.

[TABLE]

2.3 Evaluation Metrics

The f-measure metric was chosen to evaluate the performance of the Multi-Layer RPCA. Equation 4 shows the f-measure, where precision and recall are extracted from the confusion matrix. Precision is the proportion of predicted positives that are correctly real positives. In the same way, recall denotes the proportion of the real positives that are correctly predicted [17]. The confusion matrix is computed comparing the ground truth (GT) with the automatic segmented images.

[TABLE]

3 Experimental Framework

This section introduces the database used in this paper, the experiments executed, and the implementation details of our Multi-Layer RPCA.

3.1 Database

The Alexander von Humboldt Institute realizes samplings with camera traps in different regions of the Colombian forest. We select 25 cameras from 8 regions, where each camera has a relative unalterable environment. Each camera was placed in its site between 1 to 3 months. We extract 30 days and 30 nights of images from those cameras, in daytime color and nighttime infrared formats respectively. The database consists of 1065 GT images from the 30 days and 30 nights. The images have a spatial resolution of 3264x2448 pixels. The length of each day or night data set varies from 9 to 72 images, depending on the animal activity that day or night. Figure 1 shows an example of the GT images.

3.2 Experiments

The experiments computed the background models with different conditions and amount of images, observing the robustness of the Multi-Layer RPCA and the influence of pre-processing in the results. All experiments performed our method with $\beta=[0,0.05,0.1,0.15,\dots,1]$. Accordingly, Experiment 1 uses histogram equalization as pre-processing in the color transformed image, Figure 2 shows the pre-processing for each raw image. The background model is computed with entire days e.g. we take all images of day 1 and solve the RPCA problem in the Experiment 1. Experiment 2 computes the background model with entire nights; Figure 3 shows the pre-processing for each raw image. Experiment 3 takes entire days and nights e.g. we take all images of day 1 and night 1 to solve the RPCA problem. Experiment 3 uses two pre-processes, daytime images uses the pre-process in Figure 2 and nighttime images uses the pre-process in Figure 3. Experiment 4 takes entire days and nights such as the Experiment 3, but it only uses the pre-processing in Figure 2 for all images.

We tested 9 algorithms to solve the RPCA problem in this paper. Active Subspace RPCA (AS-RPCA) [9]; Exact ALM (EALM), Inexact ALM (IALM), Partial APG (APG-PARTIAL) and APG [10]; Lagrangian Alternating Direction Method (LSADM) [11]; Non-Smooth Augmented Lagrangian v1 (NSA1) and Non-Smooth Augmented Lagrangian v2 (NSA2) [12]; Probabilistic Robust Matrix Factorization (PRMF) [13].

The foreground was obtained applying a post-process to the sparse matrix. The post-processing was the same for all experiments. This stage includes a hard threshold, morphological filters, and an active contours with a negative contraction bias [18]. Figure 4 shows the post-processing used. Finally, The f-measure was computed comparing each GT with each foreground. The average f-measure was computed as the mean of all f-measures. The results are displayed as a plot of the average f-measure vs $\beta$.

3.3 Implementation Details

The RPCA algorithms were computed using the Sobral et al. library [19]. The rest of the source code was developed using the image processing toolbox of Matlab.

4 Results

This section shows the results and discussions of the experiments introduced in the Section 3. These results use the metrics explained in the Section 2.3.

Figures 5(a) and 5(b) show the average f-measure vs $\beta$ of the Experiments 1 and 2 for all RPCA algorithms chosen. Table 1 shows the summary of the best results for each experiment. APG-PARTIAL was the best algorithm in the Experiments 1 and 2. Daytime images have rich texture regions. In contrast, nighttime images have uniform color. Texture representations are more important on daytime images in the Experiment 1 due to $\beta=0.6$. On the contrary, color descriptors are more important on nighttime images in the Experiment 2 due to $\beta=0.3$. Those results show the importance of combining the color and texture descriptors. Figure 5 shows the performances of the RPCA normal algorithms when $\beta=0$. Thus, our Multi-Layer RPCA outperforms the RPCA normal methods.

Figures 5(c) and 5(d) show the average f-measure vs $\beta$ of the Experiments 3 and 4. Table 1 shows that dividing the pre-processing per daytime or nighttime in the Experiment 3 does not make a big difference in the results, but it increases the fine-tuning parameters. Table 1 shows that NSA2 was the best algorithm in the Experiments 3 and 4, contrary to the Experiments 1 and 2 where APG-PARTIAL was the best. NSA2 algorithm is a better choice than other RPCA algorithms, if we cannot differentiate between daytime and nighttime images, or if it is difficult to do so. On the other hand, APG-PARTIAL is better, if we have information about the infrared activation.

Figure 6 shows two visual results of the Multi-Layer RPCA. Figure 6(a) shows a daytime image without any pre-processing. Figure 6(d) shows an original nighttime image. Figures 6(b) and 6(e) show the sparse matrix after the hard threshold. Figures 6(c) and 6(f) show the foreground image. These color results are made with the GT images. Yellow-colored regions mean pixels that are on the GT and the automatic segmented images. Red and green regions are visual representations of the under and over segmentation.

5 Conclusions

We proposed a Multi-Layer RPCA for camera-trap image segmentation, using texture and color descriptors. The proposed algorithm is composed of pre-processing, RPCA algorithm, and post-processing. The pre-processing uses histogram equalization, Gaussian filtering, or a combination of both. The RPCA algorithm computes the sparse and low-rank matrices for background subtraction. The post-processing computes morphological filters and an active contours with a negative contraction bias. We proved the Multi-Layer RPCA algorithm in a camera-trap images database from the Colombian forest. The database was manually segmented to extract the f-measure of each automatic segmented image. We reach 0.7539 and 0.7393 of average f-measure in daytime and nighttime images respectively. The average f-measure was computed with all GT images. To our best knowledge, this paper is the first work in proposing Multi-Layer RPCA and using it for camera-trap images segmentation.

Acknowledgment. This work was supported by the Colombian National Fund for Science, Technology and Innovation, Francisco José de Caldas - COLCIENCIAS (Colombia). Project No. 111571451061.