Similarity Learning Networks for Animal Individual Re-Identification -- Beyond the Capabilities of a Human Observer

TL;DR

This paper demonstrates that deep similarity comparison networks, especially Triplet Loss, outperform traditional methods and even human performance in animal re-identification across multiple species, using various CNN architectures.

Contribution

It introduces the application of similarity comparison networks to animal re-identification and shows their superior performance across diverse species without species-specific modifications.

Findings

Triplet Loss outperforms Siamese networks across all species.

Similarity networks surpass human re-identification performance.

Models generalize well across different animal species.

Abstract

Deep learning has become the standard methodology to approach computer vision tasks when large amounts of labeled data are available. One area where traditional deep learning approaches fail to perform is one-shot learning tasks where a model must correctly classify a new category after seeing only one example. One such domain is animal re-identification, an application of computer vision which can be used globally as a method to automate species population estimates from camera trap images. Our work demonstrates both the application of similarity comparison networks to animal re-identification, as well as the capabilities of deep convolutional neural networks to generalize across domains. Few studies have considered animal re-identification methods across species. Here, we compare two similarity comparison methodologies: Siamese and Triplet-Loss, based on the AlexNet, VGG-19, DenseNet201, MobileNetV2, and InceptionV3 architectures considering mean average precision (mAP)@1 and mAP@5. We consider five data sets corresponding to five different species: humans, chimpanzees, humpback whales, fruit flies, and Siberian tigers, each with their own unique set of challenges. We demonstrate that Triplet Loss outperformed its Siamese counterpart for all species. Without any species-specific modifications, our results demonstrate that similarity comparison networks can reach a performance level beyond that of humans for the task of animal re-identification. The ability for researchers to re-identify an animal individual upon re-encounter is fundamental for addressing a broad range of questions in the study of population dynamics and community/behavioural ecology. Our expectation is that similarity comparison networks are the beginning of a major trend that could stand to revolutionize animal re-identification from camera trap data.

Tables

Table 1. Table 1: Summary of Data Splitting

Species	Ratio	Num
	Num Images	Individuals
FaceScrub
	0.7	414	73,735
	0.1	52	10,152
	0.2	52	22,976
ChimpFace
	0.7	67	3,891
	0.1	10	588
	0.2	19	1,120
HappyWhale
	0.7	2,978	6,914
	0.1	425	985
	0.2	850	1,951
FruitFly
	0.4	8	97,904
	0.1	2	24,476
	0.5	10	122,380
ATWR
	0.7	64	1467
	0.1	9	163
	0.2	18	250

Table 2. Table 2: Summary of Performance Metrics for Siamese Similarity Learning Models by Species & Data Set

Species	Model	mAP@1	mAP@5
Human	AlexNet	0.699 $\pm$ 0.342	0.721 $\pm$ 0.332
	VGG19	0.680 $\pm$ 0.288	0.703 $\pm$ 0.251
	DenseNet201	0.734 $\pm$ 0.277	0.835 $\pm$ 0.875
	ResNet152	0.756 $\pm$ 0.282	0.856 $\pm$ 0.123
	InceptionV3	0.713 $\pm$ 0.227	0.854 $\pm$ 0.126
Chimpanzee	AlexNet	0.639 $\pm$ 0.221	0.863 $\pm$ 0.121
	VGG19	0.645 $\pm$ 0.168	0.884 $\pm$ 0.094
	DenseNet201	0.725 $\pm$ 0.134	0.871 $\pm$ 0.064
	ResNet152	0.775 $\pm$ 0.134	0.901 $\pm$ 0.097
	InceptionV3	0.743 $\pm$ 0.106	0.869 $\pm$ 0.125
Whale	AlexNet	0.509 $\pm$ 0.385	0.662 $\pm$ 0.334
	VGG19	0.543 $\pm$ 0.397	0.669 $\pm$ 0.410
	DenseNet201	0.521 $\pm$ 0.445	0.691 $\pm$ 0.312
	ResNet152	0.563 $\pm$ 0.202	0.737 $\pm$ 0.298
	InceptionV3	0.576 $\pm$ 0.203	0.722 $\pm$ 0.390
Fruit Fly	AlexNet	0.621 $\pm$ 0.078	0.875 $\pm$ 0.064
	VGG19	0.590 $\pm$ 0.081	0.838 $\pm$ 0.090
	DenseNet201	0.638 $\pm$ 0.153	0.843 $\pm$ 0.180
	ResNet152	0.693 $\pm$ 0.098	0.896 $\pm$ 0.109
	InceptionV3	0.522 $\pm$ 0.021	0.873 $\pm$ 0.143
Tiger	AlexNet	0.794 $\pm$ 0.396	0.858 $\pm$ 0.289
	VGG19	0.735 $\pm$ 0.245	0.821 $\pm$ 0.243
	DenseNet201	0.803 $\pm$ 0.398	0.8756 $\pm$ 0.148
	ResNet152	0.789 $\pm$ 0.320	0.877 $\pm$ 0.172
	InceptionV3	0.701 $\pm$ 0.307	0.843 $\pm$ 0.231

Table 3. Table 3: Summary of Performance Metrics for Triplet Loss Similarity Learning Models by Species & Data Set

Species	Model	mAP@1	mAP@5
Human	AlexNet	0.739 $\pm$ 0.284	0.804 $\pm$ 0.345
	VGG19	0.811 $\pm$ 0.325	0.843 $\pm$ 0.251
	DenseNet201	0.914 $\pm$ 0.299	0.947 $\pm$ 0.187
	ResNet152	0.886 $\pm$ 0.301	0.952 $\pm$ 0.093
	InceptionV3	0.903 $\pm$ 0.235	0.940 $\pm$ 0.124
Chimpanzee	AlexNet	0.739 $\pm$ 0.241	0.886 $\pm$ 0.166
	VGG19	0.734 $\pm$ 0.188	0.890 $\pm$ 0.085
	DenseNet201	0.792 $\pm$ 0.164	0.932 $\pm$ 0.049
	ResNet152	0.811 $\pm$ 0.155	0.961 $\pm$ 0.097
	InceptionV3	0.756 $\pm$ 0.136	0.940 $\pm$ 0.075
Whale	AlexNet	0.679 $\pm$ 0.374	0.752 $\pm$ 0.274
	VGG19	0.713 $\pm$ 0.349	0.801 $\pm$ 0.287
	DenseNet201	0.691 $\pm$ 0.304	0.771 $\pm$ 0.253
	ResNet152	0.733 $\pm$ 0.252	0.830 $\pm$ 0.275
	InceptionV3	0.746 $\pm$ 0.243	0.804 $\pm$ 0.290
Fruit Fly	AlexNet	0.671 $\pm$ 0.158	0.935 $\pm$ 0.041
	VGG19	0.608 $\pm$ 0.161	0.954 $\pm$ 0.120
	DenseNet201	0.660 $\pm$ 0.194	0.978 $\pm$ 0.084
	ResNet152	0.743 $\pm$ 0.163	0.986 $\pm$ 0.089
	InceptionV3	0.561 $\pm$ 0.125	0.967 $\pm$ 0.131
Tiger	AlexNet	0.830 $\pm$ 0.296	0.978 $\pm$ 0.217
	VGG19	0.770 $\pm$ 0.205	0.940 $\pm$ 0.145
	DenseNet201	0.863 $\pm$ 0.193	0.974 $\pm$ 0.148
	ResNet152	0.811 $\pm$ 0.124	0.996 $\pm$ 0.072
	InceptionV3	0.731 $\pm$ 0.117	0.933 $\pm$ 0.121

Full text

1 Introduction

Recent decades have witnessed the emergence of deep learning systems that make use of large data volumes [1]. Modern deep learning systems no longer require ‘hard-coded’ feature extraction methods. Instead, these algorithms can learn, through their exposure to many examples, the particular features that allow for the discrimination of individuals. [2]. Deep learning methods have shown great success when considering computer vision re-ID tasks with large amounts of data. However, for the task of animal re-ID, gathering a library of images for every individual within a population is infeasible.

Similarity comparison networks, such as Siamese and Triplet-Loss networks, are a popular alternative to standard deep networks and have shown success re-identifying (re-ID) human individuals [3]. Rather than traditional softmax classification outputs, Siamese networks consider pairs of inputs and classify them as either similar or dissimilar. For re-ID, this extends to determining if two input images are of the same individual. We believe the capabilities of these systems can extend beyond those of humans. To benchmark similarity comparison networks and understand their limitations, we compare such systems in a domain that has received little focus: non-human animals. Animal species provide an excellent test bed for the capabilities of similarity comparison networks as the characteristics that distinguish animal individuals are often much more subtle than that of humans. Here we explore five architectural variants of the Siamese paradigm: AlexNet, VGG-19, DenseNet201, MobileNetV2, and InceptionV3 to test their ability to re-ID individuals of five species: humans, chimpanzees (Pan spp.), humpback whales (Megaptera novaeangliae), fruit flies (Drosophila melanogaster), and Siberian tigers (Panthera tigris altaica)[4, 5, 6, 7, 8].

Current practice requires years of training and practical experience. Ecologists rely on a variety of techniques for re-ID including: tagging, scarring, banding, and DNA analyses of hair follicles or feces [9]. While accurate, these techniques are laborious for the field research team, intrusive to the animal, and often expensive for the researcher.

Re-identification from camera trap images is a desirable alternative for ecologists due to its lower cost and reduced workload for field researchers. Despite its advantages, there are a number of practical and methodological challenges associated with its use. Primarily, even among experienced researchers, there remains an opportunity for human error and bias [10, 11]. Historically, these limitations have restricted the use of camera traps to the re-ID of animals that bear conspicuous individual markings [10].

Our objective is to test the generalization of similarity comparison networks considering re-ID tasks across multiple species as well as whether a deep learning system can expedite and reduce the human biases inherent to the task of re-identifying animals from camera trap images. Animal re-ID is used for a variety of ecological metrics, including diversity, relative abundance distribution, and carrying capacity [9]. By training an animal re-ID system, one could automate the collection of metrics relevant to health projection of ecosystem stability and species population health.

2 Brief History of Computer Vision Methods for Animal Re-Identification

Prior to the advent of deep learning, for decades, the approach to standardizing the statistical analysis of animal re-ID has involved computer vision. ‘Feature engineering’ has been the most commonly used act of engineering as programming where algorithms are designed and implemented to focus exclusively on predetermined traits, such as the detection of patterns of spots or stripes, to discriminate among individuals. The main limitations of this approach surround its impracticality [12]. Feature engineering requires programming experience, sufficient familiarity with the organisms to identify relevant features, and lacks in generality where once a feature detection algorithm has been designed for one species, it is unlikely to be useful for other taxa. For a comprehensive review of computer vision relevant to animal re-ID see Schneider et al. [13].

The success of deep learning methods for human re-identification is well documented when ample training images are available for each individual. In 2015, using standard convolutional architectures, two research teams, Lisanti et al. [14] and Martinel et al. [15] demonstrated the success of CNNs on human re-ID using ETHZ, a data set composed of 8580 images of 148 unique individuals taken from mobile platforms. CNNs were able to correctly classify individuals after seeing 5 images of an individual. Taigman et al. [16] introduced Deepface, a method of creating a 3-dimensional representation of the human face to provide more data to a neural network which improved classification accuracy on the YouTube faces data set containing videos of 1,595 individuals.

Despite the success of deep learning methods for human re-ID, few ecological studies have realized its advantages. Carter et al. [18] published one of the first works using neural networks for animal re-ID, a tool for green turtle (Chelonia mydas) re-ID. The authors collected 180 photos of 72 individuals from Lady Elliot Island in the southern Great Barrier Reef, both nesting and free swimming. They considered an undisclosed number of testing images. Their algorithm pre-processes the image by extracting a shell pattern, converting it to grey scale, unravelling the data into a raw input vector, and then training a simple feedforward network. Each individual model yields an output accuracy of 80-85%, but the authors utilize an ensemble approach by training 50 different networks and having each vote for a correct classification. The ensemble approach attains an accuracy of 95%. Carter et al.’s work has been considered a large success and is currently used to monitor the southern Great Barrier Reef green turtle population.

Freytag et al. [19] trained the CNN architecture AlexNet on the isolated faces of chimpanzees considering two chimpanzee data sets: C-Zoo and C-Tai. They report an improved accuracy of 92.0% and 75.7% in comparison to the original Support Vector Machine method of 84.0% and 68.8% [19, 20]. Brust et al. [21] trained the object detection method YOLO to extract cropped images of Gorilla (Gorilla gorilla) faces from 2,500 annotated camera trap images of 482 individuals taken in the Western Lowlands of the Nouabalé -Nodki National Park in the Republic of Congo. Once the faces are extracted, Brust et al. [21] followed the same procedure as Freytag et al. [19] to train AlexNet, achieving a 90.8% accuracy on a test set of 500 images. The authors herald the promise of deep learning for ecological studies show promise for a whole realm of new applications in the fields of basic identify, spatio-temporal coverage and socio-ecological insights.

3 Similarity Learning Networks for Animal Re-Identification

When approaching the problem of animal re-ID, traditional CNN architectures require a data set containing a large number of examples for every individual from the population. This is infeasible for real-world scenarios. Furthermore, they also require fixing the number of individuals in advance, so one cannot add individuals to the population without retraining the model. In order to utilize deep learning for animal re-ID, an alternative approach must be considered.

Bromley et al. [22] introduced a suitable neural network architecture for this problem, the Siamese network, which learns to detect if two input images are similar or dissimilar [22]. This approach allows new individuals to be recognized without example images of every individual in a population, and without any re-training of the network. Similarity comparison networks, such as the Siamese Network and Triplet Loss network, are forms of distant metric learning, which function by comparing the euclidean distance of a latent space embeddings, often at the last layer, considering two sister networks. This is in contrast to traditional network architectures which require all subjects to be identified and well represented in the training data and re-trained if a new individual were added to the data. The similarity comparison approach instead trains a network to learn how to identify similarities between two subjects. This allows new subjects to be recognized without example images of every individual in a population, and without any re-training of the network. Once trained, similarity comparison networks require only one labeled input image of an individual in order to accurately re-identify the second input image of the same individual. The main advantage for re-ID is that these systems generalize to individuals not found in the training data. For humans, Schroff et al. [3] demonstrated the Triplet-Loss similarity comparison framework, FaceNet, which currently holds the highest accuracy on the YouTube Faces data set with a 95.12% top-1 accuracy and is a promising model for animal re-ID.

Deb et al. [23] utilized Siamese networks for animal re-ID considering three species: chimpanzees, lemurs, and golden monkeys. They formulated the problem by defining three categories for testing successful re-ID: verification (determine if two images are the same individual), closed-set identification (identify an individual from a given set of images), and open-set identification (identify an individual from a given set of images or conclude the individual is absent from the data). For chimpanzees, they combined the C-Zoo and C-Tai data sets to create the ChimpFace data set which contains 5,599 images of 90 chimpanzees. For lemurs, they consider a data set known as LemurFace from the Duke Lemur Center, North Carolina which contains 3,000 face images of 129 lemur individuals from 12 different species. For golden monkeys, they extracted the faces of 241 short video clips (average 6 seconds) from Volcanoes National Park in Rwanda where 1,450 images of 49 golden monkey faces were cropped and extracted [23]. They use a custom Siamese CNN containing four convolutional layers, followed by a 512 node fully connected layer [23]. Deb et al. (2018) report the above defined verification, closed-set, and open-set accuracies respectively for lemurs: 83.1%, 93.8%, 81.3%, golden monkeys: 78.7%, 90.4%, 66.1%, and chimpanzees: 59.9%, 75.8%, and 37.1%.

Triplet Loss networks have been found to outperform Siamese networks for the task of digit and human face recognition [24]. Triplet Loss networks differ from Siamese networks by maximizing the distance of the embedding between two pair wise images per sample: an anchor and positive pair, and an anchor and negative pair. One advantage of Triplet Loss networks is the ability to consider optimal image pairings per mini-batch [3]. This allows for a form of curriculum learning, where easy pairwise samples are selected early in training, and difficulty increases as validation loss decreases [25]. In 2019, Bouma et al. [26] trained triplet loss networks for animal re-ID using images of dolphin fins. By following the describe triplet loss for the euclidean distance of positive and negative pairs, the learned embeddings were able to achieve 90.5% top-1 accuracy for dolphins considering 37 test individuals.

To date, no one has directly compared Siamese and Triplet Loss methodologies for animal re-ID across multiple data sets. We test their performances here.

4 Methods

To benchmark similarity networks on animal re-ID, we consider the verification accuracy metric proposed by Deb et al. [23] on five species using the following data sets, each with their own unique challenges:

• •

FaceScrub: 106,863 images of 530 male/female human individuals varying in pose [17]. This data set allows for a benchmark comparison of our methodology in comparison to other human similarity networks.

• •

ChimpFace: 5,599 images of 95 male/female chimpanzee (Pan troglodytes) individuals. This is a combination of two previous data sets: C-Tai and C-Zoo [19]. This data set provides the unique opportunity of comparing the performance of similarity networks to the previously reported performance of feature engineering as well as classical deep learning methods.

• •

HappyWhale: 9,850 images of 4,251 humpback whale (Megaptera novaeangliae) individuals offered as an expired Kaggle competition. This data set provides a realistic representation of the real-world application of animal re-ID as the 9,046 images only contain the fluke of the whale and are extremely sparse, having only an average of only 2 (+/- 8) individuals considering 4,251 individual classifications [27].

• •

FruitFly: 244,760 images of 20 fruit flies (Drosophila melanogaster) in a variety of poses [28]. Allows for the ability to test the capabilities of similarity learning networks on an animal species beyond the Chordata phylum, and in the Arthopoda phylum, where re-ID is beyond the capabilities of a human observer.

• •

Amur Tiger Re-identification in the Wild (ATRW): 1,870 images of 92 Siberian tigers (Panthera tigris altaica). A recent effort to provide a large scale tiger re-ID data set [29]. Tigers are captured in a diverse set of unconstrained poses and lighting conditions.

One trend that is prevalent throughout the history of animal re-ID is the limitation of data for individual animal re-ID, especially those publicly available. From the data sets we selected, the Chimpface, FruitFly and the ATRW data sets are limited in the numbers of individuals typical for re-ID in comparison to human data sets and benchmarks [3]. This is an unfortunate reality of working with animal re-ID. To account for this, we divide our data using a 0.7/0.1/0.2 train/validation/test for the FaceScrub, Chimpface, HappyWhale, and ATRW data sets to increase the number of individuals. For the FruitFly data set we use a 0.4/0.1/0.5 split to increase the number of testing individuals so that a mAP@5 is a meaningful metric. In addition, for each experiment we perform a five-fold train/validation/testing split, providing an inferred re-ID capability for all individuals within the data set.