Digital Taxonomist: Identifying Plant Species in Community Scientists' Photographs

TL;DR

This paper introduces a multimodal deep learning model called Digital Taxonomist that improves plant species identification from amateur photographs by incorporating contextual and hierarchical information, outperforming image-only classifiers.

Contribution

The paper presents a novel multimodal deep learning framework that integrates visual, spatial, temporal, ecological, and taxonomic data for plant species identification.

Findings

Achieves over 6% accuracy improvement over image-only models

Effectively handles class imbalance and visual similarity among species

Utilizes hierarchical taxonomic structure to enhance classification

Abstract

Automatic identification of plant specimens from amateur photographs could improve species range maps, thus supporting ecosystems research as well as conservation efforts. However, classifying plant specimens based on image data alone is challenging: some species exhibit large variations in visual appearance, while at the same time different species are often visually similar; additionally, species observations follow a highly imbalanced, long-tailed distribution due to differences in abundance as well as observer biases. On the other hand, most species observations are accompanied by side information about the spatial, temporal and ecological context. Moreover, biological species are not an unordered list of classes but embedded in a hierarchical taxonomic structure. We propose a multimodal deep learning model that takes into account these additional cues in a unified framework. Our Digital Taxonomist is able to identify plant species in photographs better than a classifier trained on the image content alone, the performance gained is over 6 percent points in terms of accuracy.