Application of Computer Vision to the Automated Extraction of Metadata from Natural History Specimen Labels: A Case Study on Herbarium Specimens

TL;DR

This paper presents an automated system using computer vision to extract metadata from herbarium specimen labels, improving efficiency and accuracy over traditional OCR methods.

Contribution

A novel end-to-end solution using a fine-tuned multimodal Transformer for metadata extraction from herbarium labels without preprocessing or manual labeling.

Findings

The system achieved 85% accuracy using Tree Edit Distance on a test dataset from the University of Pisa.

Multiple labels with mixed handwriting and typewritten text posed the greatest challenge for the model.

The approach offers flexibility for reuse and adaptation as newer foundational models become available.

Abstract

Extracting metadata from natural history collection labels is pivotal for the online publication of digitized specimens. Building on a pre-trained multimodal Transformer, we developed an end-to-end automated solution to extract metadata from digitally imaged herbarium specimen labels and map them to Darwin Core standard concepts. A second objective was to demonstrate the feasibility of applying state-of-the-art AI techniques to biodiversity data through a real-world use case that does not require image preprocessing or additional manual labeling for training. The proposed solution does not rely on closed-source services, is fine-tuned in-house, and can be used offline and locally. It can be flexibly reused by developers to extract metadata across different herbarium collections. Furthermore, its encoder and/or decoder component can be replaced to take advantage of newer foundational models as soon as they become available. This approach is also particularly well suited to ensuring data privacy. Finally, the publication of our baseline model could serve as a benchmark for comparison with different solutions that aim for greater generality and accuracy.

Metadata extraction from natural history collection labels is a pivotal task for the online publication of digitized specimens. However, given the scale of these collections—which are estimated to host more than 2 billion specimens worldwide, including ca. 400 million herbarium specimens—manual metadata extraction is an extremely time-consuming task. Thus, automated data extraction from digital images of specimens and their labels therefore is a promising application of state-of-the-art computer vision techniques. Extracting information from herbarium specimen labels normally involves three main steps: text segmentation, multilingual and handwriting recognition, and data parsing. The primary bottleneck in this workflow lies in the limitations of Optical Character Recognition (OCR) systems. This study explores how the general knowledge embedded in multimodal Transformer models can be transferred to the specific task of herbarium specimen label digitization. The final goal is to develop an easy-to-use, end-to-end solution to mitigate the limitations of classic OCR approaches while offering greater flexibility to adapt to different label formats. Donut-base, a pre-trained visual document understanding (VDU) transformer, was the base model selected for fine-tuning. A dataset from the University of Pisa served as a test bed. The initial attempt achieved an accuracy of 85%, measured using the Tree Edit Distance (TED), demonstrating the feasibility of fine-tuning for this task. Cases with low accuracies were also investigated to identify limitations of the approach. In particular, specimens with multiple labels, especially if combining handwritten and typewritten text, proved to be the most challenging. Strategies aimed at addressing these weaknesses are discussed.

Figures