CrypticBio: A Large Multimodal Dataset for Visually Confusing Biodiversity

TL;DR

CrypticBio is the largest multimodal dataset of visually confusing species, designed to advance AI models in biodiversity research by including rich annotations and multimodal cues like geographical and temporal data.

Contribution

It introduces CrypticBio, a comprehensive dataset with multimodal annotations for cryptic species, and provides a benchmark demonstrating the importance of contextual cues in species identification.

Findings

Geographical context significantly improves zero-shot species identification.

State-of-the-art models show substantial performance gains with multimodal data.

CrypticBio enables research on visual and contextual cues for cryptic species recognition.

Abstract

We present CrypticBio, the largest publicly available multimodal dataset of visually confusing species, specifically curated to support the development of AI models in the context of biodiversity applications. Visually confusing or cryptic species are groups of two or more taxa that are nearly indistinguishable based on visual characteristics alone. While much existing work addresses taxonomic identification in a broad sense, datasets that directly address the morphological confusion of cryptic species are small, manually curated, and target only a single taxon. Thus, the challenge of identifying such subtle differences in a wide range of taxa remains unaddressed. Curated from real-world trends in species misidentification among community annotators of iNaturalist, CrypticBio contains 52K unique cryptic groups spanning 67K species, represented in 166 million images. Rich research-grade image annotations--including scientific, multicultural, and multilingual species terminology, hierarchical taxonomy, spatiotemporal context, and associated cryptic groups--address multimodal AI in biodiversity research. For easy dataset curation, we provide an open-source pipeline CrypticBio-Curate. The multimodal nature of the dataset beyond vision-language arises from the integration of geographical and temporal data as complementary cues to identifying cryptic species. To highlight the importance of the dataset, we benchmark a suite of state-of-the-art foundation models across CrypticBio subsets of common, unseen, endangered, and invasive species, and demonstrate the substantial impact of geographical context on vision-language zero-shot learning for cryptic species. By introducing CrypticBio, we aim to catalyze progress toward real-world-ready biodiversity AI models capable of handling the nuanced challenges of species ambiguity.