TerraIncognita: A Dynamic Benchmark for Species Discovery Using Frontier Models

TL;DR

TerraIncognita introduces a dynamic benchmark dataset for evaluating multimodal AI models in the challenging task of discovering and classifying unknown insect species from images, emphasizing real-world open-world biodiversity scenarios.

Contribution

The paper presents TerraIncognita, a novel, evolving benchmark dataset for assessing AI models' ability to identify, classify, and explain unknown insect species in realistic ecological contexts.

Findings

Top models achieve over 90% F1 at Order level for known species.

Model accuracy drops below 2% at Species level for unknown species.

Benchmark highlights the difficulty gradient from coarse to fine taxonomic prediction.

Abstract

The rapid global loss of biodiversity, particularly among insects, represents an urgent ecological crisis. Current methods for insect species discovery are manual, slow, and severely constrained by taxonomic expertise, hindering timely conservation actions. We introduce TerraIncognita, a dynamic benchmark designed to evaluate state-of-the-art multimodal models for the challenging problem of identifying unknown, potentially undescribed insect species from image data. Our benchmark dataset combines a mix of expertly annotated images of insect species likely known to frontier AI models, and images of rare and poorly known species, for which few/no publicly available images exist. These images were collected from underexplored biodiversity hotspots, realistically mimicking open-world discovery scenarios faced by ecologists. The benchmark assesses models' proficiency in hierarchical taxonomic classification, their capability to detect and abstain from out-of-distribution (OOD) samples representing novel species, and their ability to generate explanations aligned with expert taxonomic knowledge. Notably, top-performing models achieve over 90\% F1 at the Order level on known species, but drop below 2\% at the Species level, highlighting the sharp difficulty gradient from coarse to fine taxonomic prediction (Order $\rightarrow$ Family $\rightarrow$ Genus $\rightarrow$ Species). TerraIncognita will be updated regularly, and by committing to quarterly dataset expansions (of both known and novel species), will provide an evolving platform for longitudinal benchmarking of frontier AI methods. All TerraIncognita data, results, and future updates are available \href{https://baskargroup.github.io/TerraIncognita/}{here}.