BioVITA: Biological Dataset, Model, and Benchmark for Visual-Textual-Acoustic Alignment

TL;DR

BioVITA introduces a comprehensive multimodal framework combining a large-scale biological dataset, a novel alignment model, and a cross-modal retrieval benchmark to improve species identification through visual, textual, and acoustic data.

Contribution

It presents a new dataset, a two-stage training model for multimodal alignment, and a comprehensive retrieval benchmark for biological species recognition.

Findings

Model effectively aligns audio, visual, and textual data.

Achieves species-level semantic understanding beyond taxonomy.

Provides a new benchmark for multimodal biodiversity research.

Abstract

Understanding animal species from multimodal data poses an emerging challenge at the intersection of computer vision and ecology. While recent biological models, such as BioCLIP, have demonstrated strong alignment between images and textual taxonomic information for species identification, the integration of the audio modality remains an open problem. We propose BioVITA, a novel visual-textual-acoustic alignment framework for biological applications. BioVITA involves (i) a training dataset, (ii) a representation model, and (iii) a retrieval benchmark. First, we construct a large-scale training dataset comprising 1.3 million audio clips and 2.3 million images, covering 14,133 species annotated with 34 ecological trait labels. Second, building upon BioCLIP2, we introduce a two-stage training framework to effectively align audio representations with visual and textual representations. Third, we develop a cross-modal retrieval benchmark that covers all possible directional retrieval across the three modalities (i.e., image-to-audio, audio-to-text, text-to-image, and their reverse directions), with three taxonomic levels: Family, Genus, and Species. Extensive experiments demonstrate that our model learns a unified representation space that captures species-level semantics beyond taxonomy, advancing multimodal biodiversity understanding. The project page is available at: https://dahlian00.github.io/BioVITA_Page/