TaxaBind: A Unified Embedding Space for Ecological Applications

TL;DR

TaxaBind introduces a comprehensive multimodal embedding space for ecological applications, integrating diverse data types to improve species characterization and ecological problem-solving.

Contribution

It proposes a novel multimodal embedding framework, new datasets for ecological data, and demonstrates strong zero-shot capabilities for ecological tasks.

Findings

Effective zero-shot species classification

Strong cross-modal retrieval performance

Emergent capabilities in ecological tasks

Abstract

We present TaxaBind, a unified embedding space for characterizing any species of interest. TaxaBind is a multimodal embedding space across six modalities: ground-level images of species, geographic location, satellite image, text, audio, and environmental features, useful for solving ecological problems. To learn this joint embedding space, we leverage ground-level images of species as a binding modality. We propose multimodal patching, a technique for effectively distilling the knowledge from various modalities into the binding modality. We construct two large datasets for pretraining: iSatNat with species images and satellite images, and iSoundNat with species images and audio. Additionally, we introduce TaxaBench-8k, a diverse multimodal dataset with six paired modalities for evaluating deep learning models on ecological tasks. Experiments with TaxaBind demonstrate its strong zero-shot and emergent capabilities on a range of tasks including species classification, cross-model retrieval, and audio classification. The datasets and models are made available at https://github.com/mvrl/TaxaBind.