What You See is (Usually) What You Get: Multimodal Prototype Networks that Abstain from Expensive Modalities

TL;DR

This paper introduces a cost-aware multimodal prototype network that intelligently balances the use of visual and genetic data for species detection, reducing reliance on expensive genetic information while maintaining high accuracy.

Contribution

The paper extends prototype networks to a multimodal, cost-aware setting, enabling interpretability and selective use of expensive genetic data for species identification.

Findings

Achieves comparable accuracy to models using both modalities.

Effectively identifies cases where genetic data is unnecessary.

Reduces reliance on costly genetic information without sacrificing performance.

Abstract

Species detection is important for monitoring the health of ecosystems and identifying invasive species, serving a crucial role in guiding conservation efforts. Multimodal neural networks have seen increasing use for identifying species to help automate this task, but they have two major drawbacks. First, their black-box nature prevents the interpretability of their decision making process. Second, collecting genetic data is often expensive and requires invasive procedures, often necessitating researchers to capture or kill the target specimen. We address both of these problems by extending prototype networks (ProtoPNets), which are a popular and interpretable alternative to traditional neural networks, to the multimodal, cost-aware setting. We ensemble prototypes from each modality, using an associated weight to determine how much a given prediction relies on each modality. We further introduce methods to identify cases for which we do not need the expensive genetic information to make confident predictions. We demonstrate that our approach can intelligently allocate expensive genetic data for fine-grained distinctions while using abundant image data for clearer visual classifications and achieving comparable accuracy to models that consistently use both modalities.