Classification of freshwater snails of the genus Radomaniola with multimodal triplet networks

TL;DR

This paper introduces a machine learning system using multimodal triplet networks to classify freshwater snails of the genus Radomaniola, addressing challenges like small, imbalanced datasets and high visual similarity.

Contribution

It is the first to apply multimodal triplet networks for snail classification, combining images, measurements, and genetic data to improve accuracy.

Findings

Achieved classification performance comparable to domain experts.

Effectively handled small, imbalanced datasets with high class similarity.

Demonstrated the utility of multimodal data in biological classification.

Abstract

In this paper, we present our first proposal of a machine learning system for the classification of freshwater snails of the genus Radomaniola. We elaborate on the specific challenges encountered during system design, and how we tackled them; namely a small, very imbalanced dataset with a high number of classes and high visual similarity between classes. We then show how we employed triplet networks and the multiple input modalities of images, measurements, and genetic information to overcome these challenges and reach a performance comparable to that of a trained domain expert.