Solutions for Fine-grained and Long-tailed Snake Species Recognition in SnakeCLEF 2022

TL;DR

This paper presents a comprehensive solution for fine-grained snake species recognition in a challenging long-tailed dataset, combining multimodal feature extraction, class imbalance mitigation, semi-supervised learning, and advanced post-processing to achieve top-tier performance.

Contribution

It introduces a novel multimodal network architecture and a hybrid learning approach, effectively addressing class imbalance and leveraging unlabeled data for snake species recognition.

Findings

Achieved 82.65% private score, ranking 3rd in SnakeCLEF 2022.

Demonstrated effectiveness of multimodal feature fusion and logit adjustment.

Showed benefits of combining supervised and self-supervised learning.

Abstract

Automatic snake species recognition is important because it has vast potential to help lower deaths and disabilities caused by snakebites. We introduce our solution in SnakeCLEF 2022 for fine-grained snake species recognition on a heavy long-tailed class distribution. First, a network architecture is designed to extract and fuse features from multiple modalities, i.e. photograph from visual modality and geographic locality information from language modality. Then, logit adjustment based methods are studied to relieve the impact caused by the severe class imbalance. Next, a combination of supervised and self-supervised learning method is proposed to make full use of the dataset, including both labeled training data and unlabeled testing data. Finally, post processing strategies, such as multi-scale and multi-crop test-time-augmentation, location filtering and model ensemble, are employed for better performance. With an ensemble of several different models, a private score 82.65%, ranking the 3rd, is achieved on the final leaderboard.