Zero-Shot Segmentation through Prototype-Guidance for Multi-Label Plant Species Identification

TL;DR

This paper introduces a novel zero-shot segmentation method using prototype guidance and Vision Transformer models for multi-label plant species identification, achieving competitive results in the PlantCLEF 2025 challenge.

Contribution

It proposes a new approach combining class prototypes, K-Means clustering, and a customized ViT with frozen DinoV2 features for zero-shot multi-label plant species segmentation.

Findings

Achieved fifth place in PlantCLEF 2025 challenge with an F1 score of 0.33331.

Demonstrated domain adaptation from single-label to multi-label classification.

Method scored close to top-performing submissions, indicating competitive performance.

Abstract

This paper presents an approach developed to address the PlantClef 2025 challenge, which consists of a fine-grained multi-label species identification, over high-resolution images. Our solution focused on employing class prototypes obtained from the training dataset as a proxy guidance for training a segmentation Vision Transformer (ViT) on the test set images. To obtain these representations, the proposed method extracts features from training dataset images and create clusters, by applying K-Means, with $K$ equals to the number of classes in the dataset. The segmentation model is a customized narrow ViT, built by replacing the patch embedding layer with a frozen DinoV2, pre-trained on the training dataset for individual species classification. This model is trained to reconstruct the class prototypes of the training dataset from the test dataset images. We then use this model to obtain attention scores that enable to identify and localize areas of interest and consequently guide the classification process. The proposed approach enabled a domain-adaptation from multi-class identification with individual species, into multi-label classification from high-resolution vegetation plots. Our method achieved fifth place in the PlantCLEF 2025 challenge on the private leaderboard, with an F1 score of 0.33331. Besides that, in absolute terms our method scored 0.03 lower than the top-performing submission, suggesting that it may achieved competitive performance in the benchmark task. Our code is available at \href{https://github.com/ADAM-UEFS/PlantCLEF2025}{https://github.com/ADAM-UEFS/PlantCLEF2025}.