Benchmarking In-the-wild Multimodal Disease Recognition and A Versatile Baseline

TL;DR

This paper introduces a large-scale in-the-wild multimodal plant disease dataset with text descriptions and proposes a versatile baseline model that effectively handles small inter-class differences and large intra-class variance, advancing real-world disease recognition.

Contribution

The paper provides the largest in-the-wild multimodal plant disease dataset with textual descriptions and develops a prototype-based baseline model for improved disease classification.

Findings

The dataset presents new challenges for plant disease recognition in real-world scenarios.

The baseline model effectively integrates multimodal data to address intra- and inter-class variability.

The model performs well in few-shot and training-free recognition scenarios.

Abstract

Existing plant disease classification models have achieved remarkable performance in recognizing in-laboratory diseased images. However, their performance often significantly degrades in classifying in-the-wild images. Furthermore, we observed that in-the-wild plant images may exhibit similar appearances across various diseases (i.e., small inter-class discrepancy) while the same diseases may look quite different (i.e., large intra-class variance). Motivated by this observation, we propose an in-the-wild multimodal plant disease recognition dataset that contains the largest number of disease classes but also text-based descriptions for each disease. Particularly, the newly provided text descriptions are introduced to provide rich information in textual modality and facilitate in-the-wild disease classification with small inter-class discrepancy and large intra-class variance issues. Therefore, our proposed dataset can be regarded as an ideal testbed for evaluating disease recognition methods in the real world. In addition, we further present a strong yet versatile baseline that models text descriptions and visual data through multiple prototypes for a given class. By fusing the contributions of multimodal prototypes in classification, our baseline can effectively address the small inter-class discrepancy and large intra-class variance issues. Remarkably, our baseline model can not only classify diseases but also recognize diseases in few-shot or training-free scenarios. Extensive benchmarking results demonstrate that our proposed in-the-wild multimodal dataset sets many new challenges to the plant disease recognition task and there is a large space to improve for future works.