Snap and Diagnose: An Advanced Multimodal Retrieval System for Identifying Plant Diseases in the Wild

TL;DR

This paper introduces a multimodal retrieval system for plant disease identification that leverages a large in-the-wild dataset and a novel CLIP-based model to enable image and text-based disease search, aiding farmers in early diagnosis.

Contribution

The paper presents a new multimodal retrieval system utilizing a CLIP-based model and the extensive PlantWild dataset for accurate cross-modal plant disease identification.

Findings

Achieved effective cross-modal retrieval with high accuracy.

Supported both image and text queries for disease diagnosis.

Demonstrated system's utility in real-world agricultural scenarios.

Abstract

Plant disease recognition is a critical task that ensures crop health and mitigates the damage caused by diseases. A handy tool that enables farmers to receive a diagnosis based on query pictures or the text description of suspicious plants is in high demand for initiating treatment before potential diseases spread further. In this paper, we develop a multimodal plant disease image retrieval system to support disease search based on either image or text prompts. Specifically, we utilize the largest in-the-wild plant disease dataset PlantWild, which includes over 18,000 images across 89 categories, to provide a comprehensive view of potential diseases relating to the query. Furthermore, cross-modal retrieval is achieved in the developed system, facilitated by a novel CLIP-based vision-language model that encodes both disease descriptions and disease images into the same latent space. Built on top of the retriever, our retrieval system allows users to upload either plant disease images or disease descriptions to retrieve the corresponding images with similar characteristics from the disease dataset to suggest candidate diseases for end users' consideration.