PestVL-Net: Enabling Multimodal Pest Learning via Fine-grained Vision-Language Interaction

TL;DR

PestVL-Net is a novel multimodal framework combining vision and language models to improve fine-grained pest recognition, leveraging expert knowledge and advanced architectures for practical agricultural pest management.

Contribution

The paper introduces PestVL-Net, a new vision-language model with a specialized architecture and multimodal reasoning for enhanced pest identification.

Findings

PestVL-Net outperforms existing methods on pest datasets.

The model effectively integrates visual features with semantic descriptions.

Experimental results demonstrate its potential for real-world pest management.

Abstract

Effective pest recognition and management are crucial for sustainable agricultural development. However, collecting pest data in real scenarios is often challenging. Compared to other domains, pests exhibit a wide variety of species with complex and diverse morphological characteristics. Existing techniques struggle to effectively model the key visual and high-level semantic features of pests in a fine-grained manner. These limitations hinder the practical application of such methods in real agricultural scenarios. To address these critical challenges, we present a synergistic approach that integrates PestVL-Net, a novel vision-language framework, with two multi-species pest datasets to facilitate fine-grained pest learning. The visual pathway of PestVL-Net utilizes the Recurrent Weighted Key Value (RWKV) architecture, incorporating a saliency-guided adaptive window partitioning scheme to effectively model the fine-grained visual characteristics of pests. Concurrently, the linguistic component generates precise pest semantic descriptions by leveraging Multimodal Large Language Models (MLLMs) priors, critically informed by agricultural expert knowledge and structured via multimodal Chain-of-Thought (CoT) reasoning. The deep fusion of these complementary visual and textual representations enables fine-grained multimodal pest learning. Extensive experimental evaluations on multiple pest datasets validate the superior performance of PestVL-Net, highlighting its potential for effective real-world pest management.