Hydra: Accurate Multi-Modal Leaf Wetness Sensing with mm-Wave and Camera Fusion

TL;DR

Hydra combines mm-Wave radar and camera data with advanced neural networks to accurately detect leaf wetness, improving robustness and precision in real-world agricultural environments.

Contribution

This paper introduces Hydra, a novel multi-modal sensing system that fuses mm-Wave radar and camera data using CNNs and transformers for accurate leaf wetness detection.

Findings

Achieves up to 96% accuracy in controlled tests.

Maintains around 90% accuracy in diverse farm conditions.

Demonstrates robustness across different environmental scenarios.

Abstract

Leaf Wetness Duration (LWD), the time that water remains on leaf surfaces, is crucial in the development of plant diseases. Existing LWD detection lacks standardized measurement techniques, and variations across different plant characteristics limit its effectiveness. Prior research proposes diverse approaches, but they fail to measure real natural leaves directly and lack resilience in various environmental conditions. This reduces the precision and robustness, revealing a notable practical application and effectiveness gap in real-world agricultural settings. This paper presents Hydra, an innovative approach that integrates millimeter-wave (mm-Wave) radar with camera technology to detect leaf wetness by determining if there is water on the leaf. We can measure the time to determine the LWD based on this detection. Firstly, we design a Convolutional Neural Network (CNN) to selectively fuse multiple mm-Wave depth images with an RGB image to generate multiple feature images. Then, we develop a transformer-based encoder to capture the inherent connection among the multiple feature images to generate a feature map, which is further fed to a classifier for detection. Moreover, we augment the dataset during training to generalize our model. Implemented using a frequency-modulated continuous-wave (FMCW) radar within the 76 to 81 GHz band, Hydra's performance is meticulously evaluated on plants, demonstrating the potential to classify leaf wetness with up to 96% accuracy across varying scenarios. Deploying Hydra in the farm, including rainy, dawn, or poorly light nights, it still achieves an accuracy rate of around 90%.