NeFF-BioNet: Crop Biomass Prediction from Point Cloud to Drone Imagery

TL;DR

This paper introduces NeFF-BioNet, a versatile deep learning framework that predicts crop biomass from point clouds and drone imagery, improving accuracy and scalability for agricultural monitoring.

Contribution

The novel BioNet architecture combines sparse 3D CNNs and transformer modules, extended with NeFF for 3D reconstruction from drone images, enabling accurate biomass prediction across data modalities.

Findings

BioNet outperforms state-of-the-art by 6.1% RI on point clouds.

NeFF enhances drone imagery biomass prediction with 7.9% RI.

The approach is scalable using inexpensive drone-mounted cameras.

Abstract

Crop biomass offers crucial insights into plant health and yield, making it essential for crop science, farming systems, and agricultural research. However, current measurement methods, which are labor-intensive, destructive, and imprecise, hinder large-scale quantification of this trait. To address this limitation, we present a biomass prediction network (BioNet), designed for adaptation across different data modalities, including point clouds and drone imagery. Our BioNet, utilizing a sparse 3D convolutional neural network (CNN) and a transformer-based prediction module, processes point clouds and other 3D data representations to predict biomass. To further extend BioNet for drone imagery, we integrate a neural feature field (NeFF) module, enabling 3D structure reconstruction and the transformation of 2D semantic features from vision foundation models into the corresponding 3D surfaces. For the point cloud modality, BioNet demonstrates superior performance on two public datasets, with an approximate 6.1% relative improvement (RI) over the state-of-the-art. In the RGB image modality, the combination of BioNet and NeFF achieves a 7.9% RI. Additionally, the NeFF-based approach utilizes inexpensive, portable drone-mounted cameras, providing a scalable solution for large field applications.