Enhancing Agricultural Environment Perception via Active Vision and Zero-Shot Learning

TL;DR

This paper presents a novel system combining Active Vision and Zero-Shot Learning to enhance agricultural environment perception, enabling robots to efficiently perceive and interact in complex, unknown farming scenarios.

Contribution

It introduces an integrated AV pipeline with NBV planning and ZSL-based semantic segmentation for improved perception in agricultural robotics.

Findings

Outperforms traditional static planning methods in complex visibility conditions.

Demonstrates high-speed, accurate segmentation without fine-tuning in real-world tests.

Effective in both simulation and real-world fruit harvesting scenarios.

Abstract

Agriculture, fundamental for human sustenance, faces unprecedented challenges. The need for efficient, human-cooperative, and sustainable farming methods has never been greater. The core contributions of this work involve leveraging Active Vision (AV) techniques and Zero-Shot Learning (ZSL) to improve the robot's ability to perceive and interact with agricultural environment in the context of fruit harvesting. The AV Pipeline implemented within ROS 2 integrates the Next-Best View (NBV) Planning for 3D environment reconstruction through a dynamic 3D Occupancy Map. Our system allows the robotics arm to dynamically plan and move to the most informative viewpoints and explore the environment, updating the 3D reconstruction using semantic information produced through ZSL models. Simulation and real-world experimental results demonstrate our system's effectiveness in complex visibility conditions, outperforming traditional and static predefined planning methods. ZSL segmentation models employed, such as YOLO World + EfficientViT SAM, exhibit high-speed performance and accurate segmentation, allowing flexibility when dealing with semantic information in unknown agricultural contexts without requiring any fine-tuning process.