Vision-guided Autonomous Dual-arm Extraction Robot for Bell Pepper Harvesting

TL;DR

This paper introduces VADER, a vision-guided dual-arm robot system for autonomous outdoor bell pepper harvesting, achieving over 60% success rate and under 100 seconds per fruit, with a novel perception dataset.

Contribution

The paper presents a novel dual-arm robotic system with integrated perception and planning for outdoor harvesting, along with a new large-scale dataset for fruit detection and pose estimation.

Findings

Harvest success rate exceeds 60%

Cycle time under 100 seconds per fruit

Robust perception enabled by a new dataset

Abstract

Agricultural robotics has emerged as a critical solution to the labor shortages and rising costs associated with manual crop harvesting. Bell pepper harvesting, in particular, is a labor-intensive task, accounting for up to 50% of total production costs. While automated solutions have shown promise in controlled greenhouse environments, harvesting in unstructured outdoor farms remains an open challenge due to environmental variability and occlusion. This paper presents VADER (Vision-guided Autonomous Dual-arm Extraction Robot), a dual-arm mobile manipulation system designed specifically for the autonomous harvesting of bell peppers in outdoor environments. The system integrates a robust perception pipeline coupled with a dual-arm planning framework that coordinates a gripping arm and a cutting arm for extraction. We validate the system through trials in various realistic conditions, demonstrating a harvest success rate exceeding 60% with a cycle time of under 100 seconds per fruit, while also featuring a teleoperation fail-safe based on the GELLO teleoperation framework to ensure robustness. To support robust perception, we contribute a hierarchically structured dataset of over 3,200 images spanning indoor and outdoor domains, pairing wide-field scene images with close-up pepper images to enable a coarse-to-fine training strategy from fruit detection to high-precision pose estimation. The code and dataset will be made publicly available upon acceptance.