Multi-vision-based Picking Point Localisation of Target Fruit for Harvesting Robots

TL;DR

This paper develops and compares multi-vision-based methods for accurately localizing picking points on fruits to enhance robotic harvesting efficiency, demonstrating significant improvements over single-camera systems.

Contribution

It introduces two multi-vision-based localization strategies, analytical and model-based, and evaluates their effectiveness in improving harvesting success rates.

Findings

Adaboost regression achieved 88.8% accuracy with 4.40 mm MED.

Analytical approach reached 81.4% success with 14.25 mm MED.

Multi-vision systems outperform single-camera setups in fruit picking success.

Abstract

This paper presents multi-vision-based localisation strategies for harvesting robots. Identifying picking points accurately is essential for robotic harvesting because insecure grasping can lead to economic loss through fruit damage and dropping. In this study, two multi-vision-based localisation methods, namely the analytical approach and model-based algorithms, were employed. The actual geometric centre points of fruits were collected using a motion capture system (mocap), and two different surface points Cfix and Ceih were extracted using two Red-Green-Blue-Depth (RGB-D) cameras. First, the picking points of the target fruit were detected using analytical methods. Second, various primary and ensemble learning methods were employed to predict the geometric centre of target fruits by taking surface points as input. Adaboost regression, the most successful model-based localisation algorithm, achieved 88.8% harvesting accuracy with a Mean Euclidean Distance (MED) of 4.40 mm, while the analytical approach reached 81.4% picking success with a MED of 14.25 mm, both demonstrating better performance than the single-camera, which had a picking success rate of 77.7% with a MED of 24.02 mm. To evaluate the effect of picking point accuracy in collecting fruits, a series of robotic harvesting experiments were performed utilising a collaborative robot (cobot). It is shown that multi-vision systems can improve picking point localisation, resulting in higher success rates of picking in robotic harvesting.