Self-Supervised Learning for Robotic Leaf Manipulation: A Hybrid Geometric-Neural Approach

TL;DR

This paper introduces a hybrid geometric-neural self-supervised approach for robotic leaf manipulation, combining traditional vision and neural networks to improve grasping success in agricultural robotics.

Contribution

It presents a novel fusion mechanism and self-supervised training framework that integrates geometric and neural methods for autonomous leaf grasping.

Findings

Achieved 88.0% success rate in controlled environments

Achieved 84.7% success rate in greenhouse conditions

Outperformed purely geometric and neural methods

Abstract

Automating leaf manipulation in agricultural settings faces significant challenges, including the variability of plant morphologies and deformable leaves. We propose a novel hybrid geometric-neural approach for autonomous leaf grasping that combines traditional computer vision with neural networks through self-supervised learning. Our method integrates YOLOv8 for instance segmentation and RAFT-Stereo for 3D depth estimation to build rich leaf representations, which feed into both a geometric feature scoring pipeline and a neural refinement module (GraspPointCNN). The key innovation is our confidence-weighted fusion mechanism that dynamically balances the contribution of each approach based on prediction certainty. Our self-supervised framework uses the geometric pipeline as an expert teacher to automatically generate training data. Experiments demonstrate that our approach achieves an 88.0% success rate in controlled environments and 84.7% in real greenhouse conditions, significantly outperforming both purely geometric (75.3%) and neural (60.2%) methods. This work establishes a new paradigm for agricultural robotics where domain expertise is seamlessly integrated with machine learning capabilities, providing a foundation for fully automated crop monitoring systems.