Find the Fruit: Zero-Shot Sim2Real RL for Occlusion-Aware Plant Manipulation

TL;DR

This paper introduces a zero-shot sim2real reinforcement learning framework for occlusion-aware plant manipulation, enabling autonomous harvesting by repositioning plant parts to reveal fruits despite structural uncertainties.

Contribution

The approach decouples high-level planning from low-level control, improving sim2real transfer and generalization across diverse plant morphologies.

Findings

Achieves up to 86.7% success rate in real-world experiments.

Robust to occlusion variation and structural uncertainty.

Generalizes across multiple plant types.

Abstract

Autonomous harvesting in the open presents a complex manipulation problem. In most scenarios, an autonomous system has to deal with significant occlusion and require interaction in the presence of large structural uncertainties (every plant is different). Perceptual and modeling uncertainty make design of reliable manipulation controllers for harvesting challenging, resulting in poor performance during deployment. We present a sim2real reinforcement learning (RL) framework for occlusion-aware plant manipulation, where a policy is learned entirely in simulation to reposition stems and leaves to reveal target fruit(s). In our proposed approach, we decouple high-level kinematic planning from low-level compliant control which simplifies the sim2real transfer. This decomposition allows the learned policy to generalize across multiple plants with different stiffness and morphology. In experiments with multiple real-world plant setups, our system achieves up to 86.7% success in exposing target fruits, demonstrating robustness to occlusion variation and structural uncertainty.