T-REX: Vision-Based System for Autonomous Leaf Detection and Grasp Estimation

TL;DR

T-Rex is an autonomous robotic system that uses stereo vision and deep learning to detect, localize, and grasp leaves in greenhouse environments, aiming to automate plant sampling tasks.

Contribution

The paper introduces a novel integrated system combining stereo vision, deep learning, and grasp planning for autonomous leaf sampling in controlled environments.

Findings

Achieved a 66.6% grasp success rate on artificial plants.

Demonstrated real-time leaf detection and 3D reconstruction capabilities.

Validated system performance across varied plant poses.

Abstract

T-Rex (The Robot for Extracting Leaf Samples) is a gantry-based robotic system developed for autonomous leaf localization, selection, and grasping in greenhouse environments. The system integrates a 6-degree-of-freedom manipulator with a stereo vision pipeline to identify and interact with target leaves. YOLOv8 is used for real-time leaf segmentation, and RAFT-Stereo provides dense depth maps, allowing the reconstruction of 3D leaf masks. These observations are processed through a leaf grasping algorithm that selects the optimal leaf based on clutter, visibility, and distance, and determines a grasp point by analyzing local surface flatness, top-down approachability, and margin from edges. The selected grasp point guides a trajectory executed by ROS-based motion controllers, driving a custom microneedle-equipped end-effector to clamp the leaf and simulate tissue sampling. Experiments conducted with artificial plants under varied poses demonstrate that the T-Rex system can consistently detect, plan, and perform physical interactions with plant-like targets, achieving a grasp success rate of 66.6\%. This paper presents the system architecture, implementation, and testing of T-Rex as a step toward plant sampling automation in Controlled Environment Agriculture (CEA).