A Novel Approach to Tomato Harvesting Using a Hybrid Gripper with Semantic Segmentation and Keypoint Detection

TL;DR

This paper introduces a hybrid robotic gripper with advanced vision-based detection for efficient tomato harvesting, combining soft caging, deep learning, and trajectory planning to improve adaptability and precision in agricultural robotics.

Contribution

It presents a novel hybrid gripper design with passive auxetic structures and a vision system integrating deep learning for keypoint detection and semantic segmentation in tomato harvesting.

Findings

Effective gripping of tomatoes with varied compliance.

Accurate localization of tomatoes in occluded and variable lighting conditions.

Successful integration of vision and grasping for automated harvesting.

Abstract

Current agriculture and farming industries are able to reap advancements in robotics and automation technology to harvest fruits and vegetables using robots with adaptive grasping forces based on the compliance or softness of the fruit or vegetable. A successful operation depends on using a gripper that can adapt to the mechanical properties of the crops. This paper proposes a new robotic harvesting approach for tomato fruit using a novel hybrid gripper with a soft caging effect. It uses its six flexible passive auxetic structures based on fingers with rigid outer exoskeletons for good gripping strength and shape conformability. The gripper is actuated through a scotch-yoke mechanism using a servo motor. To perform tomato picking operations through a gripper, a vision system based on a depth camera and RGB camera implements the fruit identification process. It incorporates deep learning-based keypoint detection of the tomato's pedicel and body for localization in an occluded and variable ambient light environment and semantic segmentation of ripe and unripe tomatoes. In addition, robust trajectory planning of the robotic arm based on input from the vision system and control of robotic gripper movements are carried out for secure tomato handling. The tunable grasping force of the gripper would allow the robotic handling of fruits with a broad range of compliance.