DexFruit: Dexterous Manipulation and Gaussian Splatting Inspection of Fruit

TL;DR

DexFruit introduces a robotic framework for gentle, autonomous fruit handling using tactile sensing and a novel 3D damage quantification method, significantly reducing bruising and improving grasp success across multiple fruit types.

Contribution

The paper presents a tactile-informed manipulation policy and a new 3D Gaussian Splatting damage representation, advancing autonomous fruit harvesting and damage assessment.

Findings

92% grasping success rate

Up to 20% reduction in bruising

31% improvement in grasp success on challenging fruits

Abstract

DexFruit is a robotic manipulation framework that enables gentle, autonomous handling of fragile fruit and precise evaluation of damage. Many fruits are fragile and prone to bruising, thus requiring humans to manually harvest them with care. In this work, we demonstrate by using optical tactile sensing, autonomous manipulation of fruit with minimal damage can be achieved. We show that our tactile informed diffusion policies outperform baselines in both reduced bruising and pick-and-place success rate across three fruits: strawberries, tomatoes, and blackberries. In addition, we introduce FruitSplat, a novel technique to represent and quantify visual damage in high-resolution 3D representation via 3D Gaussian Splatting (3DGS). Existing metrics for measuring damage lack quantitative rigor or require expensive equipment. With FruitSplat, we distill a 2D strawberry mask as well as a 2D bruise segmentation mask into the 3DGS representation. Furthermore, this representation is modular and general, compatible with any relevant 2D model. Overall, we demonstrate a 92% grasping policy success rate, up to a 20% reduction in visual bruising, and up to an 31% improvement in grasp success rate on challenging fruit compared to our baselines across our three tested fruits. We rigorously evaluate this result with over 630 trials. Please checkout our website at https://dex-fruit.github.io .