Botany-Bot: Digital Twin Monitoring of Occluded and Underleaf Plant Structures with Gaussian Splats

TL;DR

Botany-Bot introduces a robotic system that creates detailed digital twins of plants by overcoming occlusion issues with stereo cameras, robotic manipulation, and Gaussian Splat models, enabling precise plant phenotyping.

Contribution

The paper presents a novel robotic system combining stereo vision, manipulation, and Gaussian Splat modeling for detailed plant digitization and phenotyping.

Findings

90.8% leaf segmentation accuracy

86.2% leaf detection accuracy

77.9% leaf manipulation accuracy

Abstract

Commercial plant phenotyping systems using fixed cameras cannot perceive many plant details due to leaf occlusion. In this paper, we present Botany-Bot, a system for building detailed "annotated digital twins" of living plants using two stereo cameras, a digital turntable inside a lightbox, an industrial robot arm, and 3D segmentated Gaussian Splat models. We also present robot algorithms for manipulating leaves to take high-resolution indexable images of occluded details such as stem buds and the underside/topside of leaves. Results from experiments suggest that Botany-Bot can segment leaves with 90.8% accuracy, detect leaves with 86.2% accuracy, lift/push leaves with 77.9% accuracy, and take detailed overside/underside images with 77.3% accuracy. Code, videos, and datasets are available at https://berkeleyautomation.github.io/Botany-Bot/.