Category-Level 6D Object Pose Estimation in Agricultural Settings Using a Lattice-Deformation Framework and Diffusion-Augmented Synthetic Data

TL;DR

This paper presents PLANTPose, a category-level 6D object pose estimation framework for agriculture that uses a lattice-deformation model and diffusion-augmented synthetic data to handle high intra-class variability with RGB input.

Contribution

Introduces PLANTPose, a novel RGB-only framework for category-level 6D pose estimation that predicts deformation parameters to adapt a base mesh to unseen instances.

Findings

Outperforms state-of-the-art RGB-based methods like MegaPose.

Effectively handles large intra-class variations in shape and ripeness.

Uses diffusion-enhanced synthetic data to improve real-world generalization.

Abstract

Accurate 6D object pose estimation is essential for robotic grasping and manipulation, particularly in agriculture, where fruits and vegetables exhibit high intra-class variability in shape, size, and texture. The vast majority of existing methods rely on instance-specific CAD models or require depth sensors to resolve geometric ambiguities, making them impractical for real-world agricultural applications. In this work, we introduce PLANTPose, a novel framework for category-level 6D pose estimation that operates purely on RGB input. PLANTPose predicts both the 6D pose and deformation parameters relative to a base mesh, allowing a single category-level CAD model to adapt to unseen instances. This enables accurate pose estimation across varying shapes without relying on instance-specific data. To enhance realism and improve generalization, we also leverage Stable Diffusion to refine synthetic training images with realistic texturing, mimicking variations due to ripeness and environmental factors and bridging the domain gap between synthetic data and the real world. Our evaluations on a challenging benchmark that includes bananas of various shapes, sizes, and ripeness status demonstrate the effectiveness of our framework in handling large intraclass variations while maintaining accurate 6D pose predictions, significantly outperforming the state-of-the-art RGB-based approach MegaPose.