PIRATR: Parametric Object Inference for Robotic Applications with Transformers in 3D Point Clouds

TL;DR

PIRATR is an innovative 3D object detection framework that estimates multi-class 6-DoF poses and parametric attributes directly from point clouds, enabling robots to understand and interact with complex objects in real-world environments.

Contribution

It introduces a modular, end-to-end parametric detection method extending PI3DETR, capable of estimating object poses and properties from occlusion-affected point clouds without re-designing for new object types.

Findings

Achieves 0.919 mAP on real outdoor LiDAR scans without fine-tuning.

Effectively generalizes from synthetic training to real-world data.

Supports task-specific parametric property estimation, such as gripper opening.

Abstract

We present PIRATR, an end-to-end 3D object detection framework for robotic use cases in point clouds. Extending PI3DETR, our method streamlines parametric 3D object detection by jointly estimating multi-class 6-DoF poses and class-specific parametric attributes directly from occlusion-affected point cloud data. This formulation enables not only geometric localization but also the estimation of task-relevant properties for parametric objects, such as a gripper's opening, where the 3D model is adjusted according to simple, predefined rules. The architecture employs modular, class-specific heads, making it straightforward to extend to novel object types without re-designing the pipeline. We validate PIRATR on an automated forklift platform, focusing on three structurally and functionally diverse categories: crane grippers, loading platforms, and pallets. Trained entirely in a synthetic environment, PIRATR generalizes effectively to real outdoor LiDAR scans, achieving a detection mAP of 0.919 without additional fine-tuning. PIRATR establishes a new paradigm of pose-aware, parameterized perception. This bridges the gap between low-level geometric reasoning and actionable world models, paving the way for scalable, simulation-trained perception systems that can be deployed in dynamic robotic environments. Code available at https://github.com/swingaxe/piratr.