GMT: Goal-Conditioned Multimodal Transformer for 6-DOF Object Trajectory Synthesis in 3D Scenes

TL;DR

GMT is a novel multimodal transformer framework that synthesizes accurate, goal-directed 6-DOF object trajectories in 3D scenes by integrating geometric, semantic, and contextual information for robotic manipulation.

Contribution

The paper introduces GMT, a transformer-based approach that jointly leverages 3D geometry, semantics, and scene context to improve trajectory synthesis in complex 3D environments.

Findings

Outperforms state-of-the-art baselines like CHOIS and GIMO.

Achieves higher spatial accuracy and orientation control.

Demonstrates strong generalization to diverse objects and cluttered scenes.

Abstract

Synthesizing controllable 6-DOF object manipulation trajectories in 3D environments is essential for enabling robots to interact with complex scenes, yet remains challenging due to the need for accurate spatial reasoning, physical feasibility, and multimodal scene understanding. Existing approaches often rely on 2D or partial 3D representations, limiting their ability to capture full scene geometry and constraining trajectory precision. We present GMT, a multimodal transformer framework that generates realistic and goal-directed object trajectories by jointly leveraging 3D bounding box geometry, point cloud context, semantic object categories, and target end poses. The model represents trajectories as continuous 6-DOF pose sequences and employs a tailored conditioning strategy that fuses geometric, semantic, contextual, and goaloriented information. Extensive experiments on synthetic and real-world benchmarks demonstrate that GMT outperforms state-of-the-art human motion and human-object interaction baselines, such as CHOIS and GIMO, achieving substantial gains in spatial accuracy and orientation control. Our method establishes a new benchmark for learningbased manipulation planning and shows strong generalization to diverse objects and cluttered 3D environments. Project page: https://huajian- zeng.github. io/projects/gmt/.