CLaD: Planning with Grounded Foresight via Cross-Modal Latent Dynamics

TL;DR

CLaD introduces a novel framework for robotic manipulation that models coupled kinematic and semantic state transitions using cross-attention and grounded latent foresights, enabling effective planning with fewer parameters.

Contribution

It proposes a cross-modal latent dynamics model with asymmetric attention and self-supervised training to improve grounded foresight in robotic planning.

Findings

Achieves 94.7% success rate on LIBERO-LONG benchmark.

Uses fewer parameters than large VLAs with comparable performance.

Effectively models coupled kinematic and semantic transitions.

Abstract

Robotic manipulation involves kinematic and semantic transitions that are inherently coupled via underlying actions. However, existing approaches plan within either semantic or latent space without explicitly aligning these cross-modal transitions. To address this, we propose CLaD, a framework that models how proprioceptive and semantic states jointly evolve under actions through asymmetric cross-attention that allows kinematic transitions to query semantic ones. CLaD predicts grounded latent foresights via self-supervised objectives with EMA target encoders and auxiliary reconstruction losses, preventing representation collapse while anchoring predictions to observable states. Predicted foresights are modulated with observations to condition a diffusion policy for action generation. On LIBERO-LONG benchmark, CLaD achieves 94.7\% success rate, competitive with large VLAs with significantly fewer parameters.