Multimodal embodiment-aware navigation transformer

TL;DR

ViLiNT is a multimodal, transformer-based navigation model that enhances ground robot robustness across environments by fusing visual, geometric, and embodiment data, and predicting collision-free trajectories.

Contribution

The paper introduces ViLiNT, a novel multimodal transformer architecture with embodiment-aware trajectory generation and ranking, improving zero-shot navigation robustness over state-of-the-art methods.

Findings

ViLiNT increases success rate by 166% over vision-only baselines.

It effectively fuses RGB images, LiDAR, goal embeddings, and embodiment info.

Real-world tests confirm improved obstacle avoidance in diverse environments.

Abstract

Goal-conditioned navigation models for ground robots trained using supervised learning show promising zero-shot transfer, but their collision-avoidance capability nevertheless degrades under distribution shift, i.e. environmental, robot or sensor configuration changes. We propose ViLiNT a multimodal, attention-based policy for goal navigation, trained on heterogeneous data from multiple platforms and environments, which improves robustness with two key features. First, we fuse RGB images, 3D LiDAR point clouds, a goal embedding and a robot's embodiment descriptor with a transformer architecture to capture complementary geometry and appearance cues. The transformer's output is used to condition a diffusion model that generates navigable trajectories. Second, using automatically generated offline labels, we train a path clearance prediction head for scoring and ranking trajectories produced by the diffusion model. The diffusion conditioning as well as the trajectory ranking head depend on a robot's embodiment token that allows our model to generate and select trajectories with respect to the robot's dimensions. Across three simulated environments, ViLiNT improves Success Rate on average by 166\% over equivalent state-of-the-art vision-only baseline (NoMaD). This increase in performance is confirmed through real-world deployments of a rover navigating in obstacle fields. These results highlight that combining multimodal fusion with our collision prediction mechanism leads to improved off-road navigation robustness.