Learning to Localize Reference Trajectories in Image-Space for Visual Navigation

TL;DR

LoTIS is a robot-agnostic visual navigation model that localizes reference trajectories in image space, enabling zero-shot guidance across diverse robots and environments without requiring calibration or robot-specific training.

Contribution

We introduce LoTIS, a novel approach that localizes reference trajectories in image space for robot-agnostic visual navigation, decoupling perception from action and enabling cross-trajectory training.

Findings

Outperforms state-of-the-art by 20-50% in success rate.

Achieves 94-98% success in diverse environments.

Over 5x improvement on challenging backward traversal tasks.

Abstract

We present LoTIS, a model for visual navigation that provides robot-agnostic image-space guidance by localizing a reference RGB trajectory in the robot's current view, without requiring camera calibration, poses, or robot-specific training. Instead of predicting actions tied to specific robots, we predict the image-space coordinates of the reference trajectory as they would appear in the robot's current view. This creates robot-agnostic visual guidance that easily integrates with local planning. Consequently, our model's predictions provide guidance zero-shot across diverse embodiments. By decoupling perception from action and learning to localize trajectory points rather than imitate behavioral priors, we enable a cross-trajectory training strategy for robustness to viewpoint and camera changes. We outperform state-of-the-art methods by 20-50 percentage points in success rate on conventional forward navigation, achieving 94-98% success rate across diverse sim and real environments. Furthermore, we achieve over 5x improvements on challenging tasks where baselines fail, such as backward traversal. The system is straightforward to use: we show how even a video from a phone camera directly enables different robots to navigate to any point on the trajectory. Videos, demo, and code are available at https://finnbusch.com/lotis.