AnyTraverse: An off-road traversability framework with VLM and human operator in the loop

TL;DR

AnyTraverse is a zero-shot, human-in-the-loop framework that improves off-road traversability segmentation across diverse environments and robot types, reducing supervision while maintaining high performance.

Contribution

It introduces a novel natural language prompt-based, zero-shot traversability segmentation framework that adapts to various robots and environments with minimal supervision.

Findings

Outperforms GA-NAV and Off-seg in experiments

Works effectively across multiple datasets and real-world robots

Reduces human supervision by only calling operators for unknown scenes

Abstract

Off-road traversability segmentation enables autonomous navigation with applications in search-and-rescue, military operations, wildlife exploration, and agriculture. Current frameworks struggle due to significant variations in unstructured environments and uncertain scene changes, and are not adaptive to be used for different robot types. We present AnyTraverse, a framework combining natural language-based prompts with human-operator assistance to determine navigable regions for diverse robotic vehicles. The system segments scenes for a given set of prompts and calls the operator only when encountering previously unexplored scenery or unknown class not part of the prompt in its region-of-interest, thus reducing active supervision load while adapting to varying outdoor scenes. Our zero-shot learning approach eliminates the need for extensive data collection or retraining. Our experimental validation includes testing on RELLIS-3D, Freiburg Forest, and RUGD datasets and demonstrate real-world deployment on multiple robot platforms. The results show that AnyTraverse performs better than GA-NAV and Off-seg while offering a vehicle-agnostic approach to off-road traversability that balances automation with targeted human supervision.