WildOS: Open-Vocabulary Object Search in the Wild

TL;DR

WildOS is a novel robotic navigation system that combines semantic visual reasoning with geometric exploration to enable long-range, open-vocabulary object search in complex outdoor environments, improving robustness and efficiency.

Contribution

This work introduces WildOS, integrating foundation-model-based visual reasoning with geometric exploration for semantic navigation and distant target localization in outdoor robotics.

Findings

WildOS outperforms geometric-only baselines in efficiency and autonomy.

The system effectively localizes open-vocabulary targets beyond immediate depth.

Field experiments demonstrate robustness across diverse terrains.

Abstract

Autonomous navigation in complex, unstructured outdoor environments requires robots to operate over long ranges without prior maps and limited depth sensing. In such settings, relying solely on geometric frontiers for exploration is often insufficient. In such settings, the ability to reason semantically about where to go and what is safe to traverse is crucial for robust, efficient exploration. This work presents WildOS, a unified system for long-range, open-vocabulary object search that combines safe geometric exploration with semantic visual reasoning. WildOS builds a sparse navigation graph to maintain spatial memory, while utilizing a foundation-model-based vision module, ExploRFM, to score frontier nodes of the graph. ExploRFM simultaneously predicts traversability, visual frontiers, and object similarity in image space, enabling real-time, onboard semantic navigation tasks. The resulting vision-scored graph enables the robot to explore semantically meaningful directions while ensuring geometric safety. Furthermore, we introduce a particle-filter-based method for coarse localization of the open-vocabulary target query, that estimates candidate goal positions beyond the robot's immediate depth horizon, enabling effective planning toward distant goals. Extensive closed-loop field experiments across diverse off-road and urban terrains demonstrate that WildOS enables robust navigation, significantly outperforming purely geometric and purely vision-based baselines in both efficiency and autonomy. Our results highlight the potential of vision foundation models to drive open-world robotic behaviors that are both semantically informed and geometrically grounded. Project Page: https://leggedrobotics.github.io/wildos/