LOVON: Legged Open-Vocabulary Object Navigator

TL;DR

LOVON is a new framework that combines large language models with open-vocabulary visual detection to enable legged robots to perform long-range, open-world object navigation tasks effectively in dynamic environments.

Contribution

It introduces a hierarchical planning and visual detection integration, along with stabilization techniques, for robust long-range object navigation in unstructured settings.

Findings

Successful long-sequence task completion in real-world environments

Effective handling of visual jittering and target loss

Compatibility across multiple legged robot platforms

Abstract

Object navigation in open-world environments remains a formidable and pervasive challenge for robotic systems, particularly when it comes to executing long-horizon tasks that require both open-world object detection and high-level task planning. Traditional methods often struggle to integrate these components effectively, and this limits their capability to deal with complex, long-range navigation missions. In this paper, we propose LOVON, a novel framework that integrates large language models (LLMs) for hierarchical task planning with open-vocabulary visual detection models, tailored for effective long-range object navigation in dynamic, unstructured environments. To tackle real-world challenges including visual jittering, blind zones, and temporary target loss, we design dedicated solutions such as Laplacian Variance Filtering for visual stabilization. We also develop a functional execution logic for the robot that guarantees LOVON's capabilities in autonomous navigation, task adaptation, and robust task completion. Extensive evaluations demonstrate the successful completion of long-sequence tasks involving real-time detection, search, and navigation toward open-vocabulary dynamic targets. Furthermore, real-world experiments across different legged robots (Unitree Go2, B2, and H1-2) showcase the compatibility and appealing plug-and-play feature of LOVON.