Language as Cost: Proactive Hazard Mapping using VLM for Robot Navigation

TL;DR

This paper introduces a novel zero-shot language-based hazard mapping approach using vision-language models to enable robots to proactively anticipate and navigate around dynamic hazards in complex environments.

Contribution

It presents a new framework that leverages VLMs for proactive hazard assessment and integrates language as a cost for improved robot navigation.

Findings

Significantly improves navigation success rates in dynamic environments.

Reduces hazard encounters compared to reactive baseline methods.

Demonstrates effectiveness in both simulated and real-world scenarios.

Abstract

Robots operating in human-centric or hazardous environments must proactively anticipate and mitigate dangers beyond basic obstacle detection. Traditional navigation systems often depend on static maps, which struggle to account for dynamic risks, such as a person emerging from a suddenly opening door. As a result, these systems tend to be reactive rather than anticipatory when handling dynamic hazards. Recent advancements in pre-trained large language models and vision-language models (VLMs) create new opportunities for proactive hazard avoidance. In this work, we propose a zero-shot language-as-cost mapping framework that leverages VLMs to interpret visual scenes, assess potential dynamic risks, and assign risk-aware navigation costs preemptively, enabling robots to anticipate hazards before they materialize. By integrating this language-based cost map with a geometric obstacle map, the robot not only identifies existing obstacles but also anticipates and proactively plans around potential hazards arising from environmental dynamics. Experiments in simulated and diverse dynamic environments demonstrate that the proposed method significantly improves navigation success rates and reduces hazard encounters, compared to reactive baseline planners. Code and supplementary materials are available at https://github.com/Taekmino/LaC.