LISN: Language-Instructed Social Navigation with VLM-based Controller Modulating

TL;DR

This paper introduces LISN-Bench, a new simulation benchmark for language-guided social navigation, and proposes Social-Nav-Modulator, a hierarchical system that improves robot navigation by integrating language instructions with scene understanding.

Contribution

It presents the first benchmark for language-instructed social navigation and a hierarchical VLM-based controller that enhances dynamic avoidance and instruction following.

Findings

Achieved 91.3% success rate in social navigation tasks

Significantly outperformed baseline methods by over 63%

Improved performance in complex scenarios like crowd following and forbidden region avoidance

Abstract

Towards human-robot coexistence, socially aware navigation is significant for mobile robots. Yet existing studies on this area focus mainly on path efficiency and pedestrian collision avoidance, which are essential but represent only a fraction of social navigation. Beyond these basics, robots must also comply with user instructions, aligning their actions to task goals and social norms expressed by humans. In this work, we present LISN-Bench, the first simulation-based benchmark for language-instructed social navigation. Built on Rosnav-Arena 3.0, it is the first standardized social navigation benchmark to incorporate instruction following and scene understanding across diverse contexts. To address this task, we further propose Social-Nav-Modulator, a fast-slow hierarchical system where a VLM agent modulates costmaps and controller parameters. Decoupling low-level action generation from the slower VLM loop reduces reliance on high-frequency VLM inference while improving dynamic avoidance and perception adaptability. Our method achieves an average success rate of 91.3%, which is greater than 63% than the most competitive baseline, with most of the improvements observed in challenging tasks such as following a person in a crowd and navigating while strictly avoiding instruction-forbidden regions. The project website is at: https://social-nav.github.io/LISN-project/