SocialNav-MoE: A Mixture-of-Experts Vision Language Model for Socially Compliant Navigation with Reinforcement Fine-Tuning

TL;DR

This paper introduces SocialNav-MoE, an efficient Mixture-of-Experts vision language model designed for socially compliant robot navigation, leveraging reinforcement fine-tuning and semantic similarity rewards to balance accuracy and computational efficiency.

Contribution

The paper proposes SocialNav-MoE, a novel mixture-of-experts model for socially aware navigation, with reinforcement fine-tuning and a new semantic similarity reward to improve decision-making.

Findings

SocialNav-MoE balances navigation accuracy and efficiency effectively.

Semantic similarity reward outperforms other reward types.

Small VLMs with specific routing strategies enhance real-time navigation.

Abstract

For robots navigating in human-populated environments, safety and social compliance are equally critical, yet prior work has mostly emphasized safety. Socially compliant navigation that accounts for human comfort, social norms, and contextual appropriateness remains underexplored. Vision language models (VLMs) show promise for this task; however, large-scale models incur substantial computational overhead, leading to higher inference latency and energy consumption, which makes them unsuitable for real-time deployment on resource-constrained robotic platforms. To address this issue, we investigate the effectiveness of small VLM and propose SocialNav-MoE, an efficient Mixture-of-Experts vision language model for socially compliant navigation with reinforcement fine-tuning (RFT). We further introduce a semantic similarity reward (SSR) to effectively leverage RFT for enhancing the decision-making capabilities. Additionally, we study the effectiveness of different small language model types (Phi, Qwen, and StableLM), routing strategies, and vision encoders (CLIP vs. SigLIP, frozen vs. fine-tuned). Experiments on the SNEI dataset demonstrate that SocialNav-MoE achieves an excellent balance between navigation accuracy and efficiency. The proposed SSR function is more effective than hard-level and character-level rewards. Source code will be released upon acceptance.