Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces

TL;DR

This paper introduces Social-LLaVA, a vision-language model trained on a new dataset, enabling robots to reason socially in dynamic environments using human-like language-based understanding.

Contribution

The paper presents a novel dataset and a fine-tuned vision-language model that improves socially aware robot navigation through language reasoning.

Findings

Social-LLaVA outperforms GPT-4V and Gemini in VQA tasks.

The dataset contains 40K annotated interactions in crowded spaces.

Deployment demonstrates human-like reasoning in real robot navigation.

Abstract

Most existing social robot navigation techniques either leverage hand-crafted rules or human demonstrations to connect robot perception to socially compliant actions. However, there remains a significant gap in effectively translating perception into socially compliant actions, much like how human reasoning naturally occurs in dynamic environments. Considering the recent success of Vision-Language Models (VLMs), we propose using language to bridge the gap in human-like reasoning between perception and socially aware robot actions. We create a vision-language dataset, Social robot Navigation via Explainable Interactions (SNEI), featuring 40K human-annotated Visual Question Answers (VQAs) based on 2K human-robot social interactions in unstructured, crowded public spaces, spanning perception, prediction, chain-of-thought reasoning, action, and explanation. We fine-tune a VLM, Social-LLaVA, using SNEI to demonstrate the practical application of our dataset. Social-LLaVA outperforms state-of-the-art models like GPT-4V and Gemini, based on the average of fifteen different human-judge scores across 50 VQA. Deployed onboard a mobile robot, Social-LLaVA enables human-like reasoning, marking a promising step toward socially compliant robot navigation in dynamic public spaces through language reasoning.