Using Vision-Language Models as Proxies for Social Intelligence in Human-Robot Interaction

TL;DR

This paper presents a two-stage system using lightweight perceptual cues and vision-language models to improve social responsiveness in robots during human interactions, based on real-world observations.

Contribution

It introduces a novel pipeline combining perceptual detectors with VLMs to enable robots to interpret social cues and decide when to engage with humans.

Findings

Selective VLM querying improves robot social responsiveness.

The pipeline effectively interprets nonverbal cues in real-world settings.

Robots can act more appropriately by attending to natural social signals.

Abstract

Robots operating in everyday environments must often decide when and whether to engage with people, yet such decisions often hinge on subtle nonverbal cues that unfold over time and are difficult to model explicitly. Drawing on a five-day Wizard-of-Oz deployment of a mobile service robot in a university cafe, we analyze how people signal interaction readiness through nonverbal behaviors and how expert wizards use these cues to guide engagement. Motivated by these observations, we propose a two-stage pipeline in which lightweight perceptual detectors (gaze shifts and proxemics) are used to selectively trigger heavier video-based vision-language model (VLM) queries at socially meaningful moments. We evaluate this pipeline on replayed field interactions and compare two prompting strategies. Our findings suggest that selectively using VLMs as proxies for social reasoning enables socially responsive robot behavior, allowing robots to act appropriately by attending to the cues people naturally provide in real-world interactions.