SafeHumanoid: VLM-RAG-driven Control of Upper Body Impedance for Humanoid Robot

TL;DR

SafeHumanoid integrates vision-language models with retrieval-augmented generation to adapt a humanoid robot's impedance and speed in real-time, enhancing safety and task success during human-robot interactions.

Contribution

This work introduces a novel egocentric vision pipeline that links VLMs with RAG to dynamically control robot impedance based on scene context.

Findings

System adapts stiffness, damping, and speed in real-time

Maintains task success while improving safety

Demonstrates viability despite latency limitations

Abstract

Safe and trustworthy Human Robot Interaction (HRI) requires robots not only to complete tasks but also to regulate impedance and speed according to scene context and human proximity. We present SafeHumanoid, an egocentric vision pipeline that links Vision Language Models (VLMs) with Retrieval-Augmented Generation (RAG) to schedule impedance and velocity parameters for a humanoid robot. Egocentric frames are processed by a structured VLM prompt, embedded and matched against a curated database of validated scenarios, and mapped to joint-level impedance commands via inverse kinematics. We evaluate the system on tabletop manipulation tasks with and without human presence, including wiping, object handovers, and liquid pouring. The results show that the pipeline adapts stiffness, damping, and speed profiles in a context-aware manner, maintaining task success while improving safety. Although current inference latency (up to 1.4 s) limits responsiveness in highly dynamic settings, SafeHumanoid demonstrates that semantic grounding of impedance control is a viable path toward safer, standard-compliant humanoid collaboration.