HumanoidVLM: Vision-Language-Guided Impedance Control for Contact-Rich Humanoid Manipulation

TL;DR

HumanoidVLM enables humanoid robots to adapt contact behavior by selecting impedance and grasp configurations from visual input using a retrieval-based framework, improving contact-rich manipulation.

Contribution

Introduces a vision-language retrieval framework that links semantic perception with control parameter selection for humanoid manipulation.

Findings

Achieved 93% retrieval accuracy in 14 scenarios.

Maintained stable interaction with low tracking errors (1-3.5 cm).

Demonstrated effective linking of perception and control in real-world tests.

Abstract

Humanoid robots must adapt their contact behavior to diverse objects and tasks, yet most controllers rely on fixed, hand-tuned impedance gains and gripper settings. This paper introduces HumanoidVLM, a vision-language driven retrieval framework that enables the Unitree G1 humanoid to select task-appropriate Cartesian impedance parameters and gripper configurations directly from an egocentric RGB image. The system couples a vision-language model for semantic task inference with a FAISS-based Retrieval-Augmented Generation (RAG) module that retrieves experimentally validated stiffness-damping pairs and object-specific grasp angles from two custom databases, and executes them through a task-space impedance controller for compliant manipulation. We evaluate HumanoidVLM on 14 visual scenarios and achieve a retrieval accuracy of 93%. Real-world experiments show stable interaction dynamics, with z-axis tracking errors typically within 1-3.5 cm and virtual forces consistent with task-dependent impedance settings. These results demonstrate the feasibility of linking semantic perception with retrieval-based control as an interpretable path toward adaptive humanoid manipulation.