Assessing VLM-Driven Semantic-Affordance Inference for Non-Humanoid Robot Morphologies

TL;DR

This paper evaluates the ability of vision-language models to infer affordances for robots with non-humanoid shapes, highlighting their strengths and limitations across diverse object categories and robot forms.

Contribution

It introduces a hybrid dataset combining real and synthetic scenarios and provides an empirical analysis of VLM performance on non-humanoid robotic affordance inference.

Findings

VLMs show promising generalisation to non-humanoid robots.

Performance varies significantly across object categories.

VLMs tend to have low false positives but high false negatives, especially in novel tool use scenarios.

Abstract

Vision-language models (VLMs) have demonstrated remarkable capabilities in understanding human-object interactions, but their application to robotic systems with non-humanoid morphologies remains largely unexplored. This work investigates whether VLMs can effectively infer affordances for robots with fundamentally different embodiments than humans, addressing a critical gap in the deployment of these models for diverse robotic applications. We introduce a novel hybrid dataset that combines annotated real-world robotic affordance-object relations with VLM-generated synthetic scenarios, and perform an empirical analysis of VLM performance across multiple object categories and robot morphologies, revealing significant variations in affordance inference capabilities. Our experiments demonstrate that while VLMs show promising generalisation to non-humanoid robot forms, their performance is notably inconsistent across different object domains. Critically, we identify a consistent pattern of low false positive rates but high false negative rates across all morphologies and object categories, indicating that VLMs tend toward conservative affordance predictions. Our analysis reveals that this pattern is particularly pronounced for novel tool use scenarios and unconventional object manipulations, suggesting that effective integration of VLMs in robotic systems requires complementary approaches to mitigate over-conservative behaviour while preserving the inherent safety benefits of low false positive rates.