Cross-Embodied Affordance Transfer through Learning Affordance Equivalences

TL;DR

This paper introduces a deep neural network model that unifies objects, actions, and effects into a shared latent space called affordance space, enabling cross-embodiment transfer and generalization of affordances across agents and objects.

Contribution

The study presents a novel shared affordance representation called Affordance Equivalence that facilitates action generalization and cross-robot embodiment transfer.

Findings

Model generates effect trajectories from actions and objects.

Model generates action trajectories from effects and objects.

Effective in simulation and real-world imitation tasks.

Abstract

Affordances represent the inherent effect and action possibilities that objects offer to the agents within a given context. From a theoretical viewpoint, affordances bridge the gap between effect and action, providing a functional understanding of the connections between the actions of an agent and its environment in terms of the effects it can cause. In this study, we propose a deep neural network model that unifies objects, actions, and effects into a single latent vector in a common latent space that we call the affordance space. Using the affordance space, our system can generate effect trajectories when action and object are given and can generate action trajectories when effect trajectories and objects are given. Our model does not learn the behavior of individual objects acted upon by a single agent. Still, rather, it forms a `shared affordance representation' spanning multiple agents and objects, which we call Affordance Equivalence. Affordance Equivalence facilitates not only action generalization over objects but also Cross Embodiment transfer linking actions of different robots. In addition to the simulation experiments that demonstrate the proposed model's range of capabilities, we also showcase that our model can be used for direct imitation in real-world settings.