Imitation and Mirror Systems in Robots through Deep Modality Blending Networks

TL;DR

This paper introduces deep modality blending networks (DMBN), a novel deep learning approach that creates a shared multi-modal latent space enabling robots to understand, imitate, and generate multi-sensory action trajectories, inspired by biological mirror systems.

Contribution

The paper presents a new deep learning architecture, DMBN, that blends multi-modal signals into a common latent space to facilitate action understanding and imitation in robots, inspired by mirror neuron systems.

Findings

DMBN can generate multi-modal trajectories conditioned on desired sensory/motor inputs.

The system enables anatomical and effect-based imitation from different perspectives.

DMBN demonstrates robust retrieval and prediction capabilities with partial multi-modal data.

Abstract

Learning to interact with the environment not only empowers the agent with manipulation capability but also generates information to facilitate building of action understanding and imitation capabilities. This seems to be a strategy adopted by biological systems, in particular primates, as evidenced by the existence of mirror neurons that seem to be involved in multi-modal action understanding. How to benefit from the interaction experience of the robots to enable understanding actions and goals of other agents is still a challenging question. In this study, we propose a novel method, deep modality blending networks (DMBN), that creates a common latent space from multi-modal experience of a robot by blending multi-modal signals with a stochastic weighting mechanism. We show for the first time that deep learning, when combined with a novel modality blending scheme, can facilitate action recognition and produce structures to sustain anatomical and effect-based imitation capabilities. Our proposed system, can be conditioned on any desired sensory/motor value at any time-step, and can generate a complete multi-modal trajectory consistent with the desired conditioning in parallel avoiding accumulation of prediction errors. We further showed that given desired images from different perspectives, i.e. images generated by the observation of other robots placed on different sides of the table, our system could generate image and joint angle sequences that correspond to either anatomical or effect based imitation behavior. Overall, the proposed DMBN architecture not only serves as a computational model for sustaining mirror neuron-like capabilities, but also stands as a powerful machine learning architecture for high-dimensional multi-modal temporal data with robust retrieval capabilities operating with partial information in one or multiple modalities.