A Framework for Multisensory Foresight for Embodied Agents

TL;DR

This paper introduces an unsupervised, multi-modal predictive neural network framework for embodied agents that integrates visual, haptic, audio, and tactile data to improve future sensory state prediction.

Contribution

It presents a novel multi-modal, unsupervised neural network architecture that predicts future sensory states across multiple modalities, enhancing environmental understanding for embodied agents.

Findings

Multi-modal integration improves prediction accuracy.

The framework effectively predicts future sensory signals in real-world tasks.

Using non-visual modalities enhances visual prediction performance.

Abstract

Predicting future sensory states is crucial for learning agents such as robots, drones, and autonomous vehicles. In this paper, we couple multiple sensory modalities with exploratory actions and propose a predictive neural network architecture to address this problem. Most existing approaches rely on large, manually annotated datasets, or only use visual data as a single modality. In contrast, the unsupervised method presented here uses multi-modal perceptions for predicting future visual frames. As a result, the proposed model is more comprehensive and can better capture the spatio-temporal dynamics of the environment, leading to more accurate visual frame prediction. The other novelty of our framework is the use of sub-networks dedicated to anticipating future haptic, audio, and tactile signals. The framework was tested and validated with a dataset containing 4 sensory modalities (vision, haptic, audio, and tactile) on a humanoid robot performing 9 behaviors multiple times on a large set of objects. While the visual information is the dominant modality, utilizing the additional non-visual modalities improves the accuracy of predictions.