Language Model-Based Paired Variational Autoencoders for Robotic Language Learning

TL;DR

This paper introduces a neural model that links robot actions with language descriptions, leveraging variational autoencoders and pretrained language models to improve language understanding and generalization in robotic learning.

Contribution

The work extends previous PVAE models by integrating BERT, enabling robots to understand and generate natural language descriptions beyond predefined vocabularies.

Findings

VAE outperforms standard autoencoders in binding actions and descriptions.

Channel-separated visual features handle objects of different shapes.

BERT integration allows understanding of unconstrained natural language.

Abstract

Human infants learn language while interacting with their environment in which their caregivers may describe the objects and actions they perform. Similar to human infants, artificial agents can learn language while interacting with their environment. In this work, first, we present a neural model that bidirectionally binds robot actions and their language descriptions in a simple object manipulation scenario. Building on our previous Paired Variational Autoencoders (PVAE) model, we demonstrate the superiority of the variational autoencoder over standard autoencoders by experimenting with cubes of different colours, and by enabling the production of alternative vocabularies. Additional experiments show that the model's channel-separated visual feature extraction module can cope with objects of different shapes. Next, we introduce PVAE-BERT, which equips the model with a pretrained large-scale language model, i.e., Bidirectional Encoder Representations from Transformers (BERT), enabling the model to go beyond comprehending only the predefined descriptions that the network has been trained on; the recognition of action descriptions generalises to unconstrained natural language as the model becomes capable of understanding unlimited variations of the same descriptions. Our experiments suggest that using a pretrained language model as the language encoder allows our approach to scale up for real-world scenarios with instructions from human users.