Learning Bidirectional Action-Language Translation with Limited Supervision and Incongruent Input

TL;DR

This paper introduces a deep learning framework for bidirectional action-language translation that performs well with limited supervision and handles incongruent multimodal inputs, inspired by human infant learning.

Contribution

It proposes the PTAE model with Transformer-based crossmodal attention, improving translation accuracy under scarce supervision and modeling incongruence effects.

Findings

PTAE outperforms previous models in low-supervision scenarios.

Model's output degrades with conflicting multimodal input, especially with action conflicts.

PTAE remains plausible when tested with incongruent data.

Abstract

Human infant learning happens during exploration of the environment, by interaction with objects, and by listening to and repeating utterances casually, which is analogous to unsupervised learning. Only occasionally, a learning infant would receive a matching verbal description of an action it is committing, which is similar to supervised learning. Such a learning mechanism can be mimicked with deep learning. We model this weakly supervised learning paradigm using our Paired Gated Autoencoders (PGAE) model, which combines an action and a language autoencoder. After observing a performance drop when reducing the proportion of supervised training, we introduce the Paired Transformed Autoencoders (PTAE) model, using Transformer-based crossmodal attention. PTAE achieves significantly higher accuracy in language-to-action and action-to-language translations, particularly in realistic but difficult cases when only few supervised training samples are available. We also test whether the trained model behaves realistically with conflicting multimodal input. In accordance with the concept of incongruence in psychology, conflict deteriorates the model output. Conflicting action input has a more severe impact than conflicting language input, and more conflicting features lead to larger interference. PTAE can be trained on mostly unlabelled data where labeled data is scarce, and it behaves plausibly when tested with incongruent input.