Pre-Learning Environment Representations for Data-Efficient Neural Instruction Following

TL;DR

This paper introduces a method that pre-trains environment representations from language-free observations to improve data efficiency in neural instruction following tasks.

Contribution

It proposes a novel pre-learning phase that induces environment representations before instruction training, enhancing performance with limited data.

Findings

Pre-trained environment representations improve instruction-following accuracy.

The approach reduces data requirements for effective learning.

Performance gains are significant compared to baseline methods.

Abstract

We consider the problem of learning to map from natural language instructions to state transitions (actions) in a data-efficient manner. Our method takes inspiration from the idea that it should be easier to ground language to concepts that have already been formed through pre-linguistic observation. We augment a baseline instruction-following learner with an initial environment-learning phase that uses observations of language-free state transitions to induce a suitable latent representation of actions before processing the instruction-following training data. We show that mapping to pre-learned representations substantially improves performance over systems whose representations are learned from limited instructional data alone.