Pre-Trained Language Models for Interactive Decision-Making

TL;DR

This paper explores leveraging pre-trained language models to improve sequential decision-making tasks, demonstrating significant gains in task completion and generalization through policy initialization, fine-tuning, and active data gathering.

Contribution

It introduces a novel framework that uses language models to scaffold learning in decision-making, showing how LM-based representations enhance generalization and task performance.

Findings

Pre-trained LMs improve task completion rates by 43.6%.

Active data gathering boosts generalization, outperforming baselines by 25.1%.

Sequential input representations and LM initialization are key factors for success.

Abstract

Language model (LM) pre-training is useful in many language processing tasks. But can pre-trained LMs be further leveraged for more general machine learning problems? We propose an approach for using LMs to scaffold learning and generalization in general sequential decision-making problems. In this approach, goals and observations are represented as a sequence of embeddings, and a policy network initialized with a pre-trained LM predicts the next action. We demonstrate that this framework enables effective combinatorial generalization across different environments and supervisory modalities. We begin by assuming access to a set of expert demonstrations, and show that initializing policies with LMs and fine-tuning them via behavior cloning improves task completion rates by 43.6% in the VirtualHome environment. Next, we integrate an active data gathering procedure in which agents iteratively interact with the environment, relabel past "failed" experiences with new goals, and update their policies in a self-supervised loop. Active data gathering further improves combinatorial generalization, outperforming the best baseline by 25.1%. Finally, we explain these results by investigating three possible factors underlying the effectiveness of the LM-based policy. We find that sequential input representations (vs. fixed-dimensional feature vectors) and LM-based weight initialization are both important for generalization. Surprisingly, however, the format of the policy inputs encoding (e.g. as a natural language string vs. an arbitrary sequential encoding) has little influence. Together, these results suggest that language modeling induces representations that are useful for modeling not just language, but also goals and plans; these representations can aid learning and generalization even outside of language processing.