Language Models are Few-Shot Butlers

TL;DR

This paper presents a method for improving language models' performance in text-based environments by fine-tuning with minimal expert demonstrations and reinforcement learning, achieving significant success rate improvements.

Contribution

It introduces a two-stage learning procedure that combines limited demonstrations with environment interaction to enhance language model capabilities.

Findings

51% success rate improvement in ALFWorld environment

Effective learning from only 1.2% of expert demonstrations

Combines fine-tuning with reinforcement learning for better performance

Abstract

Pretrained language models demonstrate strong performance in most NLP tasks when fine-tuned on small task-specific datasets. Hence, these autoregressive models constitute ideal agents to operate in text-based environments where language understanding and generative capabilities are essential. Nonetheless, collecting expert demonstrations in such environments is a time-consuming endeavour. We introduce a two-stage procedure to learn from a small set of demonstrations and further improve by interacting with an environment. We show that language models fine-tuned with only 1.2% of the expert demonstrations and a simple reinforcement learning algorithm achieve a 51% absolute improvement in success rate over existing methods in the ALFWorld environment.