Training a Generally Curious Agent

TL;DR

This paper introduces Paprika, a fine-tuning method that enables language models to develop general decision-making and exploration skills transferable to new tasks without further training.

Contribution

The paper presents Paprika, a novel fine-tuning approach that trains models on synthetic interaction data to enhance their ability to explore and adapt in unseen environments.

Findings

Models fine-tuned with Paprika transfer decision-making skills to new tasks.

Curriculum learning improves sample efficiency in training.

Paprika reduces reliance on gradient updates for adaptation.

Abstract

Efficient exploration is essential for intelligent systems interacting with their environment, but existing language models often fall short in scenarios that require strategic information gathering. In this paper, we present Paprika, a fine-tuning approach that enables language models to develop general decision-making capabilities that are not confined to particular environments. By training on synthetic interaction data from different tasks that require diverse strategies, Paprika teaches models to explore and adapt their behavior on a new task based on environment feedback in-context without more gradient updates. Experimental results show that models fine-tuned with Paprika can effectively transfer their learned decision-making capabilities to entirely unseen tasks without additional training. Unlike traditional training, our approach's primary bottleneck lies in sampling useful interaction data instead of model updates. To improve sample efficiency, we propose a curriculum learning strategy that prioritizes sampling trajectories from tasks with high learning potential. These results suggest a promising path towards AI systems that can autonomously solve novel sequential decision-making problems that require interactions with the external world.