A Unified Framework for Zero-Shot Reinforcement Learning

TL;DR

This paper introduces a formal, unified framework for zero-shot reinforcement learning, categorizing methods by representation and learning paradigm, and decomposing error bounds to facilitate rigorous comparisons across approaches.

Contribution

It provides a comprehensive taxonomy and a unified theoretical foundation for zero-shot RL, enabling systematic analysis and comparison of diverse methods.

Findings

Proposes a taxonomy based on representation and learning paradigm.

Introduces a unified view of error bounds decomposing into inference, reward, and approximation.

Facilitates rigorous comparison of zero-shot RL methods.

Abstract

Zero-shot reinforcement learning (RL) has emerged as a setting for developing general agents, capable of solving downstream tasks without additional training or planning at test-time. While conventional RL optimizes policies for fixed rewards, zero-shot RL requires learning representations that enable immediate adaptation to arbitrary reward functions. As the field matures, the growing diversity of approaches demands a foundational framework reconciling different perspectives under a common unifying structure. In this work, we introduce a formal, unified framework for zero-shot RL, allowing for rigorous comparisons across methods. We propose a taxonomy organizing the algorithmic landscape along two levels: representation, distinguishing between compositional and direct methods based on their exploitation of action-value function decompositions; and learning paradigm, differentiating between reward-free and pseudo reward-free training. Additionally, we propose a unified view of existing error bounds, decomposing the total error into three primary contributing components: inference, reward, and approximation, serving as a foundation for more grounded comparisons of zero-shot methods.