Zero-Shot Off-Policy Learning

TL;DR

This paper introduces a novel zero-shot off-policy learning method that leverages successor measures and stationary density ratios to adapt to new tasks without additional training, demonstrated across various benchmarks.

Contribution

It establishes a theoretical link between successor measures and density ratios, enabling real-time policy adaptation in zero-shot off-policy settings.

Findings

Effective in motion tracking and continuous control tasks.

Seamless integration with forward-backward representation frameworks.

Enables fast adaptation to new tasks without extra training.

Abstract

Off-policy learning methods seek to derive an optimal policy directly from a fixed dataset of prior interactions. This objective presents significant challenges, primarily due to the inherent distributional shift and value function overestimation bias. These issues become even more noticeable in zero-shot reinforcement learning, where an agent trained on reward-free data must adapt to new tasks at test time without additional training. In this work, we address the off-policy problem in a zero-shot setting by discovering a theoretical connection of successor measures to stationary density ratios. Using this insight, our algorithm can infer optimal importance sampling ratios, effectively performing a stationary distribution correction with an optimal policy for any task on the fly. We benchmark our method in motion tracking tasks on SMPL Humanoid, continuous control on ExoRL, and for the long-horizon OGBench tasks. Our technique seamlessly integrates into forward-backward representation frameworks and enables fast-adaptation to new tasks in a training-free regime. More broadly, this work bridges off-policy learning and zero-shot adaptation, offering benefits to both research areas.