Conditional Importance Sampling for Off-Policy Learning

TL;DR

This paper introduces a new conceptual framework for off-policy reinforcement learning using conditional importance sampling, offering fresh insights and revealing new algorithmic possibilities.

Contribution

It presents a novel framework based on conditional expectations of importance sampling ratios, enhancing understanding and expanding the space of off-policy algorithms.

Findings

The framework provides new perspectives on existing algorithms.

Theoretical analysis of the algorithmic space.

Investigation of several algorithms derived from the framework.

Abstract

The principal contribution of this paper is a conceptual framework for off-policy reinforcement learning, based on conditional expectations of importance sampling ratios. This framework yields new perspectives and understanding of existing off-policy algorithms, and reveals a broad space of unexplored algorithms. We theoretically analyse this space, and concretely investigate several algorithms that arise from this framework.