Active Offline Policy Selection

TL;DR

This paper proposes an active offline policy selection method that combines logged data and limited online interactions using Bayesian optimization to identify the best policy more efficiently than traditional methods.

Contribution

It introduces a novel sequential decision approach that leverages OPE estimates and policy similarity to improve policy selection with limited online interactions.

Findings

Outperforms state-of-the-art OPE estimates.

Enhances policy selection efficiency in real-world robotics.

Reduces online interaction costs while maintaining high policy quality.

Abstract

This paper addresses the problem of policy selection in domains with abundant logged data, but with a restricted interaction budget. Solving this problem would enable safe evaluation and deployment of offline reinforcement learning policies in industry, robotics, and recommendation domains among others. Several off-policy evaluation (OPE) techniques have been proposed to assess the value of policies using only logged data. However, there is still a big gap between the evaluation by OPE and the full online evaluation. Yet, large amounts of online interactions are often not possible in practice. To overcome this problem, we introduce active offline policy selection - a novel sequential decision approach that combines logged data with online interaction to identify the best policy. We use OPE estimates to warm start the online evaluation. Then, in order to utilize the limited environment interactions wisely we decide which policy to evaluate next based on a Bayesian optimization method with a kernel that represents policy similarity. We use multiple benchmarks, including real-world robotics, with a large number of candidate policies to show that the proposed approach improves upon state-of-the-art OPE estimates and pure online policy evaluation.