Sample Efficient Active Algorithms for Offline Reinforcement Learning

TL;DR

This paper provides a theoretical analysis of Active Reinforcement Learning, showing it can efficiently learn near-optimal policies with fewer online interactions by leveraging Gaussian Process uncertainty modeling.

Contribution

It introduces a new algorithm for ActiveRL with a rigorous sample complexity analysis, demonstrating improved efficiency over offline methods using Gaussian Process techniques.

Findings

ActiveRL achieves $ ilde{O}(1/ ext{epsilon}^2)$ sample complexity.

Theoretical guarantees show near-optimal information efficiency.

Experiments validate the theoretical results.

Abstract

Offline reinforcement learning (RL) enables policy learning from static data but often suffers from poor coverage of the state-action space and distributional shift problems. This problem can be addressed by allowing limited online interactions to selectively refine uncertain regions of the learned value function, which is referred to as Active Reinforcement Learning (ActiveRL). While there has been good empirical success, no theoretical analysis is available in the literature. We fill this gap by developing a rigorous sample-complexity analysis of ActiveRL through the lens of Gaussian Process (GP) uncertainty modeling. In this respect, we propose an algorithm and using GP concentration inequalities and information-gain bounds, we derive high-probability guarantees showing that an $\epsilon$-optimal policy can be learned with ${\mathcal{O}}(1/\epsilon^2)$ active transitions, improving upon the $\Omega(1/\epsilon^2(1-\gamma)^4)$ rate of purely offline methods. Our results reveal that ActiveRL achieves near-optimal information efficiency, that is, guided uncertainty reduction leads to accelerated value-function convergence with minimal online data. Our analysis builds on GP concentration inequalities and information-gain bounds, bridging Bayesian nonparametric regression and reinforcement learning theories. We conduct several experiments to validate the algorithm and theoretical findings.