Optimization Solution Functions as Deterministic Policies for Offline Reinforcement Learning

TL;DR

This paper introduces an implicit actor-critic framework for offline reinforcement learning that uses optimization solution functions as deterministic policies, providing robustness and performance guarantees, validated on real-world tasks.

Contribution

The paper proposes a novel implicit actor-critic framework that encodes optimality in the policy and critic, improving robustness and performance in offline RL.

Findings

Robust policies via exponential decay of suboptimality sensitivity

Performance guarantees for the implicit actor-critic framework

Significant improvements over state-of-the-art offline RL methods in real-world applications

Abstract

Offline reinforcement learning (RL) is a promising approach for many control applications but faces challenges such as limited data coverage and value function overestimation. In this paper, we propose an implicit actor-critic (iAC) framework that employs optimization solution functions as a deterministic policy (actor) and a monotone function over the optimal value of optimization as a critic. By encoding optimality in the actor policy, we show that the learned policies are robust to the suboptimality of the learned actor parameters via the exponentially decaying sensitivity (EDS) property. We obtain performance guarantees for the proposed iAC framework and show its benefits over general function approximation schemes. Finally, we validate the proposed framework on two real-world applications and show a significant improvement over state-of-the-art (SOTA) offline RL methods.