DiffOP: Reinforcement Learning of Optimization-Based Control Policies via Implicit Policy Gradients

TL;DR

DiffOP introduces a novel reinforcement learning framework that learns optimization-based control policies implicitly, directly optimizing control costs without value function approximation, and demonstrates convergence and effectiveness on nonlinear control tasks.

Contribution

It presents a new method for learning implicit control policies via policy gradients with analytical derivatives, avoiding value function approximation.

Findings

Converges to an $psilon$-stationary point within $psilon^{-1}$ iterations.

Effective on nonlinear control and power system voltage control tasks.

Provides analytical policy gradients through implicit differentiation.

Abstract

Real-world control systems require policies that are not only high-performing but also interpretable and robust. A promising direction toward this goal is model-based control, which learns system dynamics and cost functions from historical data and then uses these models to inform decision-making. Building on this paradigm, we introduce DiffOP, a novel framework for learning optimization-based control policies defined implicitly through optimization control problems. Without relying on value function approximation, DiffOP jointly learns the cost and dynamics models and directly optimizes the actual control costs using policy gradients. To enable this, we derive analytical policy gradients by applying implicit differentiation to the underlying optimization problem and integrating it with the standard policy gradient framework. Under standard regularity conditions, we establish that DiffOP converges to an $\epsilon$-stationary point within $\mathcal{O}(\epsilon^{-1})$ iterations. We demonstrate the effectiveness of DiffOP through experiments on nonlinear control tasks and power system voltage control with constraints. The code is available at https://github.com/alwaysbyx/DiffOP.