Distributionally Robust Surrogate Optimal Control for High-Dimensional Systems

TL;DR

This paper introduces a distributionally robust surrogate modeling approach for high-dimensional optimal control and offline reinforcement learning, emphasizing safety and computational efficiency through uncertainty quantification and dimensionality reduction.

Contribution

It develops a novel surrogate methodology that absorbs system dynamics, applies Wasserstein ambiguity sets for robustness, and enables safe, efficient control of high-dimensional systems.

Findings

Effective in safe optimal control of lithium-ion batteries

Robust to out-of-distribution modeling errors

Reduces computational complexity in high-dimensional systems

Abstract

This paper presents a novel methodology for tractably solving optimal control and offline reinforcement learning problems for high-dimensional systems. This work is motivated by the ongoing challenges of safety, computation, and optimality in high-dimensional optimal control. We address these key questions with the following approach. First, we identify a sequence-modeling surrogate methodology which takes as input the initial state and a time series of control inputs, and outputs an approximation of the objective function and trajectories of constraint functions. Importantly this approach entirely absorbs the individual state transition dynamics. The sole dependence on the initial state means we can apply dimensionality reduction to compress the model input while retaining most of its information. Uncertainty in the surrogate objective will affect the result optimality. Critically, however, uncertainty in the surrogate constraint functions will lead to infeasibility, i.e. unsafe actions. When considering offline reinforcement learning, the most significant modeling error will be encountered on out-of-distribution data. Therefore, we apply Wasserstein ambiguity sets to ``robustify'' our surrogate modeling approach subject to worst-case out-of-sample modeling error based on the distribution of test data residuals. We demonstrate the efficacy of this combined approach through a case study of safe optimal fast charging of a high-dimensional lithium-ion battery model at low temperatures.