An Approximate Dynamic Programming Approach for Dual Stochastic Model Predictive Control

TL;DR

This paper introduces an approximate dual control method for continuous systems using scenario trees and Bayesian updates, enabling practical stochastic model predictive control with explicit exploration-exploitation trade-offs.

Contribution

It presents a novel rollout dynamic programming approach that approximates dual control in continuous domains via scenario sampling and Bayesian parameter estimation.

Findings

Enables dual control in continuous state and input spaces.

Formulates the control problem as a single optimization over scenario trees.

Facilitates explicit exploration-exploitation trade-offs in model predictive control.

Abstract

Dual control explicitly addresses the problem of trading off active exploration and exploitation in the optimal control of partially unknown systems. While the problem can be cast in the framework of stochastic dynamic programming, exact solutions are only tractable for discrete state and action spaces of very small dimension due to a series of nested minimization and expectation operations. We propose an approximate dual control method for systems with continuous state and input domain based on a rollout dynamic programming approach, splitting the control horizon into a dual and an exploitation part. The dual part is approximated using a scenario tree generated by sampling the process noise and the unknown system parameters, for which the underlying distribution is updated via Bayesian estimation along the horizon. In the exploitation part, we fix the resulting parameter estimate of each scenario branch and compute an open-loop control sequence for the remainder of the horizon. The key benefit of the proposed sampling-based approximation is that it enables the formulation as one optimization problem that computes a collection of control sequences over the scenario tree, leading to a dual model predictive control formulation.