Time-Series-Informed Closed-loop Learning for Sequential Decision Making and Control

TL;DR

This paper introduces a time-series-informed multi-fidelity Bayesian optimization method that improves the efficiency of tuning sequential decision-making controllers by leveraging temporal structure and early stopping, reducing resource use.

Contribution

It proposes a novel framework that incorporates temporal information into Bayesian optimization and develops probabilistic early stopping criteria for efficient closed-loop controller tuning.

Findings

Achieves comparable performance with half the experiments compared to standard methods.

Yields better final performance under the same resource constraints.

Demonstrates effectiveness on a nonlinear control benchmark.

Abstract

Closed-loop performance of sequential decision making algorithms, such as model predictive control, depends strongly on the choice of controller parameters. Bayesian optimization allows learning of parameters from closed-loop experiments, but standard Bayesian optimization treats this as a black-box problem and ignores the temporal structure of closed-loop trajectories, leading to slow convergence and inefficient use of experimental resources. We propose a time-series-informed multi-fidelity Bayesian optimization framework that aligns the fidelity dimension with closed-loop time, enabling intermediate performance evaluations within a closed-loop experiment to be incorporated as lower-fidelity observations. Additionally, we derive probabilistic early stopping criteria to terminate unpromising closed-loop experiments based on the surrogate model's posterior belief, avoiding full episodes for poor parameterizations and thereby reducing resource usage. Simulation results on a nonlinear control benchmark demonstrate that, compared to standard black-box Bayesian optimization approaches, the proposed method achieves comparable closed-loop performance with roughly half the experimental resources, and yields better final performance when using the same resource budget, highlighting the value of exploiting temporal structure for sample-efficient closed-loop controller tuning.