Bridging RL and MPC for mixed-integer optimal control with application to Formula 1 race strategies

TL;DR

This paper introduces a hybrid RL-MPC framework for mixed-integer optimal control, demonstrating its effectiveness on Formula 1 race strategies by combining learning and optimization for improved performance and adaptability.

Contribution

It presents a novel integrated RL and MPC approach trained on the full hybrid action space, ensuring consistency and enabling zero-retraining adaptation to disturbances.

Findings

Achieves near-optimal performance compared to offline benchmarks.

Outperforms standalone RL agents in the race strategy task.

Enables disturbance adaptation through modular MPC extensions.

Abstract

We propose a hybrid reinforcement learning (RL) and model predictive control (MPC) framework for mixed-integer optimal control, where discrete variables enter the cost and dynamics but not the constraints. Existing hierarchical approaches use RL only for the discrete action space, leaving continuous optimization to MPC. Unlike these methods, we train the RL agent on the full hybrid action space, ensuring consistency with the cost of the underlying Markov decision process. During deployment, the RL actor is rolled out over the prediction horizon to parametrize an integer-free nonlinear MPC through the discrete action sequence and provide a continuous warm-start. The learned critic serves as a terminal cost to capture long-term performance. We prove recursive feasibility, and validate the framework on a Formula 1 race strategy problem. The hybrid method achieves near-optimal performance relative to an offline mixed-integer nonlinear program benchmark, outperforming a standalone RL agent. Moreover, the hybrid scheme enables adaptation to unseen disturbances through modular MPC extensions at zero retraining cost.