Second-Order MPC-Based Distributed Q-Learning

TL;DR

This paper introduces a second-order extension to distributed MPC-based Q-learning, leveraging local information and neighbor communication to enhance convergence speed and outperform first-order methods.

Contribution

It presents a novel second-order distributed Q-learning algorithm based on MPC that improves learning speed without requiring global information.

Findings

Significantly faster convergence compared to first-order methods

Effective use of local and neighbor information for updates

Demonstrated improved performance in simulations

Abstract

The state of the art for model predictive control (MPC)-based distributed Q-learning is limited to first-order gradient updates of the MPC parameterization. In general, using secondorder information can significantly improve the speed of convergence for learning, allowing the use of higher learning rates without introducing instability. This work presents a second-order extension to MPC-based Q-learning with updates distributed across local agents, relying only on locally available information and neighbor-to-neighbor communication. In simulation the approach is demonstrated to significantly outperform first-order distributed Q-learning.