Performance-Driven Controller Tuning via Derivative-Free Reinforcement Learning

TL;DR

This paper introduces a novel derivative-free reinforcement learning framework for automatic controller tuning, addressing scalability and efficiency issues of existing methods, and demonstrates its effectiveness on autonomous driving tasks.

Contribution

The paper presents a new derivative-free RL approach that integrates parameter perturbation and advanced actor-critic architecture for efficient controller tuning.

Findings

Outperforms baseline methods in controller tuning tasks

Effective on adaptive cruise control and trajectory tracking

Shows strong potential for practical autonomous driving applications

Abstract

Choosing an appropriate parameter set for the designed controller is critical for the final performance but usually requires a tedious and careful tuning process, which implies a strong need for automatic tuning methods. However, among existing methods, derivative-free ones suffer from poor scalability or low efficiency, while gradient-based ones are often unavailable due to possibly non-differentiable controller structure. To resolve the issues, we tackle the controller tuning problem using a novel derivative-free reinforcement learning (RL) framework, which performs timestep-wise perturbation in parameter space during experience collection and integrates derivative-free policy updates into the advanced actor-critic RL architecture to achieve high versatility and efficiency. To demonstrate the framework's efficacy, we conduct numerical experiments on two concrete examples from autonomous driving, namely, adaptive cruise control with PID controller and trajectory tracking with MPC controller. Experimental results show that the proposed method outperforms popular baselines and highlight its strong potential for controller tuning.