Human-centric Reward Optimization for Reinforcement Learning-based Automated Driving using Large Language Models

TL;DR

This paper presents a novel method using large language models to optimize reinforcement learning rewards for automated driving, resulting in more human-like behavior and improved performance.

Contribution

Introduces a human-centric reward optimization framework leveraging LLMs to enhance RL-based automated driving agents.

Findings

LLMs can effectively generate rewards that promote human-like driving behavior

Prompt design significantly influences the behavior of RL agents

The approach improves both anthropomorphism and performance of automated driving systems

Abstract

One of the key challenges in current Reinforcement Learning (RL)-based Automated Driving (AD) agents is achieving flexible, precise, and human-like behavior cost-effectively. This paper introduces an innovative approach that uses large language models (LLMs) to intuitively and effectively optimize RL reward functions in a human-centric way. We developed a framework where instructions and dynamic environment descriptions are input into the LLM. The LLM then utilizes this information to assist in generating rewards, thereby steering the behavior of RL agents towards patterns that more closely resemble human driving. The experimental results demonstrate that this approach not only makes RL agents more anthropomorphic but also achieves better performance. Additionally, various strategies for reward-proxy and reward-shaping are investigated, revealing the significant impact of prompt design on shaping an AD vehicle's behavior. These findings offer a promising direction for the development of more advanced, human-like automated driving systems. Our experimental data and source code can be found here