Interactive Double Deep Q-network: Integrating Human Interventions and Evaluative Predictions in Reinforcement Learning of Autonomous Driving

TL;DR

This paper presents iDDQN, a Human-in-the-Loop reinforcement learning method for autonomous driving that integrates human interventions into the training process and evaluates their impact through offline simulation, outperforming existing approaches.

Contribution

The study introduces iDDQN, a novel RL framework that incorporates human expertise directly into the learning process and provides an offline evaluative method for intervention effectiveness.

Findings

iDDQN outperforms baseline methods in simulated autonomous driving tasks.

The integrated approach improves model performance and adaptability.

Offline evaluation effectively measures human intervention impact.

Abstract

Integrating human expertise with machine learning is crucial for applications demanding high accuracy and safety, such as autonomous driving. This study introduces Interactive Double Deep Q-network (iDDQN), a Human-in-the-Loop (HITL) approach that enhances Reinforcement Learning (RL) by merging human insights directly into the RL training process, improving model performance. Our proposed iDDQN method modifies the Q-value update equation to integrate human and agent actions, establishing a collaborative approach for policy development. Additionally, we present an offline evaluative framework that simulates the agent's trajectory as if no human intervention had occurred, to assess the effectiveness of human interventions. Empirical results in simulated autonomous driving scenarios demonstrate that iDDQN outperforms established approaches, including Behavioral Cloning (BC), HG-DAgger, Deep Q-Learning from Demonstrations (DQfD), and vanilla DRL in leveraging human expertise for improving performance and adaptability.