Action Space Reduction Strategies for Reinforcement Learning in Autonomous Driving

TL;DR

This paper introduces dynamic and relative action space reduction strategies for reinforcement learning in autonomous driving, improving training efficiency and policy performance by context-aware action masking.

Contribution

It proposes novel structured action space modification methods that adaptively reduce action spaces based on context, enhancing RL training in autonomous driving.

Findings

Action space reduction improves training stability.

Dynamic and relative schemes balance learning speed and control accuracy.

Context-aware strategies outperform fixed reduction schemes.

Abstract

Reinforcement Learning (RL) offers a promising framework for autonomous driving by enabling agents to learn control policies through interaction with environments. However, large and high-dimensional action spaces often used to support fine-grained control can impede training efficiency and increase exploration costs. In this study, we introduce and evaluate two novel structured action space modification strategies for RL in autonomous driving: dynamic masking and relative action space reduction. These approaches are systematically compared against fixed reduction schemes and full action space baselines to assess their impact on policy learning and performance. Our framework leverages a multimodal Proximal Policy Optimization agent that processes both semantic image sequences and scalar vehicle states. The proposed dynamic and relative strategies incorporate real-time action masking based on context and state transitions, preserving action consistency while eliminating invalid or suboptimal choices. Through comprehensive experiments across diverse driving routes, we show that action space reduction significantly improves training stability and policy performance. The dynamic and relative schemes, in particular, achieve a favorable balance between learning speed, control precision, and generalization. These findings highlight the importance of context-aware action space design for scalable and reliable RL in autonomous driving tasks.