Improving Q-Learning for Real-World Control: A Case Study in Series Hybrid Agricultural Tractors

TL;DR

This paper enhances Q-learning algorithms for hybrid agricultural tractor control by introducing reward shaping and expert demonstration strategies, significantly improving learning speed and policy efficiency in real-world energy management tasks.

Contribution

It evaluates and compares advanced Q-learning algorithms, proposes a reward-shaping method, and analyzes expert demonstration effects to optimize hybrid tractor energy management.

Findings

DDQN converges 70% faster than DQN.

Reward shaping biases policies toward fuel efficiency.

Expert demonstrations improve convergence speed by 33%.

Abstract

The variable and unpredictable load demands in hybrid agricultural tractors make it difficult to design optimal rule-based energy management strategies, motivating the use of adaptive, learning-based control. However, existing approaches often rely on basic fuel-based rewards and do not leverage expert demonstrations to accelerate training. In this paper, first, the performance of Q-value-based reinforcement learning algorithms is evaluated for powertrain control in a hybrid agricultural tractor. Three algorithms, Double Q-Learning (DQL), Deep Q-Networks (DQN), and Double DQN (DDQN), are compared in terms of convergence speed and policy optimality. Second, a piecewise domain-specific reward-shaping strategy is introduced to improve learning efficiency and steer agent behavior toward engine fuel-efficient operating regions. Third, the design of the experience replay buffer is examined, with a focus on the effects of seeding the buffer with expert demonstrations and analyzing how different types of expert policies influence convergence dynamics and final performance. Experimental results demonstrate that (1) DDQN achieves 70\% faster convergence than DQN in this application domain, (2) the proposed reward shaping method effectively biases the learned policy toward fuel-efficient outcomes, and (3) initializing the replay buffer with structured expert data leads to a 33\% improvement in convergence speed.