Tactical Reward Shaping: Bypassing Reinforcement Learning with Strategy-Based Goals

TL;DR

This paper introduces a strategy-based goal setting approach in Deep Reinforcement Learning that accelerates convergence to winning strategies in robotic competitions by focusing on geometric advantages rather than traditional win/loss rewards.

Contribution

It proposes a novel goal formulation method that bypasses traditional reward functions, demonstrating that geometric-based objectives can outperform standard DRL algorithms in complex environments.

Findings

Geometric goal setting accelerates learning convergence.

Geometric-based searches outperform Deep Q Learning in certain scenarios.

Multi-agent path planning improves robot cooperation.

Abstract

Deep Reinforcement Learning (DRL) has shown its promising capabilities to learn optimal policies directly from trial and error. However, learning can be hindered if the goal of the learning, defined by the reward function, is "not optimal". We demonstrate that by setting the goal/target of competition in a counter-intuitive but intelligent way, instead of heuristically trying solutions through many hours the DRL simulation can quickly converge into a winning strategy. The ICRA-DJI RoboMaster AI Challenge is a game of cooperation and competition between robots in a partially observable environment, quite similar to the Counter-Strike game. Unlike the traditional approach to games, where the reward is given at winning the match or hitting the enemy, our DRL algorithm rewards our robots when in a geometric-strategic advantage, which implicitly increases the winning chances. Furthermore, we use Deep Q Learning (DQL) to generate multi-agent paths for moving, which improves the cooperation between two robots by avoiding the collision. Finally, we implement a variant A* algorithm with the same implicit geometric goal as DQL and compare results. We conclude that a well-set goal can put in question the need for learning algorithms, with geometric-based searches outperforming DQL in many orders of magnitude.