Reinforcement Learning for Robot Navigation with Adaptive Forward Simulation Time (AFST) in a Semi-Markov Model

TL;DR

This paper introduces AFST, a novel DRL-based robot navigation method using a semi-Markov decision process with adaptive simulation time to effectively handle local minima in complex unknown environments.

Contribution

The paper presents the first DRL navigation approach modeled by a semi-Markov decision process with continuous actions, incorporating adaptive simulation time to improve navigation in unknown environments.

Findings

AFST outperforms existing methods in unknown environments.

Modified GAE enhances policy gradient estimation in SMDPs.

Experimental results validate the effectiveness of AFST.

Abstract

Deep reinforcement learning (DRL) algorithms have proven effective in robot navigation, especially in unknown environments, by directly mapping perception inputs into robot control commands. However, most existing methods ignore the local minimum problem in navigation and thereby cannot handle complex unknown environments. In this paper, we propose the first DRL-based navigation method modeled by a semi-Markov decision process (SMDP) with continuous action space, named Adaptive Forward Simulation Time (AFST), to overcome this problem. Specifically, we reduce the dimensions of the action space and improve the distributed proximal policy optimization (DPPO) algorithm for the specified SMDP problem by modifying its GAE to better estimate the policy gradient in SMDPs. Experiments in various unknown environments demonstrate the effectiveness of AFST.