Sensor-Based Navigation Using Hierarchical Reinforcement Learning

TL;DR

This paper presents a hierarchical reinforcement learning approach for lidar-based robot navigation that learns goal-directed behavior solely from local sensor data and sparse rewards, demonstrating improved performance and real-world transferability.

Contribution

The paper introduces a hierarchical DRL framework for navigation that self-assigns internal goals using only local sensor data, enhancing learning efficiency and real-world applicability.

Findings

Hierarchical structure improves success rate and efficiency.

Agent successfully navigates in simulated environments.

Real-robot experiment confirms transferability.

Abstract

Robotic systems are nowadays capable of solving complex navigation tasks. However, their capabilities are limited to the knowledge of the designer and consequently lack generalizability to initially unconsidered situations. This makes deep reinforcement learning (DRL) especially interesting, as these algorithms promise a self-learning system only relying on feedback from the environment. In this paper, we consider the problem of lidar-based robot navigation in continuous action space using DRL without providing any goal-oriented or global information. By relying solely on local sensor data to solve navigation tasks, we design an agent that assigns its own waypoints based on intrinsic motivation. Our agent is able to learn goal-directed navigation behavior even when facing only sparse feedback, i.e., delayed rewards when reaching the target. To address this challenge and the complexity of the continuous action space, we deploy a hierarchical agent structure in which the exploration is distributed across multiple layers. Within the hierarchical structure, our agent self-assigns internal goals and learns to extract reasonable waypoints to reach the desired target position only based on local sensor data. In our experiments, we demonstrate the navigation capabilities of our agent in two environments and show that the hierarchical structure seriously improves the performance in terms of success rate and success weighted by path length in comparison to a flat structure. Furthermore, we provide a real-robot experiment to illustrate that the trained agent can be easily transferred to a real-world scenario.