Hierarchical Decentralized Deep Reinforcement Learning Architecture for a Simulated Four-Legged Agent

TL;DR

This paper introduces a novel hierarchical decentralized architecture for controlling a simulated four-legged robot, demonstrating improved learning efficiency, energy efficiency, and robustness over traditional centralized methods through comprehensive benchmarking.

Contribution

The work presents a new hierarchical decentralized control architecture inspired by biological principles, showing advantages in learning, energy efficiency, and robustness for legged robots.

Findings

Decentralized hierarchical control improves learning speed.

Energy-efficient movements are achieved with the new architecture.

The architecture enhances robustness in unseen environments.

Abstract

Legged locomotion is widespread in nature and has inspired the design of current robots. The controller of these legged robots is often realized as one centralized instance. However, in nature, control of movement happens in a hierarchical and decentralized fashion. Introducing these biological design principles into robotic control systems has motivated this work. We tackle the question whether decentralized and hierarchical control is beneficial for legged robots and present a novel decentral, hierarchical architecture to control a simulated legged agent. Three different tasks varying in complexity are designed to benchmark five architectures (centralized, decentralized, hierarchical and two different combinations of hierarchical decentralized architectures). The results demonstrate that decentralizing the different levels of the hierarchical architectures facilitates learning of the agent, ensures more energy efficient movements as well as robustness towards new unseen environments. Furthermore, this comparison sheds light on the importance of modularity in hierarchical architectures to solve complex goal-directed tasks. We provide an open-source code implementation of our architecture (https://github.com/wzaielamri/hddrl).