Bayesian Soft Actor-Critic: A Directed Acyclic Strategy Graph Based Deep Reinforcement Learning

TL;DR

This paper introduces Bayesian Soft Actor-Critic (BSAC), a novel deep reinforcement learning approach that decomposes complex policies into simpler sub-policies using Bayesian strategy networks, leading to improved training efficiency.

Contribution

It proposes a new directed acyclic strategy graph decomposition method integrated with SAC, forming BSAC, which enhances policy learning in continuous control tasks.

Findings

BSAC outperforms existing DRL algorithms in training efficiency.

The approach effectively decomposes complex policies into manageable sub-policies.

Experimental results on OpenAI Gym benchmarks validate the method's advantages.

Abstract

Adopting reasonable strategies is challenging but crucial for an intelligent agent with limited resources working in hazardous, unstructured, and dynamic environments to improve the system's utility, decrease the overall cost, and increase mission success probability. This paper proposes a novel directed acyclic strategy graph decomposition approach based on Bayesian chaining to separate an intricate policy into several simple sub-policies and organize their relationships as Bayesian strategy networks (BSN). We integrate this approach into the state-of-the-art DRL method -- soft actor-critic (SAC), and build the corresponding Bayesian soft actor-critic (BSAC) model by organizing several sub-policies as a joint policy. We compare our method against the state-of-the-art deep reinforcement learning algorithms on the standard continuous control benchmarks in the OpenAI Gym environment. The results demonstrate that the promising potential of the BSAC method significantly improves training efficiency.