A Prescriptive Dirichlet Power Allocation Policy with Deep Reinforcement Learning

TL;DR

This paper introduces a Dirichlet policy for reinforcement learning to optimize power allocation in complex systems, demonstrating faster convergence and improved performance over existing methods in a lithium-ion battery case study.

Contribution

The paper proposes a novel Dirichlet policy for continuous action spaces in RL, specifically addressing power allocation with simplex constraints, and shows its advantages over Gaussian-softmax policies.

Findings

Dirichlet policy is bias-free and converges faster.

The proposed method improves efficiency and sustainability in battery systems.

Experimental results validate the effectiveness of the Dirichlet power allocation policy.

Abstract

Prescribing optimal operation based on the condition of the system and, thereby, potentially prolonging the remaining useful lifetime has a large potential for actively managing the availability, maintenance and costs of complex systems. Reinforcement learning (RL) algorithms are particularly suitable for this type of problems given their learning capabilities. A special case of a prescriptive operation is the power allocation task, which can be considered as a sequential allocation problem, where the action space is bounded by a simplex constraint. A general continuous action-space solution of such sequential allocation problems has still remained an open research question for RL algorithms. In continuous action-space, the standard Gaussian policy applied in reinforcement learning does not support simplex constraints, while the Gaussian-softmax policy introduces a bias during training. In this work, we propose the Dirichlet policy for continuous allocation tasks and analyze the bias and variance of its policy gradients. We demonstrate that the Dirichlet policy is bias-free and provides significantly faster convergence, better performance and better hyperparameters robustness over the Gaussian-softmax policy. Moreover, we demonstrate the applicability of the proposed algorithm on a prescriptive operation case, where we propose the Dirichlet power allocation policy and evaluate the performance on a case study of a set of multiple lithium-ion (Li-I) battery systems. The experimental results show the potential to prescribe optimal operation, improve the efficiency and sustainability of multi-power source systems.