Reinforcement Learning-based Home Energy Management with Heterogeneous Batteries and Stochastic EV Behaviour

TL;DR

This paper introduces a deep reinforcement learning framework for home energy management that optimally coordinates PV, EV, and stationary storage, considering battery heterogeneity and stochastic user behavior to reduce costs and degradation.

Contribution

It presents a novel DRL-based approach that explicitly models heterogeneous battery degradation and stochastic EV behavior within a constrained MDP for improved energy management.

Findings

Significant reduction in operating costs compared to rule-based methods.

Battery degradation costs decreased by 8.44%.

Effective enforcement of physical constraints and occupant comfort.

Abstract

The widespread adoption of photovoltaic (PV), electric vehicles (EVs), and stationary energy storage systems (ESS) in households increases system complexity while simultaneously offering new opportunities for energy regulation. However, effectively coordinating these resources under uncertainties remains challenging. This paper proposes a novel home energy management framework based on deep reinforcement learning (DRL) that can jointly minimise energy expenditure and battery degradation while guaranteeing occupant comfort and EV charging requirements. Distinct from existing studies, we explicitly account for the heterogeneous degradation characteristics of stationary and EV batteries in the optimisation, alongside stochastic user behaviour regarding arrival time, departure time, and driving distance. The energy scheduling problem is formulated as a constrained Markov decision process (CMDP) and solved using a Lagrangian soft actor-critic (SAC) algorithm. This approach enables the agent to learn optimal control policies that enforce physical constraints, including indoor temperature bounds and target EV state of charge upon departure, despite stochastic uncertainties. Numerical simulations over a one-year horizon demonstrate the effectiveness of the proposed framework in satisfying physical constraints while eliminating thermal oscillations and achieving significant economic benefits. Specifically, the method reduces the cumulative operating cost substantially compared to two standard rule-based baselines while simultaneously decreasing battery degradation costs by 8.44%.