Natural-gas storage modelling by deep reinforcement learning

TL;DR

This paper presents GasRL, a deep reinforcement learning-based simulator for natural gas storage management, demonstrating how optimal policies influence market prices, demand, and supply dynamics, and assessing policy impacts like storage thresholds.

Contribution

Introduction of GasRL, a novel simulator integrating market modeling with RL-trained storage policies, showing superior SAC algorithm performance and policy effects on market stability.

Findings

SAC outperforms other RL algorithms in GasRL environment.

Optimal storage policies replicate real-world price volatility and seasonality.

EU storage thresholds improve market resilience against supply shocks.

Abstract

We introduce GasRL, a simulator that couples a calibrated representation of the natural gas market with a model of storage-operator policies trained with deep reinforcement learning (RL). We use it to analyse how optimal stockpile management affects equilibrium prices and the dynamics of demand and supply. We test various RL algorithms and find that Soft Actor Critic (SAC) exhibits superior performance in the GasRL environment: multiple objectives of storage operators - including profitability, robust market clearing and price stabilisation - are successfully achieved. Moreover, the equilibrium price dynamics induced by SAC-derived optimal policies have characteristics, such as volatility and seasonality, that closely match those of real-world prices. Remarkably, this adherence to the historical distribution of prices is obtained without explicitly calibrating the model to price data. We show how the simulator can be used to assess the effects of EU-mandated minimum storage thresholds. We find that such thresholds have a positive effect on market resilience against unanticipated shifts in the distribution of supply shocks. For example, with unusually large shocks, market disruptions are averted more often if a threshold is in place.