Risk-averse policies for natural gas futures trading using distributional reinforcement learning

TL;DR

This paper explores the application of distributional reinforcement learning algorithms to natural gas futures trading, demonstrating their superiority over classical methods and their ability to produce adjustable risk-averse policies in volatile markets.

Contribution

It introduces the first application of distributional RL algorithms (C51, QR-DQN, IQN) in trading, showing their effectiveness and flexibility in risk management compared to traditional ML baselines.

Findings

Distributional RL algorithms outperform classical RL by over 32%.

Training C51 and IQN to maximize CVaR yields adjustable risk-averse policies.

Lower CVaR levels increase risk aversion, higher levels decrease it.

Abstract

Financial markets have experienced significant instabilities in recent years, creating unique challenges for trading and increasing interest in risk-averse strategies. Distributional Reinforcement Learning (RL) algorithms, which model the full distribution of returns rather than just expected values, offer a promising approach to managing market uncertainty. This paper investigates this potential by studying the effectiveness of three distributional RL algorithms for natural gas futures trading and exploring their capacity to develop risk-averse policies. Specifically, we analyze the performance and behavior of Categorical Deep Q-Network (C51), Quantile Regression Deep Q-Network (QR-DQN), and Implicit Quantile Network (IQN). To the best of our knowledge, these algorithms have never been applied in a trading context. These policies are compared against five Machine Learning (ML) baselines, using a detailed dataset provided by Predictive Layer SA, a company supplying ML-based strategies for energy trading. The main contributions of this study are as follows. (1) We demonstrate that distributional RL algorithms significantly outperform classical RL methods, with C51 achieving performance improvement of more than 32\%. (2) We show that training C51 and IQN to maximize CVaR produces risk-sensitive policies with adjustable risk aversion. Specifically, our ablation studies reveal that lower CVaR confidence levels increase risk aversion, while higher levels decrease it, offering flexible risk management options. In contrast, QR-DQN shows less predictable behavior. These findings emphasize the potential of distributional RL for developing adaptable, risk-averse trading strategies in volatile markets.