Deep Reinforcement Trading with Predictable Returns

TL;DR

This paper explores the use of deep reinforcement learning for trading in markets with predictable mean-reverting factors, demonstrating its ability to outperform benchmarks under certain market conditions.

Contribution

It introduces a framework to evaluate DRL trading algorithms in known market environments and shows how classical strategies can enhance DRL performance.

Findings

DRL agents can effectively learn trading signals in mean-reverting markets.

Combining classical strategies with DRL improves robustness to market misspecification.

The approach outperforms benchmark strategies in volatile and extreme market conditions.

Abstract

Classical portfolio optimization often requires forecasting asset returns and their corresponding variances in spite of the low signal-to-noise ratio provided in the financial markets. Modern deep reinforcement learning (DRL) offers a framework for optimizing sequential trader decisions but lacks theoretical guarantees of convergence. On the other hand, the performances on real financial trading problems are strongly affected by the goodness of the signal used to predict returns. To disentangle the effects coming from return unpredictability from those coming from algorithm un-trainability, we investigate the performance of model-free DRL traders in a market environment with different known mean-reverting factors driving the dynamics. When the framework admits an exact dynamic programming solution, we can assess the limits and capabilities of different value-based algorithms to retrieve meaningful trading signals in a data-driven manner. We consider DRL agents that leverage classical strategies to increase their performances and we show that this approach guarantees flexibility, outperforming the benchmark strategy when the price dynamics is misspecified and some original assumptions on the market environment are violated with the presence of extreme events and volatility clustering.