Deep Reinforcement Learning for Stock Portfolio Optimization

TL;DR

This paper applies deep reinforcement learning algorithms to stock portfolio optimization, incorporating transaction costs and risk, and compares various state-of-the-art continuous policy gradient methods for improved financial decision-making.

Contribution

It formulates stock portfolio optimization as a reinforcement learning problem with realistic market assumptions and evaluates multiple advanced algorithms for the first time in this context.

Findings

DDPG and GDPG outperform PPO in this setting

Incorporating transaction costs and risk factors improves model realism

Wavelet Transform enhances multi-frequency data analysis

Abstract

Stock portfolio optimization is the process of constant re-distribution of money to a pool of various stocks. In this paper, we will formulate the problem such that we can apply Reinforcement Learning for the task properly. To maintain a realistic assumption about the market, we will incorporate transaction cost and risk factor into the state as well. On top of that, we will apply various state-of-the-art Deep Reinforcement Learning algorithms for comparison. Since the action space is continuous, the realistic formulation were tested under a family of state-of-the-art continuous policy gradients algorithms: Deep Deterministic Policy Gradient (DDPG), Generalized Deterministic Policy Gradient (GDPG) and Proximal Policy Optimization (PPO), where the former two perform much better than the last one. Next, we will present the end-to-end solution for the task with Minimum Variance Portfolio Theory for stock subset selection, and Wavelet Transform for extracting multi-frequency data pattern. Observations and hypothesis were discussed about the results, as well as possible future research directions.1