Evaluation of Reinforcement Learning Techniques for Trading on a Diverse Portfolio

TL;DR

This study evaluates reinforcement learning methods for stock trading on the S&P 500, finding that on-policy techniques outperform Q-learning in certain market conditions, especially when trained with COVID-19 data.

Contribution

It compares on-policy and off-policy reinforcement learning techniques on stock data, highlighting the impact of training data and market conditions on performance.

Findings

Including COVID-19 data improves model performance.

On-policy methods outperform Q-learning during testing.

Model performance varies with market stability.

Abstract

This work seeks to answer key research questions regarding the viability of reinforcement learning over the S&P 500 index. The on-policy techniques of Value Iteration (VI) and State-action-reward-state-action (SARSA) are implemented along with the off-policy technique of Q-Learning. The models are trained and tested on a dataset comprising multiple years of stock market data from 2000-2023. The analysis presents the results and findings from training and testing the models using two different time periods: one including the COVID-19 pandemic years and one excluding them. The results indicate that including market data from the COVID-19 period in the training dataset leads to superior performance compared to the baseline strategies. During testing, the on-policy approaches (VI and SARSA) outperform Q-learning, highlighting the influence of bias-variance tradeoff and the generalization capabilities of simpler policies. However, it is noted that the performance of Q-learning may vary depending on the stability of future market conditions. Future work is suggested, including experiments with updated Q-learning policies during testing and trading diverse individual stocks. Additionally, the exploration of alternative economic indicators for training the models is proposed.