Predicting S&P500 Index direction with Transfer Learning and a Causal Graph as main Input

TL;DR

This paper introduces a multi-task learning framework combining transfer learning, causal graphs, and variational autoencoders to predict S&P500 index movements, outperforming benchmarks over 12 years.

Contribution

It presents a novel unified approach integrating transfer learning, multidisciplinary financial theories, and unstructured data within a causal graph framework for market prediction.

Findings

Achieved 74.3% accuracy in predicting index direction.

Outperformed industry benchmarks on a 12-year test period.

Effectively modeled complex market dynamics with a causal graph.

Abstract

We propose a unified multi-tasking framework to represent the complex and uncertain causal process of financial market dynamics, and then to predict the movement of any type of index with an application on the monthly direction of the S&P500 index. our solution is based on three main pillars: (i) the use of transfer learning to share knowledge and feature (representation, learning) between all financial markets, increase the size of the training sample and preserve the stability between training, validation and test sample. (ii) The combination of multidisciplinary knowledge (Financial economics, behavioral finance, market microstructure and portfolio construction theories) to represent a global top-down dynamics of any financial market, through a graph. (iii) The integration of forward looking unstructured data, different types of contexts (long, medium and short term) through latent variables/nodes and then, use a unique VAE network (parameter sharing) to learn simultaneously their distributional representation. We obtain Accuracy, F1-score, and Matthew Correlation of 74.3 %, 67 % and 0.42 above the industry and other benchmark on 12 years test period which include three unstable and difficult sub-period to predict.