Making use of supercomputers in financial machine learning

TL;DR

This study explores the use of supercomputers for financial machine learning, demonstrating that higher core counts can improve predictive accuracy but may have diminishing returns due to heuristic pruning, with potential for greater parallelization on less restrictive data.

Contribution

It shows how high-performance computing can enhance financial data exploration and prediction, highlighting the impact of heuristic thresholds on parallelization efficiency.

Findings

More explored rules improve prediction accuracy.

Using more than 40 cores yields limited time gains due to heuristics.

Potential for greater parallelization with less restrictive thresholds.

Abstract

This article is the result of a collaboration between Fujitsu and Advestis. This collaboration aims at refactoring and running an algorithm based on systematic exploration producing investment recommendations on a high-performance computer of the Fugaku, to see whether a very high number of cores could allow for a deeper exploration of the data compared to a cloud machine, hopefully resulting in better predictions. We found that an increase in the number of explored rules results in a net increase in the predictive performance of the final ruleset. Also, in the particular case of this study, we found that using more than around 40 cores does not bring a significant computation time gain. However, the origin of this limitation is explained by a threshold-based search heuristic used to prune the search space. We have evidence that for similar data sets with less restrictive thresholds, the number of cores actually used could very well be much higher, allowing parallelization to have a much greater effect.