Nowcasting the Financial Time Series with Streaming Data Analytics under Apache Spark

TL;DR

This paper introduces a real-time, streaming data analytics approach using Apache Spark for nowcasting high-frequency financial data with a two-stage method involving chaos modeling and machine learning algorithms.

Contribution

It presents a novel two-stage methodology combining chaos modeling and machine learning for real-time financial nowcasting using streaming data analytics in Apache Spark.

Findings

Effective real-time nowcasting demonstrated on stock and cryptocurrency datasets.

Machine learning models showed improved accuracy with the proposed approach.

Statistical tests confirmed the significance of model comparisons.

Abstract

This paper proposes nowcasting of high-frequency financial datasets in real-time with a 5-minute interval using the streaming analytics feature of Apache Spark. The proposed 2 stage method consists of modelling chaos in the first stage and then using a sliding window approach for training with machine learning algorithms namely Lasso Regression, Ridge Regression, Generalised Linear Model, Gradient Boosting Tree and Random Forest available in the MLLib of Apache Spark in the second stage. For testing the effectiveness of the proposed methodology, 3 different datasets, of which two are stock markets namely National Stock Exchange & Bombay Stock Exchange, and finally One Bitcoin-INR conversion dataset. For evaluating the proposed methodology, we used metrics such as Symmetric Mean Absolute Percentage Error, Directional Symmetry, and Theil U Coefficient. We tested the significance of each pair of models using the Diebold Mariano (DM) test.