Using real-time cluster configurations of streaming asynchronous features as online state descriptors in financial markets

TL;DR

This paper introduces an online, unsupervised method for discovering and detecting states in high-frequency financial market data using real-time cluster configurations of asynchronous features, enabling autonomous system understanding.

Contribution

It proposes a novel real-time clustering approach that automatically identifies system states from streaming financial data without human pre-processing.

Findings

Effective online state detection in streaming data

Automatic enumeration of system states based on feature configurations

Elimination of manual data pre-processing for state attributes

Abstract

We present a scheme for online, unsupervised state discovery and detection from streaming, multi-featured, asynchronous data in high-frequency financial markets. Online feature correlations are computed using an unbiased, lossless Fourier estimator. A high-speed maximum likelihood clustering algorithm is then used to find the feature cluster configuration which best explains the structure in the correlation matrix. We conjecture that this feature configuration is a candidate descriptor for the temporal state of the system. Using a simple cluster configuration similarity metric, we are able to enumerate the state space based on prevailing feature configurations. The proposed state representation removes the need for human-driven data pre-processing for state attribute specification, allowing a learning agent to find structure in streaming data, discern changes in the system, enumerate its perceived state space and learn suitable action-selection policies.