Poster: Benchmarking Financial Data Feed Systems

TL;DR

This paper presents a benchmarking framework for evaluating open source event-processing platforms tailored to high-volume financial data feeds, aiming to improve system performance and reliability in investment industry applications.

Contribution

It introduces an enhanced open source benchmarking tool, 'wrench', tailored for financial data feed workloads, and provides detailed workload specifications for platform evaluation.

Findings

Benchmarking of Kafka, NATS, Redis, Flink, Storm with real financial data workloads.

Enhanced open source tool 'wrench' for simulating and replaying financial data feeds.

Open source release of the benchmarking tool for community use.

Abstract

Data-driven solutions for the investment industry require event-based backend systems to process high-volume financial data feeds with low latency, high throughput, and guaranteed delivery modes. At vwd we process an average of 18 billion incoming event notifications from 500+ data sources for 30 million symbols per day and peak rates of 1+ million notifications per second using custom-built platforms that keep audit logs of every event. We currently assess modern open source event-processing platforms such as Kafka, NATS, Redis, Flink or Storm for the use in our ticker plant to reduce the maintenance effort for cross-cutting concerns and leverage hybrid deployment models. For comparability and repeatability we benchmark candidates with a standardized workload we derived from our real data feeds. We have enhanced an existing light-weight open source benchmarking tool in its processing, logging, and reporting capabilities to cope with our workloads. The resulting tool wrench can simulate workloads or replay snapshots in volume and dynamics like those we process in our ticker plant. We provide the tool as open source. As part of ongoing work we contribute details on (a) our workload and requirements for benchmarking candidate platforms for financial feed processing; (b) the current state of the tool wrench.