Poster: A Real-World Distributed Infrastructure for Processing Financial Data at Scale

TL;DR

This paper presents a large-scale, distributed infrastructure for processing and delivering high-volume financial data and event notifications in real-time, emphasizing its architecture and operational insights.

Contribution

It introduces a practical, scalable distributed event-driven system for financial data processing, highlighting its architecture and global deployment.

Findings

Handles over 1 million event notifications per second

Processes 18 billion notifications daily from 500+ sources

Operates a geo-distributed publish/subscribe broker network

Abstract

Financial markets are event- and data-driven to an extremely high degree. For making decisions and triggering actions stakeholders require notifications about significant events and reliable background information that meet their individual requirements in terms of timeliness, accuracy, and completeness. As one of Europe's leading providers of financial data and regulatory solutions vwd processes an average of 18 billion event notifications from 500+ data sources for 30 million symbols per day. Our large-scale distributed event-based systems handle daily peak rates of 1+ million event notifications per second and additional load generated by singular pivotal events with global impact. In this poster we give practical insights into our IT systems. We outline the infrastructure we operate and the event-driven architecture we apply at vwd. In particular we showcase the (geo)distributed publish/subscribe broker network we operate across locations and countries to provide market data to our customers with varying quality of information (QoI) properties.