Formal Specification for Fast ACS: Low-Latency File-Based Ordered Message Delivery at Scale

TL;DR

Fast ACS is a low-latency, file-based ordered message delivery system designed for large-scale, geographically distributed clusters, ensuring efficient, guaranteed message delivery with minimal resource usage.

Contribution

The paper introduces Fast ACS, a novel file-based messaging system that combines inter- and intra-cluster communication primitives to achieve scalable, low-latency delivery at global scale.

Findings

Successfully deployed in dozens of production clusters

Scales to thousands of consumers with Tbps intra-cluster traffic

Delivers messages within seconds or sub-seconds (p99)

Abstract

Low-latency message delivery is crucial for real-time systems. Data originating from a producer must be delivered to consumers, potentially distributed in clusters across metropolitan and continental boundaries. With the growing scale of computing, there can be several thousand consumers of the data. Such systems require a robust messaging system capable of transmitting messages containing data across clusters and efficiently delivering them to consumers. The system must offer guarantees like ordering and at-least-once delivery while avoiding overload on consumers, allowing them to consume messages at their own pace. This paper presents the design of Fast ACS (an abbreviation for Ads Copy Service), a file-based ordered message delivery system that leverages a combination of two-sided (inter-cluster) and one-sided (intra-cluster) communication primitives - namely, Remote Procedure Call and Remote Memory Access, respectively - to deliver messages. The system has been successfully deployed to dozens of production clusters and scales to accommodate several thousand consumers within each cluster, which amounts to Tbps-scale intra-cluster consumer traffic at peak. Notably, Fast ACS delivers messages to consumers across the globe within a few seconds or even sub-seconds (p99) based on the message volume and consumer scale, at a low resource cost.