Thallus: An RDMA-based Columnar Data Transport Protocol

TL;DR

Thallus introduces an RDMA-based data transport protocol for Apache Arrow, significantly reducing serialization overhead and improving performance in distributed data processing systems.

Contribution

The paper presents Thallus, a novel RDMA-based protocol for columnar data transport over Infiniband, enhancing efficiency compared to traditional RPC methods.

Findings

Substantial performance improvements with RDMA transport

Reduced serialization overhead in data transfer

Effective implementation using Thallium framework

Abstract

The volume of data generated and stored in contemporary global data centers is experiencing exponential growth. This rapid data growth necessitates efficient processing and analysis to extract valuable business insights. In distributed data processing systems, data undergoes exchanges between the compute servers that contribute significantly to the total data processing duration in adequately large clusters, necessitating efficient data transport protocols. Traditionally, data transport frameworks such as JDBC and ODBC have used TCP/IP-over-Ethernet as their underlying network protocol. Such frameworks require serializing the data into a single contiguous buffer before handing it off to the network card, primarily due to the requirement of contiguous data in TCP/IP. In OLAP use cases, this serialization process is costly for columnar data batches as it involves numerous memory copies that hurt data transport duration and overall data processing performance. We study the serialization overhead in the context of a widely-used columnar data format, Apache Arrow, and propose leveraging RDMA to transport Arrow data over Infiniband in a zero-copy manner. We design and implement Thallus, an RDMA-based columnar data transport protocol for Apache Arrow based on the Thallium framework from the Mochi ecosystem, compare it with a purely Thallium RPC-based implementation, and show substantial performance improvements can be achieved by using RDMA for columnar data transport.