Thallus: An RDMA-based Columnar Data Transport Protocol
Jayjeet Chakraborty, Matthieu Dorier, Philip Carns, Robert Ross,, Carlos Maltzahn, Heiner Litz

TL;DR
Thallus introduces an RDMA-based data transport protocol for Apache Arrow, significantly reducing serialization overhead and improving performance in distributed data processing systems.
Contribution
The paper presents Thallus, a novel RDMA-based protocol for columnar data transport over Infiniband, enhancing efficiency compared to traditional RPC methods.
Findings
Substantial performance improvements with RDMA transport
Reduced serialization overhead in data transfer
Effective implementation using Thallium framework
Abstract
The volume of data generated and stored in contemporary global data centers is experiencing exponential growth. This rapid data growth necessitates efficient processing and analysis to extract valuable business insights. In distributed data processing systems, data undergoes exchanges between the compute servers that contribute significantly to the total data processing duration in adequately large clusters, necessitating efficient data transport protocols. Traditionally, data transport frameworks such as JDBC and ODBC have used TCP/IP-over-Ethernet as their underlying network protocol. Such frameworks require serializing the data into a single contiguous buffer before handing it off to the network card, primarily due to the requirement of contiguous data in TCP/IP. In OLAP use cases, this serialization process is costly for columnar data batches as it involves numerous memory copies…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAlgorithms and Data Compression · Advanced Data Storage Technologies · DNA and Biological Computing
