Characterizing Off-path SmartNIC for Accelerating Distributed Systems

TL;DR

This paper provides a comprehensive characterization of off-path SmartNICs, specifically Bluefield-2, analyzing communication performance and proposing multi-path utilization to optimize distributed systems, demonstrated through case studies with notable performance gains.

Contribution

It offers the first holistic communication-path analysis of off-path SmartNICs and introduces a multi-path approach with guidelines for optimizing distributed system performance.

Findings

Communication among client, SoC, and host varies significantly.

Using multiple communication paths yields up to 30% performance improvement.

Guidelines for leveraging multi-path communication in distributed systems.

Abstract

SmartNICs have recently emerged as an appealing device for accelerating distributed systems. However, there has not been a comprehensive characterization of SmartNICs, and existing designs typically only leverage a single communication path for workload offloading. This paper presents the first holistic study of a representative off-path SmartNIC, specifically the Bluefield-2, from a communication-path perspective. Our experimental study systematically explores the key performance characteristics of communication among the client, on-board SoC, and host, and offers insightful findings and advice for designers. Moreover, we propose the concurrent use of multiple communication paths of a SmartNIC and present a pioneering guideline to expose new optimization opportunities for various distributed systems. To demonstrate the effectiveness of our approach, we conducted case studies on a SmartNIC-based distributed file system (LineFS) and an RDMA-based disaggregated key-value store (DrTM-KV). Our experimental results show improvements of up to 30% and 25% for LineFS and DrTM-KV, respectively.