Collie: Finding Performance Anomalies in RDMA Subsystems

TL;DR

Collie is a comprehensive tool that systematically uncovers performance anomalies in RDMA subsystems by exploring application workload space and using simulated annealing to identify extreme performance behaviors, aiding in production readiness.

Contribution

This paper introduces Collie, a novel holistic approach that uses simulated annealing to find RDMA performance anomalies without hardware access, revealing new issues acknowledged by vendors.

Findings

Discovered 15 new performance anomalies in RDMA systems.

7 anomalies were fixed after reporting; all were acknowledged by vendors.

Validated Collie's effectiveness on RDMA-based applications like RPC and machine learning.

Abstract

High-speed RDMA networks are getting rapidly adopted in the industry for their low latency and reduced CPU overheads. To verify that RDMA can be used in production, system administrators need to understand the set of application workloads that can potentially trigger abnormal performance behaviors (e.g., unexpected low throughput, PFC pause frame storm). We design and implement Collie, a tool for users to systematically uncover performance anomalies in RDMA subsystems without the need to access hardware internal designs. Instead of individually testing each hardware device (e.g., NIC, memory, PCIe), Collie is holistic, constructing a comprehensive search space for application workloads. Collie then uses simulated annealing to drive RDMA-related performance and diagnostic counters to extreme value regions to find workloads that can trigger performance anomalies. We evaluate Collie on combinations of various RDMA NIC, CPU, and other hardware components. Collie found 15 new performance anomalies. All of them are acknowledged by the hardware vendors. 7 of them are already fixed after we reported them. We also present our experience in using Collie to avoid performance anomalies for an RDMA RPC library and an RDMA distributed machine learning framework.