Engineering and Experimentally Benchmarking Open Source MQTT Broker Implementations

TL;DR

This paper benchmarks open-source MQTT broker implementations on edge devices, analyzing how hardware, network conditions, and message size affect performance, with a focus on reproducibility and open-source tools.

Contribution

It introduces an automated, reproducible benchmarking framework for MQTT brokers on diverse hardware platforms in edge computing environments.

Findings

Hardware and network conditions significantly impact broker performance.

Message size influences response times and overhead.

Open-source tools enable reproducible benchmarking.

Abstract

The Message Queuing Telemetry Transport (MQTT) protocol is one of the most widely used IoT protocol solutions. In this work, we are especially interested in open-source MQTT Broker implementations (such as Mosquitto, EMQX, RabbitMQ, VerneMQ, and HiveMQ). To this end, we engineer a network testbed to experimentally benchmark the performance of these implementations in an edge computing context with constrained devices. In more detail, we engineer an automated deployment and orchestration of the containerized MQTT broker implementations, with support for deployment across either moderately powerful AMD64 devices, or more resource constrained ARM64 devices. The proposed MQTT implementations are evaluated in terms of overhead response time and different payload sizes. Results showed that the hardware platform used as well as the message size, and the network parameters (latency, packet loss and jitter) have a significant impact on the performance differences between the brokers. All results, software tools and code are fully reproducible and free and open source.