ros2_tracing: Multipurpose Low-Overhead Framework for Real-Time Tracing of ROS 2

TL;DR

ros2_tracing is a versatile, low-overhead framework for real-time tracing of ROS 2 systems, enabling detailed performance analysis without significantly impacting system latency.

Contribution

It introduces a multipurpose, low-overhead tracing framework for ROS 2 that integrates with existing tools and supports real-time distributed system analysis.

Findings

End-to-end message latency overhead is approximately 0.0033 ms.

Enabling all instrumentation minimally impacts real-time performance.

Trace data can be combined with OS traces for comprehensive analysis.

Abstract

Testing and debugging have become major obstacles for robot software development, because of high system complexity and dynamic environments. Standard, middleware-based data recording does not provide sufficient information on internal computation and performance bottlenecks. Other existing methods also target very specific problems and thus cannot be used for multipurpose analysis. Moreover, they are not suitable for real-time applications. In this paper, we present ros2_tracing, a collection of flexible tracing tools and multipurpose instrumentation for ROS 2. It allows collecting runtime execution information on real-time distributed systems, using the low-overhead LTTng tracer. Tools also integrate tracing into the invaluable ROS 2 orchestration system and other usability tools. A message latency experiment shows that the end-to-end message latency overhead, when enabling all ROS 2 instrumentation, is on average 0.0033 ms, which we believe is suitable for production real-time systems. ROS 2 execution information obtained using ros2_tracing can be combined with trace data from the operating system, enabling a wider range of precise analyses, that help understand an application execution, to find the cause of performance bottlenecks and other issues. The source code is available at: https://github.com/ros2/ros2_tracing.