Interference-free Operating System: A 6 Years' Experience in Mitigating Cross-Core Interference in Linux

TL;DR

This paper details a 6-year industry effort to mitigate cross-core interference in Linux, significantly improving real-time performance and system predictability by fixing numerous kernel issues.

Contribution

It systematically addresses Linux kernel interference problems, providing practical fixes and lessons learned that enhance real-time system performance and are now integrated into mainstream Linux.

Findings

Worst-case jitter reduced by 8.7 times

Maximum 11.5 times improvement in system schedulability

Latency reductions of 1.6x and 1.64x for specific systems

Abstract

Real-time operating systems employ spatial and temporal isolation to guarantee predictability and schedulability of real-time systems on multi-core processors. Any unbounded and uncontrolled cross-core performance interference poses a significant threat to system time safety. However, the current Linux kernel has a number of interference issues and represents a primary source of interference. Unfortunately, existing research does not systematically and deeply explore the cross-core performance interference issue within the OS itself. This paper presents our industry practice for mitigating cross-core performance interference in Linux over the past 6 years. We have fixed dozens of interference issues in different Linux subsystems. Compared to the version without our improvements, our enhancements reduce the worst-case jitter by a factor of 8.7, resulting in a maximum 11.5x improvement over system schedulability. For the worst-case latency in the Core Flight System and the Robot Operating System 2, we achieve a 1.6x and 1.64x reduction over RT-Linux. Based on our development experience, we summarize the lessons we learned and offer our suggestions to system developers for systematically eliminating cross-core interference from the following aspects: task management, resource management, and concurrency management. Most of our modifications have been merged into Linux upstream and released in commercial distributions.