Prototype of Fault Adaptive Embedded Software for Large-Scale Real-Time Systems

TL;DR

This paper presents a prototype of fault adaptive embedded software for large-scale real-time systems, featuring self-optimizing agents and self-healing objects to enhance reliability and fault mitigation in complex environments.

Contribution

It introduces a distributed, self-adaptive fault mitigation approach with self-optimizing agents and self-healing objects for large-scale real-time embedded systems.

Findings

Successfully demonstrated proactive and reactive fault mitigation.

Enabled real-time detection and reaction to failures.

Supported large-scale deployment with 2500 DSPs.

Abstract

This paper describes a comprehensive prototype of large-scale fault adaptive embedded software developed for the proposed Fermilab BTeV high energy physics experiment. Lightweight self-optimizing agents embedded within Level 1 of the prototype are responsible for proactive and reactive monitoring and mitigation based on specified layers of competence. The agents are self-protecting, detecting cascading failures using a distributed approach. Adaptive, reconfigurable, and mobile objects for reliablility are designed to be self-configuring to adapt automatically to dynamically changing environments. These objects provide a self-healing layer with the ability to discover, diagnose, and react to discontinuities in real-time processing. A generic modeling environment was developed to facilitate design and implementation of hardware resource specifications, application data flow, and failure mitigation strategies. Level 1 of the planned BTeV trigger system alone will consist of 2500 DSPs, so the number of components and intractable fault scenarios involved make it impossible to design an `expert system' that applies traditional centralized mitigative strategies based on rules capturing every possible system state. Instead, a distributed reactive approach is implemented using the tools and methodologies developed by the Real-Time Embedded Systems group.