Machine Learning in Space: Surveying the Robustness of on-board ML models to Radiation

TL;DR

This paper surveys the impact of space radiation on onboard machine learning models, highlighting gaps in current research, demonstrating limitations of existing tools, and providing initial experiments and resources to assess and improve model robustness in space environments.

Contribution

It offers a comprehensive analysis of the effects of radiation on space ML models, critiques existing tools, and provides initial experimental methods and resources for robustness assessment.

Findings

Not all radiation faults are as damaging as previously thought

Existing open-source tools have limitations for space ML fault analysis

Public resources enable further research on space-tolerant ML models

Abstract

Modern spacecraft are increasingly relying on machine learning (ML). However, physical equipment in space is subject to various natural hazards, such as radiation, which may inhibit the correct operation of computing devices. Despite plenty of evidence showing the damage that naturally-induced faults can cause to ML-related hardware, we observe that the effects of radiation on ML models for space applications are not well-studied. This is a problem: without understanding how ML models are affected by these natural phenomena, it is uncertain "where to start from" to develop radiation-tolerant ML software. As ML researchers, we attempt to tackle this dilemma. By partnering up with space-industry practitioners specialized in ML, we perform a reflective analysis of the state of the art. We provide factual evidence that prior work did not thoroughly examine the impact of natural hazards on ML models meant for spacecraft. Then, through a "negative result", we show that some existing open-source technologies can hardly be used by researchers to study the effects of radiation for some applications of ML in satellites. As a constructive step forward, we perform simple experiments showcasing how to leverage current frameworks to assess the robustness of practical ML models for cloud detection against radiation-induced faults. Our evaluation reveals that not all faults are as devastating as claimed by some prior work. By publicly releasing our resources, we provide a foothold -- usable by researchers without access to spacecraft -- for spearheading development of space-tolerant ML models.