Active-Code Replacement in the OODIDA Data Analytics Platform

TL;DR

This paper introduces a method for dynamic, on-the-fly replacement of user-defined Python modules in the OODIDA data analytics platform, enabling seamless updates during ongoing tasks without system restarts.

Contribution

It presents a novel approach to replace Python modules dynamically in a distributed analytics system, enhancing flexibility and reducing disruption during updates.

Findings

Enables module replacement without system restart

Supports iterative A/B testing of algorithms

Facilitates real-time modifications of analytics modules

Abstract

OODIDA (On-board/Off-board Distributed Data Analytics) is a platform for distributing and executing concurrent data analytics tasks. It targets fleets of reference vehicles in the automotive industry and has a particular focus on rapid prototyping. Its underlying message-passing infrastructure has been implemented in Erlang/OTP. External Python applications perform data analytics tasks. Most work is performed by clients (on-board). A central cloud server performs supplementary tasks (off-board). OODIDA can be automatically packaged and deployed, which necessitates restarting parts of the system, or all of it. This is potentially disruptive. To address this issue, we added the ability to execute user-defined Python modules on clients as well as the server. These modules can be replaced without restarting any part of the system and they can even be replaced between iterations of an ongoing assignment. This facilitates use cases such as iterative A/B testing of machine learning algorithms or modifying experimental algorithms on-the-fly.