The Impact of Environment Configurations on the Stability of AI-Enabled Systems

TL;DR

This study investigates how different environment configurations affect the stability of AI-enabled systems, revealing that environment changes often cause instability especially in processing time and cost, which can inform better deployment practices.

Contribution

The paper systematically analyzes the impact of environment variables on AI system stability using empirical experiments across multiple configurations and metrics.

Findings

Environment changes cause instability in AI systems across multiple metrics.

Instability is more frequent in processing time and expense than in model performance.

Identifying optimal environment configurations can improve system stability and reduce costs.

Abstract

Nowadays, software systems tend to include Artificial Intelligence (AI) components. Changes in the operational environment have been known to negatively impact the stability of AI-enabled software systems by causing unintended changes in behavior. However, how an environment configuration impacts the behavior of such systems has yet to be explored. Understanding and quantifying the degree of instability caused by different environment settings can help practitioners decide the best environment configuration for the most stable AI systems. To achieve this goal, we performed experiments with eight different combinations of three key environment variables (operating system, Python version, and CPU architecture) on $30$ open-source AI-enabled systems using the Travis CI platform. We determine the existence and the degree of instability introduced by each configuration using three metrics: the output of an AI component of the system (model performance), the time required to build and run the system (processing time), and the cost associated with building and running the system (expense). Our results indicate that changes in environment configurations lead to instability across all three metrics; however, it is observed more frequently with respect to processing time and expense rather than model performance. For example, between Linux and MacOS, instability is observed in 23\%, 96.67\%, and 100\% of the studied projects in model performance, processing time, and expense, respectively. Our findings underscore the importance of identifying the optimal combination of configuration settings to mitigate drops in model performance and reduce the processing time and expense before deploying an AI-enabled system.