Architecture Decisions in AI-based Systems Development: An Empirical Study

TL;DR

This empirical study analyzes architecture decisions in AI-based systems, revealing common patterns, decision types, application domains, quality attributes, and challenges faced by practitioners, highlighting the technical nature of these challenges.

Contribution

It provides the first comprehensive empirical analysis of architecture decision-making in AI-based systems, based on data from Stack Overflow and GitHub.

Findings

Architecture decisions are expressed in six linguistic patterns.

Main decision types include Technology, Component, and Data Decisions.

Performance is the most considered quality attribute.

Abstract

Artificial Intelligence (AI) technologies have been developed rapidly, and AI-based systems have been widely used in various application domains with opportunities and challenges. However, little is known about the architecture decisions made in AI-based systems development, which has a substantial impact on the success and sustainability of these systems. To this end, we conducted an empirical study by collecting and analyzing the data from Stack Overflow (SO) and GitHub. More specifically, we searched on SO with six sets of keywords and explored 32 AI-based projects on GitHub, and finally we collected 174 posts and 128 GitHub issues related to architecture decisions. The results show that in AI-based systems development (1) architecture decisions are expressed in six linguistic patterns, among which Solution Proposal and Information Giving are most frequently used, (2) Technology Decision, Component Decision, and Data Decision are the main types of architecture decisions made, (3) Game is the most common application domain among the eighteen application domains identified, (4) the dominant quality attribute considered in architecture decision-making is Performance, and (5) the main limitations and challenges encountered by practitioners in making architecture decisions are Design Issues and Data Issues. Our results suggest that the limitations and challenges when making architecture decisions in AI-based systems development are highly specific to the characteristics of AI-based systems and are mainly of technical nature, which need to be properly confronted.