Supporting architecture evaluation for ATAM scenarios with LLMs

TL;DR

This paper explores using large language models to automate parts of software architecture evaluation, aiming to improve efficiency and accuracy in analyzing quality scenarios and tradeoffs.

Contribution

It presents an initial study validating the potential of LLMs, specifically MS Copilot, to assist in architecture evaluation by analyzing quality scenarios.

Findings

LLM produced more accurate risk and tradeoff analysis than students.

Using LLMs can reduce manual effort in architecture evaluation.

Initial results show promise for AI-assisted architecture decision-making.

Abstract

Architecture evaluation methods have long been used to evaluate software designs. Several evaluation methods have been proposed and used to analyze tradeoffs between different quality attributes. Having competing qualities leads to conflicts for selecting which quality-attribute scenarios are the most suitable ones that an architecture should tackle and for prioritizing the scenarios required by the stakeholders. In this context, architecture evaluation is carried out manually, often involving long brainstorming sessions to decide which are the most adequate quality scenarios. To reduce this effort and make the assessment and selection of scenarios more efficient, we suggest the usage of LLMs to partially automate evaluation activities. As a first step to validate this hypothesis, this work studies MS Copilot as an LLM tool to analyze quality scenarios suggested by students in a software architecture course and compares the students' results with the assessment provided by the LLM. Our initial study reveals that the LLM produces in most cases better and more accurate results regarding the risks, sensitivity points and tradeoff analysis of the quality scenarios. Overall, the use of generative AI has the potential to partially automate and support the architecture evaluation tasks, improving the human decision-making process.