Large Language Models for Analyzing Enterprise Architecture Debt in Unstructured Documentation

TL;DR

This paper presents an LLM-based approach to automatically detect and quantify Enterprise Architecture Smells in unstructured documentation, enhancing EA governance practices.

Contribution

It introduces a novel LLM-based prototype for identifying EA Smells in unstructured documents, evaluated through a case study and benchmark comparison.

Findings

LLMs can detect multiple EA Smells in unstructured text.

Benchmark models achieve higher precision and speed.

Fine-tuned models offer data protection benefits.

Abstract

Enterprise Architecture Debt (EA Debt) arises from suboptimal design decisions and misaligned components that can degrade an organization's IT landscape over time. Early indicators, Enterprise Architecture Smells (EA Smells), are currently mainly detected manually or only from structured artifacts, leaving much unstructured documentation under-analyzed. This study proposes an approach using a large language model (LLM) to identify and quantify EA Debt in unstructured architectural documentation. Following a design science research approach, we design and evaluate an LLM-based prototype for automated EA Smell detection. The artifact ingests unstructured documents (e.g., process descriptions, strategy papers), applies fine-tuned detection models, and outputs identified smells. We evaluate the prototype through a case study using synthetic yet realistic business documents, benchmarking against a custom GPT-based model. Results show that LLMs can detect multiple predefined EA Smells in unstructured text, with the benchmark model achieving higher precision and processing speed, and the fine-tuned on-premise model offering data protection advantages. The findings highlight opportunities for integrating LLM-based smell detection into EA governance practice.