Toward Architecture-Aware Evaluation Metrics for LLM Agents

TL;DR

This paper introduces an architecture-aware evaluation framework for LLM agents, linking architectural components to observable behaviors and metrics, thereby improving the diagnostic and evaluative process.

Contribution

It presents a novel, lightweight approach that incorporates architectural insights into the evaluation of LLM agents, enhancing transparency and specificity.

Findings

Framework effectively links architecture to observable behaviors.

Application demonstrates improved evaluation clarity.

Enables targeted and transparent assessment of LLM agents.

Abstract

LLM-based agents are becoming central to software engineering tasks, yet evaluating them remains fragmented and largely model-centric. Existing studies overlook how architectural components, such as planners, memory, and tool routers, shape agent behavior, limiting diagnostic power. We propose a lightweight, architecture-informed approach that links agent components to their observable behaviors and to the metrics capable of evaluating them. Our method clarifies what to measure and why, and we illustrate its application through real world agents, enabling more targeted, transparent, and actionable evaluation of LLM-based agents.