A Framework for Assessing AI Agent Decisions and Outcomes in AutoML Pipelines

TL;DR

This paper introduces a decision-centric evaluation framework for AutoML agents, enabling detailed assessment of intermediate decisions to improve transparency, reliability, and interpretability of autonomous machine learning systems.

Contribution

It proposes the Evaluation Agent (EA) that assesses decision quality at multiple stages, addressing the lack of structured decision-level metrics in existing AutoML evaluation practices.

Findings

EA detects faulty decisions with 91.9% F1 score

EA identifies reasoning inconsistencies independent of outcomes

EA attributes performance changes to specific agent decisions

Abstract

Agent-based AutoML systems rely on large language models to make complex, multi-stage decisions across data processing, model selection, and evaluation. However, existing evaluation practices remain outcome-centric, focusing primarily on final task performance. Through a review of prior work, we find that none of the surveyed agentic AutoML systems report structured, decision-level evaluation metrics intended for post-hoc assessment of intermediate decision quality. To address this limitation, we propose an Evaluation Agent (EA) that performs decision-centric assessment of AutoML agents without interfering with their execution. The EA is designed as an observer that evaluates intermediate decisions along four dimensions: decision validity, reasoning consistency, model quality risks beyond accuracy, and counterfactual decision impact. Across four proof-of-concept experiments, we demonstrate that the EA can (i) detect faulty decisions with an F1 score of 0.919, (ii) identify reasoning inconsistencies independent of final outcomes, and (iii) attribute downstream performance changes to agent decisions, revealing impacts ranging from -4.9% to +8.3% in final metrics. These results illustrate how decision-centric evaluation exposes failure modes that are invisible to outcome-only metrics. Our work reframes the evaluation of agentic AutoML systems from an outcome-based perspective to one that audits agent decisions, offering a foundation for reliable, interpretable, and governable autonomous ML systems.