WorkflowPerturb: Calibrated Stress Tests for Evaluating Multi-Agent Workflow Metrics

TL;DR

WorkflowPerturb provides a calibrated benchmark with realistic perturbations to evaluate and interpret multi-agent workflow metrics, addressing calibration issues in automatic workflow evaluation.

Contribution

It introduces a large, controlled benchmark with realistic perturbations for studying and improving workflow evaluation metrics.

Findings

Metrics show systematic differences in sensitivity and calibration.

Severity-aware interpretation improves understanding of workflow degradation.

Benchmark enables more reliable evaluation of workflow metrics.

Abstract

LLM-based systems increasingly generate structured workflows for complex tasks. In practice, automatic evaluation of these workflows is difficult, because metric scores are often not calibrated, and score changes do not directly communicate the severity of workflow degradation. We introduce WorkflowPerturb, a controlled benchmark for studying workflow evaluation metrics. It works by applying realistic, controlled perturbations to golden workflows. WorkflowPerturb contains 4,973 golden workflows and 44,757 perturbed variants across three perturbation types (Missing Steps, Compressed Steps, and Description Changes), each applied at severity levels of 10%, 30%, and 50%. We benchmark multiple metric families and analyze their sensitivity and calibration using expected score trajectories and residuals. Our results characterize systematic differences across metric families and support severity-aware interpretation of workflow evaluation scores. Our dataset will be released upon acceptance.