JudgeFlow: Agentic Workflow Optimization via Block Judge

TL;DR

JudgeFlow introduces a fine-grained, block-level evaluation and optimization framework for agentic workflows, significantly improving efficiency and interpretability in complex AI tasks like reasoning and code generation.

Contribution

It presents a novel pipeline with a dedicated Judge module that provides detailed diagnostics, enabling targeted improvements in agentic workflows.

Findings

Achieves superior performance on mathematical reasoning benchmarks.

Enhances sample efficiency in workflow optimization.

Provides scalable, interpretable diagnostics for complex tasks.

Abstract

Optimizing LLM-based agentic workflows is challenging for scaling AI capabilities. Current methods rely on coarse, end-to-end evaluation signals and lack fine-grained signals on where to refine, often resulting in inefficient or low-impact modifications. To address these limitations, we propose JudgeFlow, an Evaluation-Judge-Optimization-Update pipeline. We incorporate reusable, configurable logic blocks into agentic workflows to capture fundamental forms of logic. On top of this abstraction, we design a dedicated Judge module that inspects execution traces particularly failed runs and assigns rank-based responsibility scores to problematic blocks. These fine-grained diagnostic signals are then leveraged by an LLM-based optimizer, which focuses modifications on the most problematic block in the workflow. Our approach improves sample efficiency, enhances interpretability through block-level diagnostics, and provides a scalable foundation for automating increasingly complex agentic workflows. We evaluate JudgeFlow on mathematical reasoning and code generation benchmarks, where JudgeFlow achieves superior performance and efficiency compared to existing methods.