JAF: Judge Agent Forest
Sahil Garg, Brad Cheezum, Sridhar Dutta, Vishal Agarwal

TL;DR
JAF introduces a holistic judge agent framework that evaluates multiple query-response pairs simultaneously, enabling improved agent self-refinement through collective judgment and a novel LSH-based exemplar selection method.
Contribution
The paper presents JAF, a novel framework where judge agents perform joint inference across related responses, and develops an LSH algorithm for efficient, relation-aware exemplar selection.
Findings
JAF improves agent self-refinement through collective judgment.
The LSH-based method enhances exemplar diversity and relevance.
Empirical validation on cloud misconfiguration triage demonstrates effectiveness.
Abstract
Judge agents are fundamental to agentic AI frameworks: they provide automated evaluation, and enable iterative self-refinement of reasoning processes. We introduce JAF: Judge Agent Forest, a framework in which the judge agent conducts joint inference across a cohort of query--response pairs generated by a primary agent, rather than evaluating each in isolation. This paradigm elevates the judge from a local evaluator to a holistic learner: by simultaneously assessing related responses, the judge discerns cross-instance patterns and inconsistencies, whose aggregate feedback enables the primary agent to improve by viewing its own outputs through the judge's collective perspective. Conceptually, JAF bridges belief propagation and ensemble-learning principles: overlapping in-context neighborhoods induce a knowledge-graph structure that facilitates propagation of critique, and repeated,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Graph Neural Networks · Advanced Image and Video Retrieval Techniques · Big Data and Digital Economy
