When to Screen, When to Bypass: LLM-Judges in Resource-Scarce AI-Human Workflow

TL;DR

This paper models the decision of when to use LLM-based judges in AI-human workflows to optimize throughput and resource utilization, revealing conditions where screening is beneficial or detrimental.

Contribution

It introduces a queueing model for LLM-judged workflows, deriving optimal routing and capacity policies, and demonstrates asymptotic optimality and practical improvements over simple heuristics.

Findings

Optimal judge allocation depends on bottleneck resource.

Screening amplifies human capacity when reviewers are scarce.

The proposed policy outperforms always screen or never screen benchmarks.

Abstract

AI systems can generate outputs at scale, but most outputs require human approval before release. This creates a bottleneck: humans cannot keep pace with AI-generated volume. A natural response is to insert an LLM-judge that screens outputs before they reach humans, filtering errors and amplifying effective review capacity. But judges are imperfect. False rejections send correct outputs back for unnecessary rework; false acceptances consume judge capacity without relieving humans. When should outputs be routed through the judge, and when should they bypass it directly to human review? We model this workflow as a queueing network with three resource pools and use a fluid approximation to characterize optimal judge allocation. The analysis reveals that optimal allocation depends critically on which resource is the current bottleneck: screening amplifies human capacity when reviewers are scarce, yet generates a rework trap that crowds out new production when workers are stretched thin. For heterogeneous task classes with different error profiles, optimal priority can reverse across operating regimes, and classes with complementary error structures can be mixed to achieve throughput that neither class attains alone. We propose a policy that uses the fluid-optimal allocation fractions for routing and the fluid-optimal service levels for admission control, and establish its asymptotic optimality as system scale grows. Extensions incorporate human feedback that improves rework quality and joint capacity planning under budget constraints. Numerical experiments confirm rapid convergence to the fluid optimum and demonstrate that the policy significantly outperforms benchmarks that either always screen or never screen.