RubricRefine: Improving Tool-Use Agent Reliability with Training-Free Pre-Execution Refinement

TL;DR

RubricRefine is a training-free pre-execution method that improves tool-use agent reliability by verifying and repairing semantic contracts before execution, reducing errors and latency.

Contribution

It introduces a novel, training-free approach for semantic contract verification and repair that enhances tool-use agent reliability without additional training.

Findings

Achieves 0.86 accuracy on M3ToolEval without execution attempts

Up to 2.6x lower latency compared to prior baselines

Performance is consistent across multiple models and tasks

Abstract

Iterative self-refinement is a popular inference-time reliability technique, but its effectiveness in code-mode tool use depends heavily on the structure of the feedback signal: unstructured critique helps inconsistently across models, and even revision with real execution feedback improves only modestly ($0.75$ vs. $0.65$ baseline). The dominant failures are inter-tool contract violations (wrong output shape, incorrect tool routing, broken argument provenance) that run to completion without raising errors, making runtime feedback insufficient. We introduce RubricRefine, a training-free method for pre-execution semantic contract verification that generates task- and registry-specific rubrics, scores candidate code against explicit contract checks, and iteratively repairs failures before any execution occurs. RubricRefine reaches $0.86$, averaged across seven models, on M3ToolEval with zero execution attempts, improving over prior inference-time baselines with up to $2.6\times$ lower latency. Performance remains flat on the predominantly single-step API-Bank, consistent with the method's reliance on inter-tool contract structure. A rubric-category ablation and calibration analysis further characterize when and why the method works.