ExecTune: Effective Steering of Black-Box LLMs with Guide Models

TL;DR

This paper introduces ExecTune, a training method for Guide-Core Policies that enhances the performance and efficiency of black-box LLM systems by optimizing guide strategies for better execution and cost savings.

Contribution

It proposes ExecTune, a novel training recipe combining multiple techniques to improve guide strategies in GCoP, leading to better accuracy and lower inference costs.

Findings

GCoP with ExecTune improves accuracy by up to 9.2% over prior methods.

Inference costs are reduced by up to 22.4% with ExecTune.

ExecTune enables models like Claude Haiku 3.5 to outperform competitors on math and code tasks.

Abstract

For large language models deployed through black-box APIs, recurring inference costs often exceed one-time training costs. This motivates composed agentic systems that amortize expensive reasoning into reusable intermediate representations. We study a broad class of such systems, termed Guide-Core Policies (GCoP), in which a guide model generates a structured strategy that is executed by a black-box core model. This abstraction subsumes base, supervised, and advisor-style approaches, which differ primarily in how the guide is trained. We formalize GCoP under a cost-sensitive utility objective and show that end-to-end performance is governed by guide-averaged executability: the probability that a strategy generated by the guide can be faithfully executed by the core. Our analysis shows that existing GCoP instantiations often fail to optimize executability under deployment constraints, resulting in brittle strategies and inefficient computation. Motivated by these insights, we propose ExecTune, a principled training recipe that combines teacher-guided acceptance sampling, supervised fine-tuning, and structure-aware reinforcement learning to directly optimize syntactic validity, execution success, and cost efficiency. Across mathematical reasoning and code-generation benchmarks, GCoP with ExecTune improves accuracy by up to 9.2% over prior state-of-the-art baselines while reducing inference cost by up to 22.4%. It enables Claude Haiku 3.5 to outperform Sonnet 3.5 on both math and code tasks, and to come within 1.7% absolute accuracy of Sonnet 4 at 38% lower cost. Beyond efficiency, GCoP also supports modular adaptation by updating the guide without retraining the core.