TIER: Trajectory-Invariant Execution Rewards for Multi-Step Tool Composition

TL;DR

TIER introduces a new reward framework for multi-step tool use in language models, enabling scalable, flexible, and interpretable reinforcement learning without reliance on reference trajectories.

Contribution

It proposes trajectory-invariant rewards derived from function schemas and execution, supporting multiple solutions and improving multi-step compositional reasoning.

Findings

Achieves >90% accuracy on DepthBench across steps

Outperforms trajectory-supervised rewards beyond step-4

Demonstrates gains on BFCL v3 and NestFUL benchmarks

Abstract

Tool use enables large language models to solve complex tasks through sequences of API calls, yet existing reinforcement learning approaches fail to scale to multi-step composition settings. Outcome-based rewards provide only sparse feedback, while trajectory-supervised rewards depend on annotated reference solutions, penalizing valid alternatives and limiting scalability. We propose TIER: Trajectory-Invariant Execution Rewards, a reward framework that derives supervision directly from function schemas and runtime execution, rather than from reference trajectories. The reward decomposes into format validity, schema adherence, execution success, and answer correctness, providing dense, interpretable sequence-level feedback derived from fine-grained verification of individual steps of tool use. This design allows any valid execution path to receive credit, naturally supporting multiple solution strategies and adapting to evolving tool interfaces. On DepthBench, a compositional benchmark stratified by depth (1 to 6 steps), TIER achieves >90% accuracy across steps, where trajectory-supervised rewards collapse beyond step-4. We further demonstrate consistent gains on benchmarks like BFCL v3 and NestFUL. Ablation studies confirm that all reward components are necessary, highlighting the importance of multi-level supervision for compositional reasoning.