GTR-Turbo: Merged Checkpoint is Secretly a Free Teacher for Agentic VLM Training

TL;DR

GTR-Turbo enhances vision-language model training by merging checkpoints to create a free, effective teacher, significantly improving accuracy and efficiency without relying on costly external models.

Contribution

It introduces a checkpoint merging technique that replaces expensive teacher models, enabling more practical and efficient RL training for vision-language agents.

Findings

Improves accuracy by 10-30% across tasks.

Reduces training time by 50%.

Cuts compute cost by 60%.

Abstract

Multi-turn reinforcement learning (RL) for multi-modal agents built upon vision-language models (VLMs) is hampered by sparse rewards and long-horizon credit assignment. Recent methods densify the reward by querying a teacher that provides step-level feedback, e.g., Guided Thought Reinforcement (GTR) and On-Policy Distillation, but rely on costly, often privileged models as the teacher, limiting practicality and reproducibility. We introduce GTR-Turbo, a highly efficient upgrade to GTR that matches its performance without training on or querying an expensive teacher model. Specifically, GTR-Turbo merges the weights of checkpoints produced during ongoing RL training and then uses the resulting merged model as a "free" teacher to guide subsequent RL via supervised fine-tuning or soft logit distillation. This design removes dependence on privileged VLMs (e.g., GPT or Gemini), mitigates the "entropy collapse" observed in prior work, and maintains stable training. Across diverse visual agentic tasks, GTR-Turbo improves the accuracy of the baseline model by 10-30% while reducing wall-clock training time by 50% and compute cost by 60% relative to GTR.