MentorCollab: Selective Large-to-Small Inference-Time Guidance for Efficient Reasoning

TL;DR

MentorCollab enables small language models to benefit from large models' reasoning abilities through selective, inference-time guidance, improving performance with minimal additional computational cost.

Contribution

This paper introduces MentorCollab, a novel method for sparse, inference-time guidance from large to small models, reducing costs while maintaining reasoning quality.

Findings

Improves small model reasoning performance by up to 8%.

Reduces mentor model token usage to 18.4%.

Effective with short segments and selective probing.

Abstract

Large reasoning models (LRMs) achieve strong performance by producing long chains of thought, but their inference costs are high and often generate redundant reasoning. Small language models (SLMs) are far more efficient, yet struggle on multi-step reasoning tasks. A natural idea is to let a large model guide a small one at inference time as a mentor, yet existing collaboration methods often promote imitation, resulting in verbose reasoning without consistent error correction. We propose MentorCollab, an inference-time collaboration method in which an LRM selectively and sparsely guides an SLM, rather than taking over generation. At randomly sampled token positions, we probe for divergences between the two models and use a lightweight verifier to decide whether the SLM should follow a short lookahead segment from its mentor or continue on its own. Across 15 SLM--LRM pairs and 3 domains (math reasoning, general knowledge, and commonsense reasoning), our method improves performance in 12 settings, with average gains of 3.0% and up to 8.0%, while adopting only having 18.4% tokens generated by the expensive mentor model on average. We find that short segments and selective probing are sufficient for effective collaboration. Our results show that selective inference-time guidance restores large-model reasoning ability without substantial inference overhead.