RelayLLM: Efficient Reasoning via Collaborative Decoding

TL;DR

RelayLLM introduces a token-level collaborative decoding framework that enables small language models to handle most reasoning tasks independently, invoking large models only for critical tokens, thereby significantly reducing computational costs while maintaining high accuracy.

Contribution

The paper presents a novel token-level collaborative decoding approach and a training framework that allows small language models to efficiently delegate critical reasoning tokens to large models, reducing costs.

Findings

Achieves 49.52% accuracy across six benchmarks.

Invokes LLM for only 1.07% of tokens on average.

Reduces computational cost by 98.2% compared to random routing.

Abstract

Large Language Models (LLMs) for complex reasoning is often hindered by high computational costs and latency, while resource-efficient Small Language Models (SLMs) typically lack the necessary reasoning capacity. Existing collaborative approaches, such as cascading or routing, operate at a coarse granularity by offloading entire queries to LLMs, resulting in significant computational waste when the SLM is capable of handling the majority of reasoning steps. To address this, we propose RelayLLM, a novel framework for efficient reasoning via token-level collaborative decoding. Unlike routers, RelayLLM empowers the SLM to act as an active controller that dynamically invokes the LLM only for critical tokens via a special command, effectively "relaying" the generation process. We introduce a two-stage training framework, including warm-up and Group Relative Policy Optimization (GRPO) to teach the model to balance independence with strategic help-seeking. Empirical results across six benchmarks demonstrate that RelayLLM achieves an average accuracy of 49.52%, effectively bridging the performance gap between the two models. Notably, this is achieved by invoking the LLM for only 1.07% of the total generated tokens, offering a 98.2% cost reduction compared to performance-matched random routers.