Think When Needed: Model-Aware Reasoning Routing for LLM-based Ranking

TL;DR

This paper introduces a reasoning routing framework for LLM-based ranking that dynamically decides when to apply reasoning, improving ranking performance while reducing computational costs by using pre-generation signals and adaptive policy selection.

Contribution

It proposes a lightweight, plug-and-play router head that determines whether to reason or not before generation, optimizing the accuracy-efficiency trade-off in LLM ranking tasks.

Findings

Achieves +6.3% NDCG@10 on MovieLens with fewer tokens

Reduces token consumption by 49.5% while maintaining performance

Demonstrates effectiveness across multiple datasets and models

Abstract

Large language models (LLMs) are increasingly applied to ranking tasks in retrieval and recommendation. Although reasoning prompting can enhance ranking utility, our preliminary exploration reveals that its benefits are inconsistent and come at a substantial computational cost, suggesting that when to reason is as crucial as how to reason. To address this issue, we propose a reasoning routing framework that employs a lightweight, plug-and-play router head to decide whether to use direct inference (Non-Think) or reasoning (Think) for each instance before generation. The router head relies solely on pre-generation signals: i) compact ranking-aware features (e.g., candidate dispersion) and ii) model-aware difficulty signals derived from a diagnostic checklist reflecting the model's estimated need for reasoning. By leveraging these features before generation, the router outputs a controllable token that determines whether to apply the Think mode. Furthermore, the router can adaptively select its operating policy along the validation Pareto frontier during deployment, enabling dynamic allocation of computational resources toward instances most likely to benefit from Think under varying system constraints. Experiments on three public ranking datasets with different scales of open-source LLMs show consistent improvements in ranking utility with reduced token consumption (e.g., +6.3\% NDCG@10 with -49.5\% tokens on MovieLens with Qwen3-4B), demonstrating reasoning routing as a practical solution to the accuracy-efficiency trade-off.