SynapseRoute: An Auto-Route Switching Framework on Dual-State Large Language Model

TL;DR

This paper introduces SynapseRoute, a dynamic routing framework for dual-mode large language models that intelligently assigns queries to either high-cost reasoning or low-cost non-thinking modes, optimizing accuracy and efficiency.

Contribution

It proposes a machine learning-based routing method that effectively balances accuracy and operational cost by classifying queries for appropriate mode selection.

Findings

Non-thinking mode can answer 58% of medical questions accurately.

SynapseRoute improves accuracy from 0.8272 to 0.8390.

It reduces inference time by 36.8% and token consumption by 39.66%.

Abstract

With the widespread adoption of large language models (LLMs) in practical applications, selecting an appropriate model requires balancing not only performance but also operational cost. The emergence of reasoning-capable models has further widened the cost gap between "thinking" (high reasoning) and "non-thinking" (fast, low-cost) modes. In this work, we reveal that approximately 58% of medical questions can be accurately answered by the non-thinking mode alone, without requiring the high-cost reasoning process. This highlights a clear dichotomy in problem complexity and suggests that dynamically routing queries to the appropriate mode based on complexity could optimize accuracy, cost-efficiency, and overall user experience. Based on this, we further propose SynapseRoute, a machine learning-based dynamic routing framework that intelligently assigns input queries to either thinking or non-thinking modes. Experimental results on several medical datasets demonstrate that SynapseRoute not only improves overall accuracy (0.8390 vs. 0.8272) compared to the thinking mode alone but also reduces inference time by 36.8% and token consumption by 39.66%. Importantly, qualitative analysis indicates that over-reasoning on simpler queries can lead to unnecessary delays and even decreased accuracy, a pitfall avoided by our adaptive routing. Finally, this work further introduces the Accuracy-Inference-Token (AIT) index to comprehensively evaluate the trade-offs among accuracy, latency, and token cost.