SkewRoute: Training-Free LLM Routing for Knowledge Graph Retrieval-Augmented Generation via Score Skewness of Retrieved Context

TL;DR

SkewRoute is a training-free routing method for knowledge graph retrieval-augmented generation that uses score skewness to efficiently direct queries to appropriate-sized LLMs, significantly reducing inference costs.

Contribution

It introduces the first dedicated, training-free routing framework for KG-RAG based on score skewness, achieving over 3x higher effectiveness and minimal runtime overhead.

Findings

Over 3x higher routing effectiveness

Reduces runtime to less than 0.001x of existing methods

Effective in balancing performance and cost in KG-RAG

Abstract

Large language models excel at many tasks but often incur high inference costs during deployment. To mitigate hallucination, many systems use a knowledge graph to enhance retrieval-augmented generation (KG-RAG). However, the large amount of retrieved knowledge contexts increase these inference costs further. A promising solution to balance performance and cost is LLM routing, which directs simple queries to smaller LLMs and complex ones to larger LLMs. However, no dedicated routing methods currently exist for RAG, and existing training-based routers face challenges scaling to this domain due to the need for extensive training data. We observe that the score distributions produced by the retrieval scorer strongly correlate with query difficulty. Based on this, we propose an extremely simple yet effective routing framework, the first specifically designed for KG-RAG that efficiently balances performance and cost in a plug-and-play manner. It delivers over 3x higher routing effectiveness while reducing runtime to less than 0.001x compared to existing methods. Our code is available at https://github.com/hrwang00/SkewRoute.