Real-time Adapting Routing (RAR): Improving Efficiency Through Continuous Learning in Software Powered by Layered Foundation Models

TL;DR

This paper introduces Real-time Adaptive Routing (RAR), a method that continuously learns to route requests efficiently among foundation models, reducing reliance on expensive models while maintaining high response quality.

Contribution

RAR is a novel approach that adapts routing decisions in real-time and uses guided in-context learning to improve weaker models' capabilities.

Findings

Routes 50.2% fewer requests to expensive models

Maintains 90.5% of response quality

Guided learning enhances weaker model performance

Abstract

To balance the quality and inference cost of a Foundation Model (FM, such as large language models (LLMs)) powered software, people often opt to train a routing model that routes requests to FMs with different sizes and capabilities. Existing routing models rely on learning the optimal routing decision from carefully curated data, require complex computations to be updated, and do not consider the potential evolution of weaker FMs. In this paper, we propose Real-time Adaptive Routing (RAR), an approach to continuously adapt FM routing decisions while using guided in-context learning to enhance the capabilities of weaker FM. The goal is to reduce reliance on stronger, more expensive FMs. We evaluate our approach on different subsets of the popular MMLU benchmark. Over time, our approach routes 50.2% fewer requests to computationally expensive models while maintaining around 90.5% of the general response quality. In addition, the guides generated from stronger models have shown intra-domain generalization and led to a better quality of responses compared to an equivalent approach with a standalone weaker FM.