On Optimal Caching and Model Multiplexing for Large Model Inference

TL;DR

This paper introduces an optimal approach combining caching and model multiplexing to significantly reduce inference costs and latency for large models, supported by theoretical analysis and empirical validation.

Contribution

It provides a theoretically optimal algorithm for joint caching and model multiplexing, and demonstrates substantial empirical improvements over baselines.

Findings

Up to 50x improvement over baselines in simulations.

4.3x reduction in FLOPs on real datasets.

1.8x decrease in latency with real models.

Abstract

Large Language Models (LLMs) and other large foundation models have achieved noteworthy success, but their size exacerbates existing resource consumption and latency challenges. In particular, the large-scale deployment of these models is hindered by the significant resource requirements during inference. In this paper, we study two approaches for mitigating these challenges: employing a cache to store previous queries and learning a model multiplexer to choose from an ensemble of models for query processing. Theoretically, we provide an optimal algorithm for jointly optimizing both approaches to reduce the inference cost in both offline and online tabular settings. By combining a caching algorithm, namely Greedy Dual Size with Frequency (GDSF) or Least Expected Cost (LEC), with a model multiplexer, we achieve optimal rates in both offline and online settings. Empirically, simulations show that the combination of our caching and model multiplexing algorithms greatly improves over the baselines, with up to $50\times$ improvement over the baseline when the ratio between the maximum cost and minimum cost is $100$. Experiments on real datasets show a $4.3\times$ improvement in FLOPs over the baseline when the ratio for FLOPs is $10$, and a $1.8\times$ improvement in latency when the ratio for average latency is $1.85$.