MemBoost: A Memory-Boosted Framework for Cost-Aware LLM Inference

TL;DR

MemBoost is a framework that reduces LLM inference costs by reusing answers and retrieving information, escalating only difficult queries to stronger models, suitable for interactive applications.

Contribution

It introduces a memory-boosted serving framework that enables cost-efficient, interactive LLM inference with answer reuse and selective escalation.

Findings

Significantly reduces large-model invocation costs

Maintains high answer quality comparable to strong models

Supports continual memory growth and cost-aware routing

Abstract

Large Language Models (LLMs) deliver strong performance but incur high inference cost in real-world services, especially under workloads with repeated or near-duplicate queries across users and sessions. In this work, we propose MemBoost, a memory-boosted LLM serving framework that enables a lightweight model to reuse previously generated answers and retrieve relevant supporting information for cheap inference, while selectively escalating difficult or uncertain queries to a stronger model. Unlike standard retrieval-augmented generation, which primarily grounds a single response, MemBoost is designed for interactive settings by supporting answer reuse, continual memory growth, and cost-aware routing. Experiments across multiple models under simulated workloads show that MemBoost substantially reduces expensive large-model invocations and overall inference cost, while maintaining high answer quality comparable to the strong model baseline.