TweakLLM: A Routing Architecture for Dynamic Tailoring of Cached Responses

TL;DR

TweakLLM is a routing architecture that dynamically adapts cached responses in large language models, improving efficiency and maintaining response quality in high-volume, personalized chatbot interactions.

Contribution

It introduces a lightweight LLM-based routing system that enhances cache relevance and effectiveness for LLMs, addressing personalization challenges in response caching.

Findings

Maintains response quality comparable to frontier models

Significantly improves cache effectiveness

Scalable and resource-efficient for high-volume deployments

Abstract

Large Language Models (LLMs) process millions of queries daily, making efficient response caching a compelling optimization for reducing cost and latency. However, preserving relevance to user queries using this approach proves difficult due to the personalized nature of chatbot interactions and the limited accuracy of semantic similarity search. To address this, we present TweakLLM, a novel routing architecture that employs a lightweight LLM to dynamically adapt cached responses to incoming prompts. Through comprehensive evaluation, including user studies with side-by-side comparisons, satisfaction voting, as well as multi-agent LLM debates, we demonstrate that TweakLLM maintains response quality comparable to frontier models while significantly improving cache effectiveness. Our results across real-world datasets highlight TweakLLM as a scalable, resource-efficient caching solution for high-volume LLM deployments without compromising user experience.