SOMA: Efficient Multi-turn LLM Serving via Small Language Model

TL;DR

SOMA introduces a framework that uses small language models with learned prompts to efficiently serve multi-turn LLM conversations, reducing costs while maintaining response quality.

Contribution

It proposes a novel method to adapt small models to local dialogue regions using soft prompts and LoRA, balancing efficiency and coherence in multi-turn LLM serving.

Findings

SOMA achieves significant cost reduction in multi-turn dialogue serving.

The method maintains high response quality comparable to larger models.

Extensive experiments validate the effectiveness of SOMA.

Abstract

Large Language Models (LLMs) are increasingly deployed in multi-turn dialogue settings where preserving conversational context across turns is essential. A standard serving practice concatenates the full dialogue history at every turn, which reliably maintains coherence but incurs substantial cost in latency, memory, and API expenditure, especially when queries are routed to large proprietary models. Existing approaches often struggle to balance the trade-off between response quality and efficiency. We propose a framework that exploits the early turns of a session to estimate a local response manifold and then adapt a smaller surrogate model to this local region for the remainder of the conversation. Concretely, we learn soft prompts that maximize semantic divergence between the large and surrogate small language models' responses to surface least-aligned local directions, stabilize training with anti-degeneration control, and distill the mined cases into localized LoRA fine-tuning so the surrogate runs without prompts at inference. A simple gate enables a one-time switch with rollback on drift. We further provide a theoretical analysis for key components in SOMA. Extensive experiments show the effectiveness of SOMA. The source code is provided at: https://github.com/LabRAI/SOMA.