OrchestraLLM: Efficient Orchestration of Language Models for Dialogue State Tracking

TL;DR

This paper introduces a novel routing framework that combines small and large language models to improve dialogue state tracking, significantly reducing computational costs while enhancing accuracy.

Contribution

The work presents a new SLM/LLM routing method that leverages exemplar pools and context similarity to improve efficiency and performance in dialogue state tracking.

Findings

Reduces computational costs by over 50%.

Enhances dialogue state tracking performance.

Effectively combines SLMs and LLMs for structured knowledge extraction.

Abstract

Large language models (LLMs) have revolutionized the landscape of Natural Language Processing systems, but are computationally expensive. To reduce the cost without sacrificing performance, previous studies have explored various approaches to harness the potential of Small Language Models (SLMs) as cost-effective alternatives to their larger counterparts. Driven by findings that SLMs and LLMs exhibit complementary strengths in a structured knowledge extraction task, this work presents a novel SLM/LLM routing framework designed to improve computational efficiency and enhance task performance. First, exemplar pools are created to represent the types of contexts where each LM provides a more reliable answer, leveraging a sentence embedding fine-tuned so that context similarity is close to dialogue state similarity. Then, during inference, the k-nearest exemplars to the testing instance are retrieved, and the instance is routed according to majority vote. In dialogue state tracking tasks, the proposed routing framework enhances performance substantially compared to relying solely on LLMs, while reducing the computational costs by over 50%.