Confidence-Driven Multi-Scale Model Selection for Cost-Efficient Inference

TL;DR

This paper introduces a confidence-driven multi-scale model selection method that dynamically chooses models based on confidence estimates to maintain accuracy while significantly reducing computational costs in large language model inference.

Contribution

It presents a novel confidence-based approach for selecting models during inference, improving cost efficiency without sacrificing accuracy.

Findings

Achieves accuracy comparable to the largest models with 20-40% less computation.

Reduces token usage by approximately 60% in GPT-4o API calls.

Demonstrates effectiveness in resource-constrained settings like edge devices.

Abstract

Large Language Models (LLMs) have revolutionized inference across diverse natural language tasks, with larger models performing better but at higher computational costs. We propose a confidence-driven strategy that dynamically selects the most suitable model based on confidence estimates. By assessing a model's confidence in handling the task and response accuracy, tasks that are likely to be solved correctly are retained, while more uncertain or complex cases are delegated to a larger model, ensuring reliability while minimizing computation. Specifically, we evaluate a model's likelihood of knowing the correct answer and the probability that its response is accurate. Experiments on the Massive Multitask Language Understanding (MMLU) benchmark show that our approach achieves accuracy comparable to the largest model while reducing computational costs by 20\% to 40\%. When applied to GPT-4o API calls, it reduces token usage by approximately 60\%, further improving cost efficiency. These findings indicate the potential of confidence-based model selection to enhance real-world LLM deployment, particularly in resource-constrained settings such as edge devices and commercial API applications.