Cost-Saving LLM Cascades with Early Abstention

TL;DR

This paper explores early abstention in LLM cascades, allowing smaller models to abstain before expensive models are invoked, which reduces costs and errors while maintaining performance in risk-sensitive domains.

Contribution

It introduces and empirically evaluates the concept of early abstention in LLM cascades, demonstrating its benefits in cost reduction and error minimization across multiple benchmarks.

Findings

Early abstention reduces overall test loss by 2.2%.

It decreases costs by 13.0% and error rates by 5.0%.

Allows more effective use of abstention by leveraging error pattern correlations.

Abstract

LLM cascades deploy small LLMs to answer most queries, limiting the use of large and expensive LLMs to difficult queries. This approach can significantly reduce costs without impacting performance. However, risk-sensitive domains such as finance or medicine place an additional premium on avoiding model errors. Since even the most expensive models are susceptible to making mistakes, applications in these domains benefit from allowing LLM systems to completely abstain from answering difficult queries. Introducing abstention poses a design question for LLM cascades: should abstention only be allowed at the final model or also at earlier models? Since the error patterns of small and large models are correlated, allowing earlier models to abstain may reduce inference costs and latency by anticipating abstention decisions by expensive and slow models, thus avoiding the need to run these models. We investigate the benefits of such "early abstention" in LLM cascades and find that it reduces overall test loss by 2.2% on average across six benchmarks (GSM8K, MedMCQA, MMLU, TriviaQA, TruthfulQA, and XSum). These gains result from a more effective use of abstention, trading a 4.1% average increase in the overall abstention rate for a 13.0% reduction in cost and a 5.0% reduction in error rate. Our findings demonstrate the possibility of leveraging correlations between the error patterns of different language models to drive performance improvements for LLM systems with abstention.