COSMOS: Predictable and Cost-Effective Adaptation of LLMs

TL;DR

COSMOS is a unified framework that accurately predicts LLM adaptation performance and costs, significantly reducing computational overhead and enabling cost-effective model deployment.

Contribution

We introduce COSMOS, a novel prediction framework that estimates adaptation outcomes for LLMs with minimal cost, combining lightweight proxies and scaling laws.

Findings

Achieves high prediction accuracy across benchmarks.

Reduces computational costs by up to 98.71%.

Maintains performance while saving resources.

Abstract

Large language models (LLMs) achieve remarkable performance across numerous tasks by using a diverse array of adaptation strategies. However, optimally selecting a model and adaptation strategy under resource constraints is challenging and often requires extensive experimentation. We investigate whether it is possible to accurately predict both performance and cost without expensive trials. We formalize the strategy selection problem for LLMs and introduce COSMOS, a unified prediction framework that efficiently estimates adaptation outcomes at minimal cost. We instantiate and study the capability of our framework via a pair of powerful predictors: embedding-augmented lightweight proxy models to predict fine-tuning performance, and low-sample scaling laws to forecast retrieval-augmented in-context learning. Extensive evaluation across eight representative benchmarks demonstrates that COSMOS achieves high prediction accuracy while reducing computational costs by 92.72% on average, and up to 98.71% in resource-intensive scenarios. Our results show that efficient prediction of adaptation outcomes is not only feasible but can substantially reduce the computational overhead of LLM deployment while maintaining performance standards.