Optimal Budgeted Adaptation of Large Language Models

TL;DR

This paper introduces a budget-aware fine-tuning framework for large language models using a game-theoretic approach, optimizing label efficiency within a fixed supervision budget.

Contribution

It formulates LLM adaptation as a contextual Stackelberg game and develops algorithms with regret bounds that explicitly incorporate supervision budgets.

Findings

Achieves $ ilde{O}(drac{eta}{eta} ext{regret})$ regret in full-feedback setting.

Extends to a label-querying strategy with $ ilde{O}( ext{regret})$ bounds.

Provides a principled approach for budget-aware LLM fine-tuning.

Abstract

The trade-off between labeled data availability and downstream accuracy remains a central challenge in fine-tuning large language models (LLMs). We propose a principled framework for \emph{budget-aware supervised fine-tuning} by casting LLM adaptation as a contextual Stackelberg game. In our formulation, the learner (leader) commits to a scoring policy and a label-querying strategy, while an adaptive environment (follower) selects challenging supervised alternatives in response. To explicitly address label efficiency, we incorporate a finite supervision budget directly into the learning objective. Our algorithm operates in the full-feedback regime and achieves $\tilde{O}(d\sqrt{T})$ regret under standard linear contextual assumptions. We extend the framework with a Largest-Latency-First (LLF) confidence gate that selectively queries labels, achieving a budget-aware regret bound of $\tilde{O}(\sqrt{dB} + c\sqrt{B})$ with $B=\beta T$.