LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection

TL;DR

This paper introduces LENSLLM, a theoretical and neural tangent kernel-based framework for efficient large language model selection, accurately predicting performance and reducing computational costs across diverse tasks.

Contribution

We develop a novel theoretical framework and NTK-based model for understanding LLM fine-tuning dynamics, enabling efficient and accurate model selection.

Findings

Achieves up to 91.1% accuracy in LLM selection

Reduces computational cost by up to 88.5%

Outperforms 5 state-of-the-art methods

Abstract

The proliferation of open-sourced Large Language Models (LLMs) and diverse downstream tasks necessitates efficient model selection, given the impracticality of fine-tuning all candidates due to computational constraints. Despite the recent advances in LLM selection, a fundamental research question largely remains nascent: how can we model the dynamic behaviors of LLMs during fine-tuning, thereby enhancing our understanding of their generalization performance across diverse downstream tasks? In this work, we propose a novel theoretical framework that provides a proper lens to assess the generalization capabilities of LLMs, thereby enabling accurate and efficient LLM selection for downstream applications. In particular, we first derive a PAC-Bayesian Generalization Bound that unveils fine-tuning dynamics of LLMs and then introduce LENSLLM, a Neural Tangent Kernel (NTK)-based Rectified Scaling Model that enables accurate performance predictions across diverse tasks while maintaining computational efficiency. Extensive empirical results on 3 large-scale benchmarks demonstrate that our model achieves up to 91.1% accuracy and reduces up to 88.5% computational cost in LLM selection, outperforming 5 state-of-the-art methods. We open-source our proposed LENSLLM model and corresponding results at LensLLM.io.