LEAD: Exploring Logit Space Evolution for Model Selection

TL;DR

LEAD introduces a novel method for predicting model transferability by modeling the nonlinear evolution of logits during fine-tuning, enabling efficient model selection without extensive fine-tuning.

Contribution

The paper presents a theoretical framework using ODEs to model logit evolution, aligning better with fine-tuning dynamics and improving transferability prediction.

Findings

Effective model transferability prediction across diverse models and datasets.

Broad applicability demonstrated even in low-data scenarios.

Outperforms existing methods in experimental evaluations.

Abstract

The remarkable success of pretrain-then-finetune paradigm has led to a proliferation of available pre-trained models for vision tasks. This surge presents a significant challenge in efficiently choosing the most suitable pre-trained models for downstream tasks. The critical aspect of this challenge lies in effectively predicting the model transferability by considering the underlying fine-tuning dynamics. Existing methods often model fine-tuning dynamics in feature space with linear transformations, which do not precisely align with the fine-tuning objective and fail to grasp the essential nonlinearity from optimization. To this end, we present LEAD, a finetuning-aligned approach based on the network output of logits. LEAD proposes a theoretical framework to model the optimization process and derives an ordinary differential equation (ODE) to depict the nonlinear evolution toward the final logit state. Additionally, we design a class-aware decomposition method to consider the varying evolution dynamics across classes and further ensure practical applicability. Integrating the closely aligned optimization objective and nonlinear modeling capabilities derived from the differential equation, our method offers a concise solution to effectively bridge the optimization gap in a single step, bypassing the lengthy fine-tuning process. The comprehensive experiments on 24 supervised and self-supervised pre-trained models across 10 downstream datasets demonstrate impressive performances and showcase its broad adaptability even in low-data scenarios.