ModelLens: Finding the Best for Your Task from Myriads of Models

TL;DR

ModelLens is a unified framework that leverages scattered leaderboard data to recommend the best models for new datasets without expensive evaluations, improving model selection across diverse tasks.

Contribution

It introduces a performance-aware latent space learned from public leaderboard interactions, enabling effective model ranking without running models on target datasets.

Findings

Outperforms baselines relying on metadata or direct evaluation.

Improves model routing methods by up to 81%.

Generalizes well to text and vision-language tasks.

Abstract

The open-source model ecosystem now contains hundreds of thousands of pretrained models, yet picking the best model for a new dataset is increasingly infeasible: new models and unbenchmarked datasets emerge continuously, leaving practitioners with no prior records on either side. Existing approaches handle only fragments of this in-the-wild setting: AutoML and transferability estimation select models from small predefined pools or require expensive per-model forward passes on the target dataset, while model routing presupposes a given candidate pool. We introduce ModelLens, a unified framework for model recommendation in the wild. Our key insight is that public leaderboard interactions, though scattered and noisy, collectively trace out an implicit atlas of model capabilities across heterogeneous evaluation settings, a signal rich enough to learn from directly. By learning a performance-aware latent space over model--dataset--metric tuples, ModelLens ranks unseen models on unseen datasets without running candidates on the target dataset. On a new benchmark of 1.62M evaluation records spanning 47K models and 9.6K datasets, ModelLens surpasses baselines that either rely on metadata alone or require running each candidate on the target dataset. Its recommended Top-K pools further improve multiple representative routing methods by up to 81% across diverse QA benchmarks. Case studies on recently released benchmarks further confirm generalization to both text and vision-language tasks.