Can We Predict Performance of Large Models across Vision-Language Tasks?

TL;DR

This paper introduces a probabilistic matrix factorization framework using MCMC to predict performance scores of large vision-language models across various tasks, reducing evaluation costs and estimating uncertainties.

Contribution

It proposes a novel matrix completion approach with enhancements for sparse data to accurately predict model performances and their uncertainties across tasks.

Findings

High accuracy in performance prediction

Reliable uncertainty estimates for model evaluation

Effective handling of sparse observed data

Abstract

Evaluating large vision-language models (LVLMs) is very expensive, due to high computational cost and the wide variety of tasks. The good news is that if we already have some observed performance scores, we may be able to infer unknown ones. In this study, we propose a new framework for predicting unknown performance scores based on observed ones from other LVLMs or tasks. We first formulate the performance prediction as a matrix completion task. Specifically, we construct a sparse performance matrix $\boldsymbol{R}$, where each entry $R_{mn}$ represents the performance score of the $m$-th model on the $n$-th dataset. By applying probabilistic matrix factorization (PMF) with Markov chain Monte Carlo (MCMC), we can complete the performance matrix, i.e., predict unknown scores. Additionally, we estimate the uncertainty of performance prediction based on MCMC. Practitioners can evaluate their models on untested tasks with higher uncertainty first, which quickly reduces the prediction errors. We further introduce several improvements to enhance PMF for scenarios with sparse observed performance scores. Our experiments demonstrate the accuracy of PMF in predicting unknown scores, the reliability of uncertainty estimates in ordering evaluations, and the effectiveness of our enhancements for handling sparse data.