Can this Model Also Recognize Dogs? Zero-Shot Model Search from Weights

TL;DR

ProbeLog is a novel method for retrieving classification models capable of recognizing specific concepts like 'Dog' without needing metadata or training data, using probing and collaborative filtering to improve efficiency and scalability.

Contribution

The paper introduces ProbeLog, a new probing-based retrieval method that enables zero-shot and logit-based model search for specific concepts, reducing computational costs.

Findings

High retrieval accuracy demonstrated in real-world tasks

Supports both zero-shot and logit-based retrieval

Reduces encoding costs by 3x using collaborative filtering

Abstract

With the increasing numbers of publicly available models, there are probably pretrained, online models for most tasks users require. However, current model search methods are rudimentary, essentially a text-based search in the documentation, thus users cannot find the relevant models. This paper presents ProbeLog, a method for retrieving classification models that can recognize a target concept, such as "Dog", without access to model metadata or training data. Differently from previous probing methods, ProbeLog computes a descriptor for each output dimension (logit) of each model, by observing its responses on a fixed set of inputs (probes). Our method supports both logit-based retrieval ("find more logits like this") and zero-shot, text-based retrieval ("find all logits corresponding to dogs"). As probing-based representations require multiple costly feedforward passes through the model, we develop a method, based on collaborative filtering, that reduces the cost of encoding repositories by 3x. We demonstrate that ProbeLog achieves high retrieval accuracy, both in real-world and fine-grained search tasks and is scalable to full-size repositories.