Evaluating Representations with Readout Model Switching

TL;DR

This paper introduces a novel evaluation metric for deep learning representations based on the Minimum Description Length principle, employing a hybrid readout model switching strategy to assess data efficiency and model suitability.

Contribution

It proposes a unified, MDL-based evaluation method using hybrid readout models, improving consistency over accuracy-based metrics for representation quality assessment.

Findings

The MDL-based metric aligns with model scaling properties.

Hybrid readout models outperform single-capacity models in evaluation.

The method reveals insights into data efficiency and model preferences.

Abstract

Although much of the success of Deep Learning builds on learning good representations, a rigorous method to evaluate their quality is lacking. In this paper, we treat the evaluation of representations as a model selection problem and propose to use the Minimum Description Length (MDL) principle to devise an evaluation metric. Contrary to the established practice of limiting the capacity of the readout model, we design a hybrid discrete and continuous-valued model space for the readout models and employ a switching strategy to combine their predictions. The MDL score takes model complexity, as well as data efficiency into account. As a result, the most appropriate model for the specific task and representation will be chosen, making it a unified measure for comparison. The proposed metric can be efficiently computed with an online method and we present results for pre-trained vision encoders of various architectures (ResNet and ViT) and objective functions (supervised and self-supervised) on a range of downstream tasks. We compare our methods with accuracy-based approaches and show that the latter are inconsistent when multiple readout models are used. Finally, we discuss important properties revealed by our evaluations such as model scaling, preferred readout model, and data efficiency.