Manifold Characteristics That Predict Downstream Task Performance

TL;DR

This paper introduces a new framework and metric to analyze the geometry of representation manifolds in pretraining models, revealing how manifold smoothness correlates with downstream task performance.

Contribution

It proposes a novel metric, RMQM, for comparing representation manifolds and demonstrates its effectiveness in predicting downstream task success.

Findings

Self-supervised methods produce smoother representation manifolds.

Larger RMQM values correlate with better downstream performance.

The framework enables detailed comparison of pretraining methods.

Abstract

Pretraining methods are typically compared by evaluating the accuracy of linear classifiers, transfer learning performance, or visually inspecting the representation manifold's (RM) lower-dimensional projections. We show that the differences between methods can be understood more clearly by investigating the RM directly, which allows for a more detailed comparison. To this end, we propose a framework and new metric to measure and compare different RMs. We also investigate and report on the RM characteristics for various pretraining methods. These characteristics are measured by applying sequentially larger local alterations to the input data, using white noise injections and Projected Gradient Descent (PGD) adversarial attacks, and then tracking each datapoint. We calculate the total distance moved for each datapoint and the relative change in distance between successive alterations. We show that self-supervised methods learn an RM where alterations lead to large but constant size changes, indicating a smoother RM than fully supervised methods. We then combine these measurements into one metric, the Representation Manifold Quality Metric (RMQM), where larger values indicate larger and less variable step sizes, and show that RMQM correlates positively with performance on downstream tasks.