SynBench: Task-Agnostic Benchmarking of Pretrained Representations using Synthetic Data

TL;DR

SynBench is a task-agnostic benchmarking framework that evaluates the quality of pretrained representations using synthetic data based on a theoretical robustness-accuracy tradeoff, applicable across various models and tasks.

Contribution

It introduces a novel, task-agnostic benchmarking method for pretrained models using synthetic data and a theoretical robustness-accuracy tradeoff as a reference.

Findings

SynBench scores correlate well with actual downstream performance.

Framework can guide robust linear probing to improve model robustness.

Applicable to diverse pretrained models and data types.

Abstract

Recent success in fine-tuning large models, that are pretrained on broad data at scale, on downstream tasks has led to a significant paradigm shift in deep learning, from task-centric model design to task-agnostic representation learning and task-specific fine-tuning. As the representations of pretrained models are used as a foundation for different downstream tasks, this paper proposes a new task-agnostic framework, \textit{SynBench}, to measure the quality of pretrained representations using synthetic data. We set up a reference by a theoretically-derived robustness-accuracy tradeoff of the class conditional Gaussian mixture. Given a pretrained model, the representations of data synthesized from the Gaussian mixture are used to compare with our reference to infer the quality. By comparing the ratio of area-under-curve between the raw data and their representations, SynBench offers a quantifiable score for robustness-accuracy performance benchmarking. Our framework applies to a wide range of pretrained models taking continuous data inputs and is independent of the downstream tasks and datasets. Evaluated with several pretrained vision transformer models, the experimental results show that our SynBench score well matches the actual linear probing performance of the pre-trained model when fine-tuned on downstream tasks. Moreover, our framework can be used to inform the design of robust linear probing on pretrained representations to mitigate the robustness-accuracy tradeoff in downstream tasks.