What Variables Affect Out-of-Distribution Generalization in Pretrained Models?

TL;DR

This paper investigates how various factors like architecture, data, and training conditions influence the out-of-distribution generalization of pre-trained neural network embeddings, challenging previous assumptions about layer compression effects.

Contribution

It provides a comprehensive analysis of factors affecting OOD transferability, highlighting the importance of high-resolution, multi-class training data for better generalization.

Findings

High-resolution, multi-class training data improves transferability.

Layer compression is not a universal phenomenon affecting OOD generalization.

Toy dataset results may not generalize to real-world scenarios.

Abstract

Embeddings produced by pre-trained deep neural networks (DNNs) are widely used; however, their efficacy for downstream tasks can vary widely. We study the factors influencing transferability and out-of-distribution (OOD) generalization of pre-trained DNN embeddings through the lens of the tunnel effect hypothesis, which is closely related to intermediate neural collapse. This hypothesis suggests that deeper DNN layers compress representations and hinder OOD generalization. Contrary to earlier work, our experiments show this is not a universal phenomenon. We comprehensively investigate the impact of DNN architecture, training data, image resolution, and augmentations on transferability. We identify that training with high-resolution datasets containing many classes greatly reduces representation compression and improves transferability. Our results emphasize the danger of generalizing findings from toy datasets to broader contexts.