Unsupervised pre-training helps to conserve views from input distribution

TL;DR

This paper explores how unsupervised pre-training preserves multiple input views and hierarchical representations, leading to better disentanglement and generalization, especially in binary feature settings, compared to supervised learning.

Contribution

It introduces a theoretical perspective on how unsupervised pre-training preserves views and disentangles supervision, improving representation quality and model robustness.

Findings

Unsupervised pre-training conserves multiple input views.

Disentanglement enables linear models to extract label information.

Pre-training reduces overfitting by preserving diverse views.

Abstract

We investigate the effects of the unsupervised pre-training method under the perspective of information theory. If the input distribution displays multiple views of the supervision, then unsupervised pre-training allows to learn hierarchical representation which communicates these views across layers, while disentangling the supervision. Disentanglement of supervision leads learned features to be independent conditionally to the label. In case of binary features, we show that conditional independence allows to extract label's information with a linear model and therefore helps to solve under-fitting. We suppose that representations displaying multiple views help to solve over-fitting because each view provides information that helps to reduce model's variance. We propose a practical method to measure both disentanglement of supervision and quantity of views within a binary representation. We show that unsupervised pre-training helps to conserve views from input distribution, whereas representations learned using supervised models disregard most of them.