Perceptual Inductive Bias Is What You Need Before Contrastive Learning

TL;DR

This paper proposes a multi-stage pretraining approach inspired by Marr's theory, which improves convergence speed, representation quality, and robustness in contrastive learning by incorporating perceptual inductive biases from early visual processing.

Contribution

It introduces a novel pretraining stage based on human visual perception to enhance contrastive learning, leading to faster convergence and better downstream performance.

Findings

Achieves 2x faster convergence on ResNet18.

Improves semantic segmentation, depth estimation, and object recognition results.

Enhances robustness and out-of-distribution generalization.

Abstract

David Marr's seminal theory of human perception stipulates that visual processing is a multi-stage process, prioritizing the derivation of boundary and surface properties before forming semantic object representations. In contrast, contrastive representation learning frameworks typically bypass this explicit multi-stage approach, defining their objective as the direct learning of a semantic representation space for objects. While effective in general contexts, this approach sacrifices the inductive biases of vision, leading to slower convergence speed and learning shortcut resulting in texture bias. In this work, we demonstrate that leveraging Marr's multi-stage theory-by first constructing boundary and surface-level representations using perceptual constructs from early visual processing stages and subsequently training for object semantics-leads to 2x faster convergence on ResNet18, improved final representations on semantic segmentation, depth estimation, and object recognition, and enhanced robustness and out-of-distribution capability. Together, we propose a pretraining stage before the general contrastive representation pretraining to further enhance the final representation quality and reduce the overall convergence time via inductive bias from human vision systems.