On the Bias Against Inductive Biases

TL;DR

This paper investigates how removing inductive biases in large transformer models affects their performance in visual feature learning, highlighting that biases can be beneficial even in deep, self-supervised networks.

Contribution

It provides an analysis of the impact of inductive biases on small to medium-sized isotropic networks, challenging the assumption that bias removal is always advantageous.

Findings

Removing inductive biases can negatively impact model performance.

Inductive biases remain beneficial in certain network sizes and tasks.

Analysis helps bridge the gap between large transformer models and accessible research.

Abstract

Borrowing from the transformer models that revolutionized the field of natural language processing, self-supervised feature learning for visual tasks has also seen state-of-the-art success using these extremely deep, isotropic networks. However, the typical AI researcher does not have the resources to evaluate, let alone train, a model with several billion parameters and quadratic self-attention activations. To facilitate further research, it is necessary to understand the features of these huge transformer models that can be adequately studied by the typical researcher. One interesting characteristic of these transformer models is that they remove most of the inductive biases present in classical convolutional networks. In this work, we analyze the effect of these and more inductive biases on small to moderately-sized isotropic networks used for unsupervised visual feature learning and show that their removal is not always ideal.