Interpolated-MLPs: Controllable Inductive Bias

TL;DR

This paper introduces Interpolated-MLPs, a method to controllably adjust inductive bias in neural networks, demonstrating how incremental bias increases improve low-compute vision task performance.

Contribution

The paper proposes a novel interpolation-based approach to control inductive bias in MLPs, enabling fractional bias adjustment and analyzing its impact on low-compute vision tasks.

Findings

Performance improves logarithmically with increased inductive bias in low-compute regimes.

Interpolation allows fractional control of inductive bias between models.

Continuous relationship between bias and accuracy observed across different architectures.

Abstract

Due to their weak inductive bias, Multi-Layer Perceptrons (MLPs) have subpar performance at low-compute levels compared to standard architectures such as convolution-based networks (CNN). Recent work, however, has shown that the performance gap drastically reduces as the amount of compute is increased without changing the amount of inductive bias. In this work, we study the converse: in the low-compute regime, how does the incremental increase of inductive bias affect performance? To quantify inductive bias, we propose a "soft MLP" approach, which we coin Interpolated MLP (I-MLP). We control the amount of inductive bias in the standard MLP by introducing a novel algorithm based on interpolation between fixed weights from a prior model with high inductive bias. We showcase our method using various prior models, including CNNs and the MLP-Mixer architecture. This interpolation scheme allows fractional control of inductive bias, which may be attractive when full inductive bias is not desired (e.g. in the mid-compute regime). We find experimentally that for Vision Tasks in the low-compute regime, there is a continuous and two-sided logarithmic relationship between inductive bias and performance when using CNN and MLP-Mixer prior models.