Instilling Inductive Biases with Subnetworks

TL;DR

This paper introduces Subtask Induction, a mechanistic approach to instill inductive biases in neural networks by discovering and leveraging functional subnetworks, improving data efficiency and task generalization.

Contribution

The paper proposes a novel Subtask Induction method to embed inductive biases via subnetworks, enhancing model efficiency and generalization capabilities.

Findings

Reduces training data needed for modular arithmetic tasks.

Induces human-like shape bias in image classification models.

Effective for both convolutional and transformer architectures.

Abstract

Despite the recent success of artificial neural networks on a variety of tasks, we have little knowledge or control over the exact solutions these models implement. Instilling inductive biases -- preferences for some solutions over others -- into these models is one promising path toward understanding and controlling their behavior. Much work has been done to study the inherent inductive biases of models and instill different inductive biases through hand-designed architectures or carefully curated training regimens. In this work, we explore a more mechanistic approach: Subtask Induction. Our method discovers a functional subnetwork that implements a particular subtask within a trained model and uses it to instill inductive biases towards solutions utilizing that subtask. Subtask Induction is flexible and efficient, and we demonstrate its effectiveness with two experiments. First, we show that Subtask Induction significantly reduces the amount of training data required for a model to adopt a specific, generalizable solution to a modular arithmetic task. Second, we demonstrate that Subtask Induction successfully induces a human-like shape bias while increasing data efficiency for convolutional and transformer-based image classification models.