Inheritance Between Feedforward and Convolutional Networks via Model Projection

TL;DR

This paper formalizes the relationship between feedforward and convolutional networks, introducing model projection as an efficient transfer learning method that inherits feedforward techniques while reducing parameters.

Contribution

It provides a unified formalization of FFNs and CNNs, and proposes model projection, a novel parameter-efficient transfer learning approach for CNNs.

Findings

Model projection reduces trained parameters significantly.

Projected CNNs inherit feedforward techniques.

Experiments show strong transfer learning performance.

Abstract

Techniques for feedforward networks (FFNs) and convolutional networks (CNNs) are frequently reused across families, but the relationship between the underlying model classes is rarely made explicit. We introduce a unified node-level formalization with tensor-valued activations and show that generalized feedforward networks form a strict subset of generalized convolutional networks. Motivated by the mismatch in per-input parameterization between the two families, we propose model projection, a parameter-efficient transfer learning method for CNNs that freezes pretrained per-input-channel filters and learns a single scalar gate for each (output channel, input channel) contribution. Projection keeps all convolutional layers adaptable to downstream tasks while substantially reducing the number of trained parameters in convolutional layers. We prove that projected nodes take the generalized FFN form, enabling projected CNNs to inherit feedforward techniques that do not rely on homogeneous layer inputs. Experiments across multiple ImageNet-pretrained backbones and several downstream image classification datasets show that model projection is a strong transfer learning baseline under simple training recipes.