Modularizing Deep Learning via Pairwise Learning With Kernels

TL;DR

This paper introduces a modular deep learning framework based on kernel methods that reduces supervision needs, simplifies training workflows, and enhances model reusability and transferability.

Contribution

It presents a kernel-based modular training approach that eliminates the need for backpropagation between modules and improves label efficiency and model reusability.

Findings

Achieves 94.88% accuracy on CIFAR-10 with only 10 labeled examples.

Provides a method to estimate reusability and transferability of modules.

Simplifies deep learning pipeline design and maintenance.

Abstract

By redefining the conventional notions of layers, we present an alternative view on finitely wide, fully trainable deep neural networks as stacked linear models in feature spaces, leading to a kernel machine interpretation. Based on this construction, we then propose a provably optimal modular learning framework for classification that does not require between-module backpropagation. This modular approach brings new insights into the label requirement of deep learning: It leverages only implicit pairwise labels (weak supervision) when learning the hidden modules. When training the output module, on the other hand, it requires full supervision but achieves high label efficiency, needing as few as 10 randomly selected labeled examples (one from each class) to achieve 94.88% accuracy on CIFAR-10 using a ResNet-18 backbone. Moreover, modular training enables fully modularized deep learning workflows, which then simplify the design and implementation of pipelines and improve the maintainability and reusability of models. To showcase the advantages of such a modularized workflow, we describe a simple yet reliable method for estimating reusability of pre-trained modules as well as task transferability in a transfer learning setting. At practically no computation overhead, it precisely described the task space structure of 15 binary classification tasks from CIFAR-10.