Learning Non-Parametric Invariances from Data with Permanent Random Connectomes

TL;DR

This paper introduces PRC-NPTN, a novel convolutional network architecture that learns non-parametric invariances directly from data using permanent random connectomes, improving generalization and outperforming existing methods.

Contribution

The paper proposes a new layer with permanent random connectomes for convolutional networks to learn invariances from data, offering a biologically plausible and effective approach.

Findings

PRC-NPTN outperforms larger ConvNet baselines.

Permanent random connectomes improve generalization.

Effective learning of invariances from data itself.

Abstract

One of the fundamental problems in supervised classification and in machine learning in general, is the modelling of non-parametric invariances that exist in data. Most prior art has focused on enforcing priors in the form of invariances to parametric nuisance transformations that are expected to be present in data. Learning non-parametric invariances directly from data remains an important open problem. In this paper, we introduce a new architectural layer for convolutional networks which is capable of learning general invariances from data itself. This layer can learn invariance to non-parametric transformations and interestingly, motivates and incorporates permanent random connectomes, thereby being called Permanent Random Connectome Non-Parametric Transformation Networks (PRC-NPTN). PRC-NPTN networks are initialized with random connections (not just weights) which are a small subset of the connections in a fully connected convolution layer. Importantly, these connections in PRC-NPTNs once initialized remain permanent throughout training and testing. Permanent random connectomes make these architectures loosely more biologically plausible than many other mainstream network architectures which require highly ordered structures. We motivate randomly initialized connections as a simple method to learn invariance from data itself while invoking invariance towards multiple nuisance transformations simultaneously. We find that these randomly initialized permanent connections have positive effects on generalization, outperform much larger ConvNet baselines and the recently proposed Non-Parametric Transformation Network (NPTN) on benchmarks that enforce learning invariances from the data itself.