Trainable back-propagated functional transfer matrices

TL;DR

This paper introduces functional transfer matrices as trainable alternatives to weight matrices in neural networks, enabling the use of real functions with trainable parameters for connections, and demonstrates their effectiveness on MNIST and sequence memorization tasks.

Contribution

The paper proposes functional transfer matrices with trainable functions as a novel alternative to traditional weight matrices in neural networks, extending back-propagation training methods.

Findings

Functional transfer matrices can be trained with revised back-propagation rules.

Networks with functional transfer matrices achieve high accuracy on MNIST.

A functional transfer matrix with memory can memorize 400 digits.

Abstract

Connections between nodes of fully connected neural networks are usually represented by weight matrices. In this article, functional transfer matrices are introduced as alternatives to the weight matrices: Instead of using real weights, a functional transfer matrix uses real functions with trainable parameters to represent connections between nodes. Multiple functional transfer matrices are then stacked together with bias vectors and activations to form deep functional transfer neural networks. These neural networks can be trained within the framework of back-propagation, based on a revision of the delta rules and the error transmission rule for functional connections. In experiments, it is demonstrated that the revised rules can be used to train a range of functional connections: 20 different functions are applied to neural networks with up to 10 hidden layers, and most of them gain high test accuracies on the MNIST database. It is also demonstrated that a functional transfer matrix with a memory function can roughly memorise a non-cyclical sequence of 400 digits.