Transfer entropy-based feedback improves performance in artificial neural networks

TL;DR

This paper demonstrates that incorporating transfer entropy-based feedback in small neural networks significantly enhances their performance on benchmark tasks, mimicking cortical feedback structures.

Contribution

It introduces a novel method of structuring feedback in neural networks using transfer entropy, inspired by cortical architectures, to improve performance.

Findings

Feedback with transfer entropy boosts small network performance.

Small feedback networks can match large feed-forward networks on benchmarks.

Transfer entropy guides effective feedback connectivity in neural models.

Abstract

The structure of the majority of modern deep neural networks is characterized by uni- directional feed-forward connectivity across a very large number of layers. By contrast, the architecture of the cortex of vertebrates contains fewer hierarchical levels but many recurrent and feedback connections. Here we show that a small, few-layer artificial neural network that employs feedback will reach top level performance on a standard benchmark task, otherwise only obtained by large feed-forward structures. To achieve this we use feed-forward transfer entropy between neurons to structure feedback connectivity. Transfer entropy can here intuitively be understood as a measure for the relevance of certain pathways in the network, which are then amplified by feedback. Feedback may therefore be key for high network performance in small brain-like architectures.