Deep Networks with Internal Selective Attention through Feedback Connections

TL;DR

This paper introduces dasNet, a feedback-enabled deep neural network that dynamically adjusts its internal attention during classification, inspired by brain mechanisms, leading to improved performance on CIFAR datasets.

Contribution

It presents a novel feedback architecture for CNNs that uses natural evolution strategies to train internal attention mechanisms, enhancing classification accuracy.

Findings

DasNet outperforms previous state-of-the-art models on CIFAR-10 and CIFAR-100.

Feedback mechanisms enable dynamic filter sensitivity adjustments during classification.

The approach demonstrates the effectiveness of internal selective attention in deep networks.

Abstract

Traditional convolutional neural networks (CNN) are stationary and feedforward. They neither change their parameters during evaluation nor use feedback from higher to lower layers. Real brains, however, do. So does our Deep Attention Selective Network (dasNet) architecture. DasNets feedback structure can dynamically alter its convolutional filter sensitivities during classification. It harnesses the power of sequential processing to improve classification performance, by allowing the network to iteratively focus its internal attention on some of its convolutional filters. Feedback is trained through direct policy search in a huge million-dimensional parameter space, through scalable natural evolution strategies (SNES). On the CIFAR-10 and CIFAR-100 datasets, dasNet outperforms the previous state-of-the-art model.