Squeeze-and-Remember Block

TL;DR

The paper introduces the Squeeze-and-Remember (SR) block, a novel CNN component that adds dynamic memory-like features, improving accuracy on image datasets with minimal extra computation.

Contribution

The SR block is a new architectural unit that enables CNNs to dynamically memorize and reapply important features, enhancing their contextual understanding.

Findings

Improved ImageNet top-1 accuracy by 0.52% with SR block in ResNet50.

Increased Cityscapes mean IOU by 0.20% using SR in DeepLab v3.

Achieved these improvements with minimal computational overhead.

Abstract

Convolutional Neural Networks (CNNs) are important for many machine learning tasks. They are built with different types of layers: convolutional layers that detect features, dropout layers that help to avoid over-reliance on any single neuron, and residual layers that allow the reuse of features. However, CNNs lack a dynamic feature retention mechanism similar to the human brain's memory, limiting their ability to use learned information in new contexts. To bridge this gap, we introduce the "Squeeze-and-Remember" (SR) block, a novel architectural unit that gives CNNs dynamic memory-like functionalities. The SR block selectively memorizes important features during training, and then adaptively re-applies these features during inference. This improves the network's ability to make contextually informed predictions. Empirical results on ImageNet and Cityscapes datasets demonstrate the SR block's efficacy: integration into ResNet50 improved top-1 validation accuracy on ImageNet by 0.52% over dropout2d alone, and its application in DeepLab v3 increased mean Intersection over Union in Cityscapes by 0.20%. These improvements are achieved with minimal computational overhead. This show the SR block's potential to enhance the capabilities of CNNs in image processing tasks.