WLD-Reg: A Data-dependent Within-layer Diversity Regularizer
Firas Laakom, Jenni Raitoharju, Alexandros Iosifidis, Moncef, Gabbouj

TL;DR
This paper introduces WLD-Reg, a regularizer that promotes within-layer diversity by measuring pairwise neuron output similarities, improving neural network performance across various tasks.
Contribution
It proposes a novel within-layer diversity regularizer that complements traditional feedback, enhancing neural network training and performance.
Findings
Improves accuracy of state-of-the-art models
Enhances diversity within neural network layers
Demonstrates effectiveness across multiple tasks
Abstract
Neural networks are composed of multiple layers arranged in a hierarchical structure jointly trained with a gradient-based optimization, where the errors are back-propagated from the last layer back to the first one. At each optimization step, neurons at a given layer receive feedback from neurons belonging to higher layers of the hierarchy. In this paper, we propose to complement this traditional 'between-layer' feedback with additional 'within-layer' feedback to encourage the diversity of the activations within the same layer. To this end, we measure the pairwise similarity between the outputs of the neurons and use it to model the layer's overall diversity. We present an extensive empirical study confirming that the proposed approach enhances the performance of several state-of-the-art neural network models in multiple tasks. The code is publically available at…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Taxonomy
TopicsNeural Networks and Applications · Advanced Neural Network Applications · Domain Adaptation and Few-Shot Learning
