Grassmannian Packings in Neural Networks: Learning with Maximal Subspace Packings for Diversity and Anti-Sparsity

TL;DR

This paper introduces a novel initialization method for CNN kernels using Grassmannian packings, which enhances diversity, reduces kernel sparsity, and improves classification accuracy by maximizing subspace separation.

Contribution

It applies Grassmannian subspace packing principles from coding theory to CNN kernel initialization, promoting diversity and anti-sparsity in deep learning models.

Findings

Kernel initializations with Grassmannian packings increase feature diversity.

Grassmannian packings reduce kernel sparsity and dying ReLUs.

Improved classification accuracy and convergence rates in CNNs.

Abstract

Kernel sparsity ("dying ReLUs") and lack of diversity are commonly observed in CNN kernels, which decreases model capacity. Drawing inspiration from information theory and wireless communications, we demonstrate the intersection of coding theory and deep learning through the Grassmannian subspace packing problem in CNNs. We propose Grassmannian packings for initial kernel layers to be initialized maximally far apart based on chordal or Fubini-Study distance. Convolutional kernels initialized with Grassmannian packings exhibit diverse features and obtain diverse representations. We show that Grassmannian packings, especially in the initial layers, address kernel sparsity and encourage diversity, while improving classification accuracy across shallow and deep CNNs with better convergence rates.