Joint Training Across Multiple Activation Sparsity Regimes

TL;DR

This paper proposes a training strategy that cycles a neural network through various activation sparsity levels, leading to improved generalization by maintaining effective representations across dense and sparse regimes.

Contribution

It introduces a novel joint training method across multiple activation sparsity regimes using global top-k constraints and cyclic training, which outperforms dense baseline methods.

Findings

Adaptive keep-ratio control strategies outperform dense training.

Joint training across sparsity regimes improves generalization.

Preliminary results on CIFAR-10 show promising gains.

Abstract

Generalization in deep neural networks remains only partially understood. Inspired by the stronger generalization tendency of biological systems, we explore the hypothesis that robust internal representations should remain effective across both dense and sparse activation regimes. To test this idea, we introduce a simple training strategy that applies global top-k constraints to hidden activations and repeatedly cycles a single model through multiple activation budgets via progressive compression and periodic reset. Using CIFAR-10 without data augmentation and a WRN-28-4 backbone, we find in single-run experiments that two adaptive keep-ratio control strategies both outperform dense baseline training. These preliminary results suggest that joint training across multiple activation sparsity regimes may provide a simple and effective route to improved generalization.