MID-L: Matrix-Interpolated Dropout Layer with Layer-wise Neuron Selection

TL;DR

MID-L is a novel neural network module that dynamically activates only the most informative neurons, significantly reducing computation and improving efficiency while maintaining or enhancing accuracy across multiple benchmarks.

Contribution

We introduce MID-L, a differentiable, input-dependent neuron selection layer that seamlessly integrates into existing models for efficient dynamic computation.

Findings

Achieves up to 55% reduction in active neurons

Provides 1.7× FLOPs savings without accuracy loss

Enhances robustness under noisy and overfitting conditions

Abstract

Modern neural networks often activate all neurons for every input, leading to unnecessary computation and inefficiency. We introduce Matrix-Interpolated Dropout Layer (MID-L), a novel module that dynamically selects and activates only the most informative neurons by interpolating between two transformation paths via a learned, input-dependent gating vector. Unlike conventional dropout or static sparsity methods, MID-L employs a differentiable Top-k masking strategy, enabling per-input adaptive computation while maintaining end-to-end differentiability. MID-L is model-agnostic and integrates seamlessly into existing architectures. Extensive experiments on six benchmarks, including MNIST, CIFAR-10, CIFAR-100, SVHN, UCI Adult, and IMDB, show that MID-L achieves up to average 55\% reduction in active neurons, 1.7$\times$ FLOPs savings, and maintains or exceeds baseline accuracy. We further validate the informativeness and selectivity of the learned neurons via Sliced Mutual Information (SMI) and observe improved robustness under overfitting and noisy data conditions. Additionally, MID-L demonstrates favorable inference latency and memory usage profiles, making it suitable for both research exploration and deployment on compute-constrained systems. These results position MID-L as a general-purpose, plug-and-play dynamic computation layer, bridging the gap between dropout regularization and efficient inference.