Prevention is Better than Cure: Handling Basis Collapse and Transparency in Dense Networks

TL;DR

This paper identifies basis collapse as a key issue in dense networks, proposes a modified loss to prevent it, and demonstrates that this leads to more concise, interpretable models with better performance and transparency.

Contribution

The work introduces a new loss function to address basis collapse, provides guidelines for activation choice and scaling, and experimentally verifies the impact on network efficiency and interpretability.

Findings

Basis collapse causes redundant networks.

Modified loss reduces parameters by 100x and lowers MSE by 10x.

Network width depends on feature complexity, not just dimension.

Abstract

Dense nets are an integral part of any classification and regression problem. Recently, these networks have found a new application as solvers for known representations in various domains. However, one crucial issue with dense nets is it's feature interpretation and lack of reproducibility over multiple training runs. In this work, we identify a basis collapse issue as a primary cause and propose a modified loss function that circumvents this problem. We also provide a few general guidelines relating the choice of activations to loss surface roughness and appropriate scaling for designing low-weight dense nets. We demonstrate through carefully chosen numerical experiments that the basis collapse issue leads to the design of massively redundant networks. Our approach results in substantially concise nets, having $100 \times$ fewer parameters, while achieving a much lower $(10\times)$ MSE loss at scale than reported in prior works. Further, we show that the width of a dense net is acutely dependent on the feature complexity. This is in contrast to the dimension dependent width choice reported in prior theoretical works. To the best of our knowledge, this is the first time these issues and contradictions have been reported and experimentally verified. With our design guidelines we render transparency in terms of a low-weight network design. We share our codes for full reproducibility available at https://github.com/smjtgupta/Dense_Net_Regress.