# Only sparsity based loss function for learning representations

## TL;DR

This paper investigates the origins of sparsity in neural network representations, linking it to data manifold properties, and introduces a new sparsity-based loss function for learning efficient representations.

## Contribution

It presents a novel loss function based solely on sparsity, applicable as regularization or cost function for various neural network models.

## Key findings

- Sparsity emerges from data on non-linear or discontinuous manifolds.
- The new loss function effectively promotes sparse representations.
- Experiments support the hypothesis linking data distribution to sparsity.

## Abstract

We study the emergence of sparse representations in neural networks. We show that in unsupervised models with regularization, the emergence of sparsity is the result of the input data samples being distributed along highly non-linear or discontinuous manifold. We also derive a similar argument for discriminatively trained networks and present experiments to support this hypothesis. Based on our study of sparsity, we introduce a new loss function which can be used as regularization term for models like autoencoders and MLPs. Further, the same loss function can also be used as a cost function for an unsupervised single-layered neural network model for learning efficient representations.