# On Correlation of Features Extracted by Deep Neural Networks

## TL;DR

This paper investigates how network size, activation functions, and initialization influence the tendency of deep neural networks to extract redundant features, using clustering methods on features from MLP and CNN models.

## Contribution

It identifies key factors like network size and activation functions that promote feature redundancy in DNNs, providing insights into model overparameterization.

## Key findings

- Network size and activation functions significantly increase feature redundancy.
- Redundant features can be estimated via hierarchical clustering based on cosine distances.
- Redundancy patterns are demonstrated on MNIST and CIFAR-10 datasets.

## Abstract

Redundancy in deep neural network (DNN) models has always been one of their most intriguing and important properties. DNNs have been shown to overparameterize, or extract a lot of redundant features. In this work, we explore the impact of size (both width and depth), activation function, and weight initialization on the susceptibility of deep neural network models to extract redundant features. To estimate the number of redundant features in each layer, all the features of a given layer are hierarchically clustered according to their relative cosine distances in feature space and a set threshold. It is shown that both network size and activation function are the two most important components that foster the tendency of DNNs to extract redundant features. The concept is illustrated using deep multilayer perceptron and convolutional neural networks on MNIST digits recognition and CIFAR-10 dataset, respectively.