# Representational Capacity of Deep Neural Networks -- A Computing Study

## TL;DR

This study investigates whether the theoretical advantage of deep neural networks in representing complex functions translates into practical benefits during training, finding that deep networks often do not outperform shallow ones in reaching optimal solutions.

## Contribution

It provides an empirical evaluation of the representational capacity of deep versus shallow networks, highlighting the challenges in training deep architectures despite their theoretical advantages.

## Key findings

- Deep networks often find worse minima than shallow networks in practice.
- Training deep networks to reach theoretical optimal representations is more difficult.
- Theoretical advantages of depth may not be realized with current training methods.

## Abstract

There is some theoretical evidence that deep neural networks with multiple hidden layers have a potential for more efficient representation of multidimensional mappings than shallow networks with a single hidden layer. The question is whether it is possible to exploit this theoretical advantage for finding such representations with help of numerical training methods. Tests using prototypical problems with a known mean square minimum did not confirm this hypothesis. Minima found with the help of deep networks have always been worse than those found using shallow networks. This does not directly contradict the theoretical findings---it is possible that the superior representational capacity of deep networks is genuine while finding the mean square minimum of such deep networks is a substantially harder problem than with shallow ones.