Entanglement Entropy of Target Functions for Image Classification and Convolutional Neural Network

TL;DR

This paper introduces entanglement entropy as a new information measure for image classification functions, revealing that such functions occupy a small subspace of the Hilbert space and that deeper CNNs are more efficient.

Contribution

It establishes a mathematical connection between image classification and quantum spin models, proposing entanglement entropy as a tool to analyze neural network efficiency and expressive power.

Findings

Target functions of image classification obey sub-volume-law entanglement entropy bounds.

Deeper CNNs with more layers are more efficient, with channel number scaling as D∼D₀^{1/n_c}.

Neural network expressiveness can be characterized using entanglement entropy.

Abstract

The success of deep convolutional neural network (CNN) in computer vision especially image classification problems requests a new information theory for function of image, instead of image itself. In this article, after establishing a deep mathematical connection between image classification problem and quantum spin model, we propose to use entanglement entropy, a generalization of classical Boltzmann-Shannon entropy, as a powerful tool to characterize the information needed for representation of general function of image. We prove that there is a sub-volume-law bound for entanglement entropy of target functions of reasonable image classification problems. Therefore target functions of image classification only occupy a small subspace of the whole Hilbert space. As a result, a neural network with polynomial number of parameters is efficient for representation of such target functions of image. The concept of entanglement entropy can also be useful to characterize the expressive power of different neural networks. For example, we show that to maintain the same expressive power, number of channels $D$ in a convolutional neural network should scale with the number of convolution layers $n_c$ as $D\sim D_0^{\frac{1}{n_c}}$. Therefore, deeper CNN with large $n_c$ is more efficient than shallow ones.